anshulsingh-py commented on PR #19850: URL: https://github.com/apache/hudi/pull/19850#issuecomment-5558851375
Nice fix — this closes the "silent empty result" gap for genuinely unsupported/misspelled functions with a clear, actionable error, which is the safer default. One thing I noticed while looking at this: `evaluateFilter` itself is unchanged here, so a caller that evaluates without calling `validateFilterExpression` first would still hit the original silent-drop behavior for a function like `concat` — it's caught upstream by validation, but the evaluator's own gap is still there. I was working from the same issue with a complementary angle: instead of only rejecting functions outside the hardcoded table, fall back to resolving them through Spark's own `FunctionRegistry` when they're not in the table. That way a legitimate builtin like `concat` or `instr` actually evaluates correctly instead of being rejected — while anything not in Spark's registry either (a real typo) still falls through safely and can be caught by validation, same as this PR does. Would it make sense to combine the two: keep this PR's validation-based rejection for truly unknown functions, and add the FunctionRegistry fallback so known-but-unlisted builtins work instead of being blocked? Happy to open that as a follow-up on top of this branch, or fold it in here if that's preferred — whatever's easiest for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
