anshulsingh-py commented on PR #19853: URL: https://github.com/apache/hudi/pull/19853#issuecomment-5677563996
Rebased onto current master and addressed the latest round (debug log before the registry fallback's catch-all, plus a pass over the comments that had grown into narratives). @voonhous @yihua - when you have a moment, this could use a committer look. The automated reviewer has closed the last several rounds with "no correctness issues found" and the remaining comments have been style/readability, so I think the substance is settled and it needs a human call rather than another automated pass. Where it stands: - Closes #19852: procedure filters can now use any Spark builtin outside the hardcoded table (`concat`, `instr`, `nvl`, `split_part`, ...), instead of silently dropping every row. - Anything genuinely not row-evaluable - aggregates, generators, non-deterministic functions, `Unevaluable` results - still lands on #19850's existing rejection path rather than escaping as an error. - Cross-version work is the bulk of the review history: `TypeCoercionBase`'s rule objects across 3.3-4.2, `With`/`CommonExpressionDef` inlining on 4.0+, and 4.2's `system.builtin` registry keying, which is reached reflectively because `FunctionIdentifier` only gained its catalog parameter in 3.4. - Two real bugs came out of that: the 4.2 registry keying (caught by the one CI lane that runs this suite against 4.2 - the 3.5/4.0/4.1 variants of that job are commented out), and `ILIKE`, which the parser emits directly as a `RuntimeReplaceable` and which the per-row rethrow guard would otherwise have surfaced as an `INTERNAL_ERROR`. Verified locally against real Spark 4.2.0/scala-2.13 as well as the default profile - full `procedure` package, 272/272 both - and `hudi-spark` compiles under `-Pspark3.3`, which CI no longer covers. Happy to split this up or adjust the approach if you'd prefer it land differently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
