xiaoh1024 opened a new pull request, #58948: URL: https://github.com/apache/spark/pull/58948
### What changes were proposed in this pull request? Extract necessary predicates when a V2 scan filter cannot be fully translated, for both `SupportsPushDownFilters` and `SupportsPushDownV2Filters`. Retain the original expression for post-scan filtering. For example, `id = 1 OR (id = 2 AND split(name, ',')[0] = 'fred')` can push `id = 1 OR id = 2` while Spark evaluates the original condition. ### Why are the changes needed? An unsupported expression in one branch currently prevents the entire OR predicate from being pushed down, even when both branches imply a translatable condition. Extracting that condition lets the data source filter rows earlier. ### Does this PR introduce _any_ user-facing change? Yes. Eligible V2 scans push an additional filter; query results are unchanged. ### How was this patch tested? Added regression tests covering both filter APIs, residual filtering, NULL values, unsupported OR branches, NOT, non-deterministic expressions, and fully translatable filters. Added an H2 JDBC test checking the pushed predicate and query results. ### Was this patch authored or co-authored using generative AI tooling? Yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
