szehon-ho opened a new pull request, #57235: URL: https://github.com/apache/spark/pull/57235
### What changes were proposed in this pull request? This PR updates Catalyst filter pushdown for `SupportsPushDownCatalystFilters` so Spark only passes deterministic filters to the connector pushdown API. Nondeterministic filters are kept as post-scan filters and evaluated by Spark after the scan. This also adds a regression test with a test data source implementing `SupportsPushDownCatalystFilters`; the source fails if a nondeterministic predicate reaches `pushFilters`. ### Why are the changes needed? Pushing nondeterministic filters into a data source can change query semantics because the filter may be evaluated at a different point or a different number of times than Spark expects. The existing V2 file scan path already avoids pushing nondeterministic filters; this makes the generic Catalyst filter pushdown utility apply the same guard before invoking `SupportsPushDownCatalystFilters`. ### Does this PR introduce _any_ user-facing change? Yes. Data sources using `SupportsPushDownCatalystFilters` will no longer receive nondeterministic filters for pushdown. Those filters remain in Spark as post-scan filters. ### How was this patch tested? Added a regression test in `DataSourceV2Suite`: `catalyst filter pushdown skips non-deterministic filters`. Local checks run: ```bash git diff --check grep -rn -P "[^\x00-\x7F]" sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2Suite.scala ``` I did not run the SBT test suite locally. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
