xgbtxy opened a new pull request, #59903: URL: https://github.com/apache/spark/pull/59903
### What changes were proposed in this pull request? Backport #59732 to branch-4.2, as requested in the original PR. Return partition predicates rejected by pushdown for post-scan evaluation, preserve this branch's data-filter translation loop, and include the shared Python UDF regression and 4.2.2 migration note. ### Why are the changes needed? A deterministic Python UDF on a partition column can be dropped from both the pushed and residual filters in a V2 file scan. In the regression, a filter that should select two rows returns all six. ### Does this PR introduce any user-facing change? Yes. Affected Data Source V2 file scans now exclude rows that do not satisfy these predicates. Built-in file sources use V1 by default; that path already evaluates the filters correctly. The migration note explains that data derived from affected results may need to be recomputed. ### How was this patch tested? - Built the original branch with Hive support, then rebuilt the patched SQL reactor. `PruneFileSourcePartitionsSuite`: 9/9 passed. - The two shared regressions in `UDFTests` and `ArrowPythonUDFTests` both failed with the original production code and both passed with this patch. Runtime checks verified that the 4.2 PySpark sources and newly compiled `FileScanBuilder` class were loaded. - These local runs used a Windows fixture-only adapter to close the temporary file handle before unlinking it, as described in #59732. - Branch-pinned Ruff 0.14.8 and scalafmt checks passed. The fork workflow is being tracked separately; no full CI success is claimed here. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
