xgbtxy opened a new pull request, #59732: URL: https://github.com/apache/spark/pull/59732
### What changes were proposed in this pull request? Preserve partition predicates rejected by `FileScanBuilder` as post-scan filters. Python UDFs remain excluded from partition pruning, while their predicates remain available for subsequent Python UDF extraction and evaluation. Extend the existing shared PySpark DSv2 UDF regression with a partitioned input and exact-row checks for a partition UDF, `AND`, and `OR`. Add a SQL migration note for the corrected results. ### Why are the changes needed? [SPARK-60486](https://issues.apache.org/jira/browse/SPARK-60486) reports incorrect results when filtering partitioned files through Data Source V2 with Python UDFs. The pruning guard added by [SPARK-48666](https://github.com/apache/spark/pull/47313) removes such predicates from the pushed partition filters, but the returned post-scan filters currently omit them as well. The scan can therefore return rows that do not satisfy the original filter. For example, after writing six rows with `i = 0..5` and partition column `p = i % 3`, filtering a V2 Parquet read with `udf(lambda p: p == 1, "boolean")("p")` returns all six rows on the unmodified master. It should return only `(i=1, p=1)` and `(i=4, p=1)`. ### Does this PR introduce _any_ user-facing change? Yes. Affected Data Source V2 file queries now apply the Python UDF filter and exclude rows that were incorrectly retained. This was independently reproduced on released Spark 4.2.0 and current master; this PR targets master. ### How was this patch tested? - Built the unmodified master with Hive support, then rebuilt the candidate SQL reactor with Maven. `PruneFileSourcePartitionsSuite`: **9/9 passed**. - The two expanded regressions: **2 failures before the production change, 2/2 passed afterward**. An additional **20/20** existing filter, join, subquery, and nondeterministic UDF tests passed across pickle and Arrow UDF classes. - An independent **44-case** matrix covering Parquet/ORC, V1/V2, pickle/Arrow, NULL partitions, compound predicates, and controls: **12 incorrect results before the fix, 44/44 correct afterward**, with identical input file bytes and expected results. The 32 control cases remained correct. - Ruff check/format, Scala formatter validation, and `git diff --check` passed. The only difference between the locally built Scala source and the final source is two whitespace-only scalafmt changes (indentation and argument wrapping). - [Fork CI](https://github.com/xgbtxy/spark/actions/runs/39878240762) is running for commit `f886f78b`. Local baseline: master `7e9a966c`, Java 17, Python 3.12. Local PySpark tests use Python 3.12 on Windows through `unittest`. A process-local fixture adapter closes only `ReusedSQLTestCase.setUpClass`'s temporary file before its existing unlink call, because Windows rejects unlinking the open handle. Product code, test assertions, and SQL/Arrow settings are unchanged by that adapter. The stock `python/run-tests` driver requires POSIX `pty` support. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-6) This contribution is original work submitted under the project's Apache License 2.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
