LuciferYang opened a new pull request, #60053: URL: https://github.com/apache/spark/pull/60053
### What changes were proposed in this pull request? `CleanupDynamicPruningFilters` drops a DPP filter on a partition key that a static equality already fixes (SPARK-38148). For a `DataSourceV2ScanRelation` it only reads the `Filter` above the scan. When a source fully accepts `part = 1` at scan build time, no `Filter` remains for it, so the DPP filter stays. This PR makes the rule also read the top-level `=` and `<=>` conjuncts in `DataSourceV2ScanRelation.pushedFilters`, which holds the deterministic filters the scan fully accepted. A pushed `part <> 1` or `part = 1 OR part = 2` still leaves the DPP filter in place. For tests, `InMemoryBaseTable` gets a `push-partition-filters` table property. With it set, the scan builder fully accepts `=`, `<=>`, `NOT` and `OR` over an integral, boolean or string identity partition column and applies them with SQL's NULL semantics. ### Why are the changes needed? The DPP filter can't prune anything once an equality fixes the key, and V1 and Hive tables already lose it. On a V2 source that fully pushes the equality, Spark still plans a DPP subquery that prunes nothing. ### Does this PR introduce _any_ user-facing change? No. Query results are the same. Such a query no longer plans the unused dynamic pruning subquery. ### How was this patch tested? New tests in `DynamicPartitionPruningV2Suite`: - "SPARK-38148: no DPP over a partition column a fully pushed static equality fixes": on a table that fully pushes partition filters, `=` and `<=>` with the literal on either side remove the DPP filter, while `>` (not pushed), `<>` and `OR` keep it. Each case is also checked against the same query on a table without the property. The test fails without the rule change. - "push-partition-filters returns what Spark's filter does, including a NULL key": checks the fixture itself. On a table with a NULL partition, each condition must be fully pushed and return what the same query returns on a table without the property, including `NOT (p <=> 1 OR p = 2)`, which runs with `BooleanSimplification` excluded so the `OR` reaches the scan. A filter on a DATE partition column stays post-scan. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
