kumarUjjawal opened a new pull request, #24954: URL: https://github.com/apache/datafusion/pull/24954
## Which issue does this PR close? - Closes #24711. ## Rationale for this change #24781 added compact pruning for non-NULL `NOT IN` lists. Issue #24711 still needs support for NULL-containing lists. Large string lists can create expensive per-value pruning expressions. The compact path reduces this cost when `max_in_list_size` permits the rewrite. NULL values require special handling. Filter pruning treats FALSE and UNKNOWN equally, but full-match detection must keep these results separate. ## What changes are included in this PR? This PR makes these changes: - Supports compact pruning for large string `IN` lists that contain NULL. - Treats `NOT IN` with NULL as unable to match rows. - Treats all-NULL `IN` lists as unable to match rows. - Records whether a pruning predicate is safe for full-match inversion. - Uses the same configured list-size cap when it builds the inverse predicate. - Adds unit tests, Parquet integration tests, documentation, and benchmarks. ## What is the testing strategy for this PR? The tests cover positive and negative lists, all-NULL lists, nested `OR` expressions, configured caps, and Parquet full-match detection. These commands passed: ```shell cargo test -p datafusion-pruning cargo test -p datafusion-datasource-parquet cargo test -p datafusion --test parquet_integration string_in_list_pruning cargo fmt --all -- --check cargo clippy --all-targets --all-features -- -D warnings ``` Focused benchmarks also cover compact `IN` and constant-false `NOT IN` expressions with NULL. ## Are there any user-facing changes? Large string `IN` and `NOT IN` lists that contain NULL can now use compact pruning when the cap permits it. Query results and configuration defaults do not change. This PR adds two non-breaking `PruningPredicate` accessors: `max_in_list_size` and `can_be_inverted_for_full_match`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
