zhuqi-lucas opened a new pull request, #24074: URL: https://github.com/apache/datafusion/pull/24074
## Which issue does this PR close? Closes #24059. ## Rationale `PruningPredicate` rewrites `col IN (v1..vn)` into a chain of per-value min/max checks (via `build_predicate_expression`), but only when `n <= MAX_LIST_VALUE_SIZE_REWRITE` — currently a hardcoded `20`. Beyond that, the IN branch falls through to `unhandled_hook`, which by default returns `TRUE`, so row-group and file-range statistics pruning does not fire at all for IN lists longer than 20. This is problematic for query patterns that pass a batch of identifiers as `col IN (...)` — REST endpoints filtering by a page of ~25-100 values, ORM-generated `WHERE id IN (25 items)` queries, batched crawlers. On a table sorted by `col`, the reader is forced to materialize the filter column across every row group instead of skipping row groups whose stats disagree with the IN set. Full context in #24059. ## What changes are included in this PR? - New config option `datafusion.execution.parquet.pruning_max_in_list_size: usize` (default `20`, preserving existing behaviour), placed next to `max_predicate_cache_size` on `TableParquetOptions.global`. - `MAX_LIST_VALUE_SIZE_REWRITE` promoted to `pub const` so callers can reference the historical default explicitly. - `PredicateRewriter::with_max_in_list_size(usize) -> Self` builder, mirroring the existing `with_unhandled_hook`. - `PruningPredicate::try_new_with_max_in_list_size` variant. - `build_pruning_predicate_with_max_in_list_size` variant of the public helper. - Value threaded through `datasource-parquet`: `ParquetSource::pruning_max_in_list_size()` reads from `TableParquetOptions.global`, propagates through `ParquetMorselizer` → `PreparedParquetOpen` → `RowGroupPruner`, then flows into `build_pruning_predicates` at the opener and `build_pruning_predicate_with_max_in_list_size` inside the dynamic row-group pruner. Internal `build_predicate_expression` gains a new `usize` parameter (crate-private). ## Backward compatibility - `PruningPredicate::try_new` and `build_pruning_predicate` are preserved as thin wrappers that pass the historical `MAX_LIST_VALUE_SIZE_REWRITE` default. All existing callers continue to work with unchanged behaviour. - The config option default is `20`, so behaviour is unchanged unless the option is set explicitly. ## Are these changes tested? Two new unit tests in `datafusion-pruning`: - `row_group_predicate_in_list_rewritten_at_raised_cap`: `PredicateRewriter::with_max_in_list_size(32)` rewrites a 25-item IN into per-value min/max checks OR'd together, instead of falling through to `true`. - `row_group_predicate_in_list_disabled_at_zero_cap`: `cap = 0` skips the IN rewrite even for small lists (opt-out path). The existing `row_group_predicate_in_list_to_many_values` continues to pass, guarding the default-20 behaviour. ## Are there any user-facing changes? Yes — one new config option (`datafusion.execution.parquet.pruning_max_in_list_size`, default `20`). Users who want row-group / file-range pruning for IN lists longer than 20 items can raise it (e.g., `SET datafusion.execution.parquet.pruning_max_in_list_size = 128`). New public API on `datafusion-pruning`: - `MAX_LIST_VALUE_SIZE_REWRITE: usize` (re-exported) - `PredicateRewriter::with_max_in_list_size(usize) -> Self` - `PruningPredicate::try_new_with_max_in_list_size(expr, schema, size)` - `build_pruning_predicate_with_max_in_list_size(predicate, schema, errors, size)` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
