zhuqi-lucas opened a new issue, #24059: URL: https://github.com/apache/datafusion/issues/24059
### Describe the bug or feature **Is your feature request related to a problem or challenge?** `PredicateRewriter::rewrite_predicate_to_statistics_predicate` (and `PruningPredicate::try_new`) treats `col IN (v1..vn)` specially: when `n <= MAX_LIST_VALUE_SIZE_REWRITE` (currently a hardcoded `20`), the IN list is expanded into a chain of per-value min/max statistics checks; otherwise it falls back to `unhandled_hook`, which by default returns the constant `TRUE` — so row-group / file-range statistics pruning does not fire at all for IN lists longer than 20. The `20` limit was introduced by [#8815](https://github.com/apache/datafusion/pull/8815) as a "minor extract-const" refactor and, before that, was a bare literal inside `build_predicate_expression`. Neither change discusses why 20 specifically was chosen, and there is no way at runtime for a query engine to opt into a higher (or lower) cap. **Real-world impact** Query patterns that pass a batch of identifiers as `col IN (...)` are common — REST endpoints that filter by a page of ~25-100 tickers/customer IDs/document IDs, ORM-generated `WHERE id IN (25 items)` queries, and batched crawlers. On a table whose leading sort column matches the filter column (a very common physical layout for a view intended to serve exactly these queries), row-group and file-range statistics pruning would eliminate the vast majority of containers cheaply — but the current 20 cap forces the reader to fall back to row-level filter pushdown on every surviving row group, which materializes the filter column across the full logical row count. Example: on a table sorted by \`col\`, an \`IN (25 items)\` query on \`col\`: - Today: \`row_groups_pruned_statistics = 0 / N\` (all row groups survive), every row group's \`col\` chunk is fetched and decoded, matches identified only during row-level pushdown. - With a higher cap: the IN would rewrite into a 25-way OR of range checks, most row groups would be pruned before their column chunks are fetched, only a small handful decoded. Also affects file-level pruning where the same `PruningPredicate` is evaluated against file-range stats before opening the parquet footer. **Describe the solution you'd like** Expose the cap as a runtime config option: - Add `datafusion.execution.parquet.pruning_max_in_list_size: usize` (default `20`, preserving existing behaviour) to `TableParquetOptions.global`. - Add `PredicateRewriter::with_max_in_list_size(usize) -> Self` builder method, mirroring the existing `with_unhandled_hook`. - Add `PruningPredicate::try_new_with_max_in_list_size(expr, schema, max_in_list_size)` variant so callers can opt in without going through `PredicateRewriter`. - Add `build_pruning_predicate_with_max_in_list_size(...)` variant for the public helper. - Wire the parquet source / opener to read the config value and pass it through when building row-group + file-range pruning predicates. The internal signature of `build_predicate_expression` gets an extra `max_in_list_size: usize` argument (private crate-local, safe to break). Existing `PruningPredicate::try_new` and `build_pruning_predicate` are preserved as thin wrappers that pass `MAX_LIST_VALUE_SIZE_REWRITE` (which becomes `pub const` so callers can reference the default), so no downstream API breakage. **Describe alternatives you've considered** 1. Just bump the default. Simpler, but takes away the ability to opt out for engines that measured 20 as their sweet spot. 2. Make the constant `pub` without a config knob. Callers would have to fork or wrap the predicate rewriter; doesn't compose with `SessionConfig`. 3. Rewrite IN → OR at the caller side before passing to `PruningPredicate::try_new`. Works, but every downstream engine reinvents the wheel and pays the cost of not knowing DF's own internal 20. **Additional context** Willing to submit a PR. Change is mechanical: - Add config option to `datafusion/common/src/config.rs`. - Thread `max_in_list_size: usize` through `build_predicate_expression`, `PredicateRewriter`, `PruningPredicate::try_new`, and `build_pruning_predicate` (backward-compatible wrappers). - Wire it through `datasource-parquet` alongside the existing `max_predicate_cache_size` field on `ParquetMorselizer` / `PreparedParquetOpen`. - Tests: unit tests on `PredicateRewriter` at cap=0/20/32 verifying rewrite behaviour + an integration test showing row-group pruning for a `col IN (>20 items)` predicate against a sorted table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
