adriangb opened a new pull request, #25752: URL: https://github.com/apache/datafusion/pull/25752
## Which issue does this PR close? - Part of https://github.com/apache/datafusion/issues/22883. Design notes: https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt - **Do not merge.** Each piece is its own PR (table below). ## Rationale for this change This branch is the combined end state of the optional filters work of #22883. Use it to benchmark and review the whole design. Review the code in the PR of each piece. ## What changes are included in this PR? Linear history on `main`. Each change is here one time, in the order of the table (30 commits). | PR | Content | State | |---|---|---| | https://github.com/apache/datafusion/pull/25729 | Page index: no row selection when it prunes nothing | Open, draft | | https://github.com/apache/datafusion/pull/25677 | Hash join: collapse partitioned `InList` dynamic filters into one IN list | Open, draft | | https://github.com/apache/datafusion/pull/25713 | Hash join: push bounds and membership as separate filters, benchmark | Open, draft | | https://github.com/apache/datafusion/pull/22384 | Parquet: accept all pushable filters, run rejected conjuncts post-scan, flush fix | Open, draft | | https://github.com/apache/datafusion/pull/25673 | `OptionalFilterPhysicalExpr` (A1) | Open, draft | | https://github.com/apache/datafusion/pull/25722 | Post-scan filter keeps optional filters pruning-only | Open, draft | | https://github.com/apache/datafusion/pull/25721 | `filter_stats` and `FilterExec` adaptive conjunct reordering | Open, draft | | https://github.com/apache/datafusion/pull/25674 | `OptionalFilterGate` | Open, draft | | https://github.com/apache/datafusion/pull/25682 | `optional_filter_mode` config and Parquet consumer | Open, draft | | https://github.com/apache/datafusion/pull/25683 | `FilterExec` gates optional filters | Open, draft | | https://github.com/apache/datafusion/pull/25681 | Producers mark hash join, TopK and aggregate dynamic filters as optional | Open, draft | | https://github.com/apache/datafusion/pull/25726 | Per-conjunct pruning statistics (E2) | Open, draft | | https://github.com/apache/datafusion/pull/25727 | Adaptive placement of scan filter conjuncts (E3 v3) | Open, draft | | https://github.com/apache/datafusion/pull/25715 | Defaults: `optional_filter_mode = adaptive` and `adaptive_filter_placement = true` (A5) | Open, draft | ```mermaid graph LR PI["#25729 page index select-all"] C1["#25677 InList collapse"] C2["#25713 split join filter"] P["#22384 post-scan filter"] A1["#25673 Optional wrapper"] F["#25722 post-scan skips optional"] B2["#25721 filter_stats + reordering"] A3["#25674 gate"] A4["#25682 mode + Parquet consumer"] B1["#25683 FilterExec consumer"] A2["#25681 producers"] E2["#25726 per-conjunct stats"] E3["#25727 adaptive placement"] A5["#25715 adaptive defaults"] C1 --> C2 --> A2 A1 --> A2 P --> F A1 --> F B2 -. shares filter_stats .- A3 A3 --> A4 A1 --> A4 A3 --> B1 A1 --> B1 B2 --> B1 A4 --> E3 F --> E3 E2 --> E3 PI --> E3 E3 --> A5 B1 --> A5 A2 --> A5 ``` | Integration change | Detail | |---|---| | `FilterExec` (#25721 + #25683) | Adaptive reordering applies to the required conjuncts. The gated optional conjuncts run after them | | Parquet consumer | The copy from #25727, which is adapted to the post-scan path of #22384 | | A5 | Changes two defaults: `datafusion.execution.optional_filter_mode = adaptive` and `datafusion.execution.adaptive_filter_placement = true`. `ParquetSource::new` keeps placement off, because the source has no setter for it | | Tests | Tests that check exact row filter metrics set the previous values (runtime row group pruning, predicate cache, optional filters in the row filter, `EXPLAIN ANALYZE` categories, TopK metrics). Plan snapshots show `Optional(DynamicFilter [...])` | ## Known issues | Issue | Effect | |---|---| | Adaptive placement coalesces small batches | With a placement, the scan sends batches smaller than half the batch size through the batch coalescer. On tiny row groups the TopK sees no row before the end of the file, and runtime row group pruning stops (`dynamic_row_group_pruning.slt`: 4 pruned row groups become 0). Full-size batches are not coalesced | | `optional_filter_rows_skipped` counts too many rows | When the placement skips a paused optional conjunct, it counts down the pause with batches of `batch_size` rows, also for row groups with fewer rows (16.38 K skipped rows for a 15-row file) | | TPC-H Q12 | Local SF1: 1.24x slower than `main` with `pushdown_filters = false`. The cause is the hash join dynamic filter in the scan: without it, 0.83x (#25727) | ## What is the testing strategy for this PR? | Command | Result | |---|---| | `cargo test -p datafusion-physical-expr -p datafusion-pruning` | Pass | | `cargo test -p datafusion-physical-plan --lib` | Pass | | `cargo test -p datafusion-datasource-parquet` | Pass | | `cargo test -p datafusion-proto` | Pass | | `cargo test -p datafusion --test parquet_integration --test core_integration` | Pass | | `cargo test --test sqllogictests` | Pass | | `cargo clippy --lib --tests --features parquet_encryption -- -D warnings` (touched crates) | Pass | Benchmarks: bot runs in the comments. | Run | Baseline | Changed | |---|---|---| | 1 | `main`, defaults (`pushdown_filters = false`) | This branch, `pushdown_filters = true` | | 2 | `main`, defaults | This branch, defaults | Success criterion: in run 1, no query is slower than `main` with `pushdown_filters = false`. ## Are there any user-facing changes? Yes, from the pieces: new config options `optional_filter_mode`, `optional_filter_min_saving_ns_per_row`, `adaptive_filter_reordering` and `adaptive_filter_placement`, new metrics, and `Optional(...)` in plans. A5 changes two defaults (upgrade guide note in `56.0.0.md`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
