adriangb opened a new pull request, #25677: URL: https://github.com/apache/datafusion/pull/25677
## Which issue does this PR close? - Related to #19858 and #22883. Extracted from the prototype #24235. ## Rationale for this change In partitioned mode, the hash join dynamic filter routes each probe row to its partition's filter: ``` CASE hash_repartition % N WHEN 0 THEN bounds_0 AND l IN (list_0) WHEN 1 THEN ... END ``` The `CASE` routing is a large part of the per-row cost of this filter (#19858, #3463), and pruning cannot use a `CASE` expression. When every non-empty build partition uses an `InList`, the routing is not needed: a probe key can only match build rows in the partition it routes to, so the union of all lists gives exactly the same result. The per-partition bounds add nothing, because every value of a list is inside its partition's bounds. The prototype #24235 found that this collapse is the part of that PR that pays for itself (the bounds/membership split alone did not). ## What changes are included in this PR? `build_partitioned_filter` (`shared_bounds.rs`) first tries `union_inlist_filter`. It pushes one `IN` list over all partitions' values when: - at least two build partitions have rows (with one, main already skips the `CASE`), - no partition was canceled and no partition uses a hash table lookup, - all lists have the same data type, and - the lists together are at most 1 MiB. It works for hash and range routing and for multi-column keys, and keeps the NULL-key handling for null-equal joins. Example (`range_partitioning.slt`): ``` before: DynamicFilter [ CASE range_partition WHEN 0 THEN range_key@0 >= 5 AND range_key@0 <= 5 AND range_key@0 IN (SET) ([5]) WHEN 2 THEN range_key@0 >= 20 AND range_key@0 <= 20 AND range_key@0 IN (SET) ([20]) ELSE false END ] after: DynamicFilter [ range_key@0 IN (SET) ([5, 20]) ] ``` The scan in this example now also gets a `pruning_predicate` and `required_guarantees`. ## What is the testing strategy for this PR? - Five new unit tests for `union_inlist_filter` in `shared_bounds.rs`. - Existing tests that check the `CASE` path now force hash table lookups, so that path stays covered. - `filter_pushdown.rs` snapshot and `range_partitioning.slt` updated. - `cargo test -p datafusion-physical-plan --lib hash_join` (also with `force_hash_collisions`), `core_integration physical_optimizer::filter_pushdown`, and the sqllogictest suite pass. Local TPC-H SF1 (noisy machine, 3 rounds × 5 iterations): with `pushdown_filters = true`, q17 went from 129.5 ms to 93.1 ms (min), faster in 3 of 3 rounds. The collapse fires in q17's plan. No measurable change with pushdown off. A benchmark bot run would help. Follow-ups, not in this PR: add the combined min/max next to large lists so pruning can use them; tune the 1 MiB cap. ## Are there any user-facing changes? The text of hash join dynamic filters in `EXPLAIN` changes when the collapse applies. No API change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
