adriangb opened a new pull request, #27091: URL: https://github.com/apache/datafusion/pull/27091
## Which issue does this PR close? - This PR does not close an issue. It follows up on https://github.com/apache/datafusion/pull/26249, which merged the build partitions of a `Partitioned` hash join into one routed lookup. > [!NOTE] > Stacked on https://github.com/apache/datafusion/pull/27090, which adds the benchmark queries used below. Review only the last commit, `perf: route probe rows to ArrayMap build partitions in the merged filter`. ## Rationale for this change A `Partitioned` hash join on dense integer keys runs much slower with Parquet filter pushdown on than with it off. On TPC-H SF10 the `hj` Q26 join takes 93.5 ms with `pushdown_filters=true`, against 50.0 ms with it off. A similar join on sparse keys (Q27) does not slow down this way. The cause is the filter the join pushes into its probe-side scan. Each build partition of a dense-key join builds an `ArrayMap`. The routed lookup from https://github.com/apache/datafusion/pull/26249 accepts only hash tables, so these joins fall back to a `CASE` with one branch per partition. The scan evaluates that `CASE` on every row. ## What changes are included in this PR? The routed lookup now probes an `ArrayMap` partition with the key values of the rows routed to it, because an `ArrayMap` is keyed by value, not by hash. Every non-empty partition can now merge into the routed lookup. The `CASE` fallback remains only for canceled partitions. A follow-up PR removes it. ## What is the testing strategy for this PR? `routed_lookup_fuzz` now builds an `ArrayMap` for about half of its partitions. It compares the routed lookup with one lookup over the whole build side. All `datafusion-physical-plan` tests and all sqllogictests pass. `hj` on TPC-H SF10, 16 partitions, median of 10 iterations: | Query | `pushdown_filters` | `main` + #27090 | This PR | |---|---|---|---| | Q26 (dense keys) | `true` | 93.5 ms | 44.7 ms | | Q26 (dense keys) | `false` | 50.0 ms | 48.2 ms | | Q27 (sparse keys) | `true` | 59.2 ms | 57.3 ms | | Q27 (sparse keys) | `false` | 54.5 ms | 55.6 ms | Only Q26 with pushdown on changes. Its probe filter goes from `CASE hash_repartition % 16 WHEN 0 THEN ... hash_lookup ...` to one `hash_lookup`. ## Are there any user-facing changes? No API changes. A `Partitioned` hash join on dense integer keys runs faster when Parquet filter pushdown is on. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
