Dandandan opened a new pull request, #24447: URL: https://github.com/apache/datafusion/pull/24447
## Which issue does this PR close? - N/A ## Rationale for this change `perfect_hash_join_small_build_threshold` decides when a hash join can use the array-map fast path regardless of key density. The default of 1024 admits only very small key ranges — an array of 1024 entries is about 4 KB, far below any sensible memory budget. The threshold is really a memory budget in disguise: the array map holds one entry per value in the key range, and the allocation is taken from the memory pool, so a build side that does not fit is rejected rather than overcommitting. Raising it to 256 K admits ranges costing roughly a megabyte. TPC-H `p_partkey` has a range of exactly 200,000, so at the old default every join on it fell back to hashing. ## What changes are included in this PR? Default raised from 1024 to `256 * 1024`. No logic changes. ## Results TPC-H SF1, sum of `join_time` + `build_time` across all `HashJoinExec` (the metric the change acts on; wall clock on my machine was too noisy to resolve reliably): | | default | 256K | | |---|---|---|---| | q17 | 383.3 ms | 45.6 ms | 8.4x | | q8 | 227.6 ms | 94.6 ms | 2.4x | | q9 | 556.3 ms | 440.0 ms | 1.26x | | all 9 join-heavy queries | 4237 ms | 3657 ms | **-15.9%** | Array maps built across those queries go from 160 to 195. The effect saturates at 256 K — 1 M and 4 M give identical map counts and times — which is why this is the value chosen rather than something larger. Peak build memory is essentially unchanged, except on the query that gains most: | | default | 256K | |---|---|---| | q17 | 3.4 MB | 11.0 MB | | q8 | 28.3 MB | 29.5 MB | | others | — | +0 to +1.2 MB | All 22 TPC-H queries return identical row counts. ## Are these changes tested? Existing join tests pass. The value is exercised by TPC-H above; there is no new code path, only a different default for one already-tested branch. ## Are there any user-facing changes? One default changes. Joins on integer keys with ranges under 256 K now use the array-map path, trading up to ~1 MB per build side for faster probes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
