mbutrovich opened a new pull request, #27559: URL: https://github.com/apache/datafusion/pull/27559
## Which issue does this PR close? - None. #26194 worked on the same loop. ## Rationale for this change Hash joins, hash aggregation, and hash repartitioning hash their key columns with `create_hashes`. For a primitive column without nulls, `hash_array_primitive` hashes 16 rows per loop iteration (8 when it combines with an earlier column's hash). Once a column has any null, it walks `valid_indices()` and hashes one row per iteration. In the benchmark bot's Neoverse V2 run on #26194, `int64: single, nulls` (3% nulls) took 8.7 us on `main`, against 4.6 us for `int64: single, no nulls` ([results](https://github.com/apache/datafusion/pull/26194#issuecomment-6117293637)). Nullable keys are common. In TPC-DS, the foreign keys in `store_sales` are 4.5% null and those in `store_returns` are 3.5% null. ## What changes are included in this PR? `hash_array_primitive` hashes an array with nulls 64 rows at a time, against the matching word of the validity bitmap. A chunk with at least `MIN_VALID_FOR_DENSE` (40) valid rows goes through the existing unrolled loops into a stack buffer. A mask built from each row's validity bit then blends the result into the output, so null rows keep their previous hash. A chunk with fewer valid rows, and the last `len % 64` rows, hash only their valid rows by walking the set bits of the word. Deciding per chunk handles batches whose nulls cluster, as after a sort or an outer join. An all-null array returns early, because every row keeps its previous hash. ROLLUP and GROUPING SETS hash such arrays. #26194 also walked 64-row chunks, but took its unrolled path only for chunks without nulls. At 3% random nulls, about 14% of chunks have none (0.97^64). The null path no longer calls `value_unchecked` in an `unsafe` block. The first commit adds the benchmark cases, so the bot can compare against a baseline that has them. ## What is the testing strategy for this PR? `hash_array_primitive_with_nulls` in `datafusion/common/src/hash_utils.rs` compares the result with a per-row reference, both for the first column and for later columns. It covers lengths 1, 63, 64, 65, 127, 128, 129, and 200, four validity patterns including all null, offsets 0 and 5, and `Int64`, `Decimal128`, and `Float64` (with `-0.0` and `NaN`). The first commit extends the `with_hashes` benchmark in `datafusion/common/benches/with_hashes.rs`. It adds null density sweeps for `int64`, `int32`, and `decimal128`, including all-null columns, nulls that fill whole 64-row blocks, and a slice at a bit offset that is not a multiple of 64. ## Are there any user-facing changes? No. The new items are private, and hash values don't change. # AI usage disclosure I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
