mbutrovich opened a new pull request, #27559:
URL: https://github.com/apache/datafusion/pull/27559

   ## Which issue does this PR close?
   
   - None. #26194 worked on the same loop.
   
   ## Rationale for this change
   
   Hash joins, hash aggregation, and hash repartitioning hash their key columns 
with `create_hashes`. For a primitive column without nulls, 
`hash_array_primitive` hashes 16 rows per loop iteration (8 when it combines 
with an earlier column's hash). Once a column has any null, it walks 
`valid_indices()` and hashes one row per iteration. In the benchmark bot's 
Neoverse V2 run on #26194, `int64: single, nulls` (3% nulls) took 8.7 us on 
`main`, against 4.6 us for `int64: single, no nulls` 
([results](https://github.com/apache/datafusion/pull/26194#issuecomment-6117293637)).
 Nullable keys are common. In TPC-DS, the foreign keys in `store_sales` are 
4.5% null and those in `store_returns` are 3.5% null.
   
   ## What changes are included in this PR?
   
   `hash_array_primitive` hashes an array with nulls 64 rows at a time, against 
the matching word of the validity bitmap. A chunk with at least 
`MIN_VALID_FOR_DENSE` (40) valid rows goes through the existing unrolled loops 
into a stack buffer. A mask built from each row's validity bit then blends the 
result into the output, so null rows keep their previous hash. A chunk with 
fewer valid rows, and the last `len % 64` rows, hash only their valid rows by 
walking the set bits of the word. Deciding per chunk handles batches whose 
nulls cluster, as after a sort or an outer join. An all-null array returns 
early, because every row keeps its previous hash. ROLLUP and GROUPING SETS hash 
such arrays.
   
   #26194 also walked 64-row chunks, but took its unrolled path only for chunks 
without nulls. At 3% random nulls, about 14% of chunks have none (0.97^64).
   
   The null path no longer calls `value_unchecked` in an `unsafe` block.
   
   The first commit adds the benchmark cases, so the bot can compare against a 
baseline that has them.
   
   ## What is the testing strategy for this PR?
   
   `hash_array_primitive_with_nulls` in `datafusion/common/src/hash_utils.rs` 
compares the result with a per-row reference, both for the first column and for 
later columns. It covers lengths 1, 63, 64, 65, 127, 128, 129, and 200, four 
validity patterns including all null, offsets 0 and 5, and `Int64`, 
`Decimal128`, and `Float64` (with `-0.0` and `NaN`).
   
   The first commit extends the `with_hashes` benchmark in 
`datafusion/common/benches/with_hashes.rs`. It adds null density sweeps for 
`int64`, `int32`, and `decimal128`, including all-null columns, nulls that fill 
whole 64-row blocks, and a slice at a bit offset that is not a multiple of 64.
   
   ## Are there any user-facing changes?
   
   No. The new items are private, and hash values don't change.
   
   # AI usage disclosure
   
   I wrote this PR with help from Claude but understand and support the core 
approach, implementation, and test coverage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to