Rich-T-kid opened a new pull request, #26849: URL: https://github.com/apache/datafusion/pull/26849
## Which issue does this PR close? Closes #26848. ## Rationale for this change After dictionary values are hashed, nullable dictionaries still decode and branch on every key while scattering the value hashes back to rows. This O(rows) loop becomes more visible once the value hashes are reused as proposed in #26780. ## What changes are included in this PR? The nullable-key path now reads the validity bitmap 64 rows at a time: - all-valid words reuse the existing dense loop - mostly-valid words gather eight dictionary hashes ahead so the loads can overlap - words with at least 25% nulls walk only the valid bits Null key slots can contain arbitrary raw keys, so speculative gathers clamp those indices before their results are masked out. The unchecked accesses are limited to validated keys and fixed-size chunks. The existing test now covers slices, null keys and values, 8/64-row boundaries, an empty dictionary, and both single and composite hashes. I also tried explicit AArch64 NEON for hash combining and masked selection. Both variants were slower locally, so they are not in the final change. ## Results Apple M4 Max, 8,192-row `with_hashes` benchmark with 3% null keys. These are geometric means of two Criterion medians in candidate/baseline/baseline/candidate order: | case | main | this PR | delta | | --- | ---: | ---: | ---: | | one dictionary column, nullable | 5.90 us | 3.05 us | -48.3% | | three dictionary columns, nullable | 16.97 us | 8.84 us | -47.9% | The unchanged no-null path stayed within run-to-run noise. Exploratory runs over Int8/Int16/Int32/Int64 keys and 1%-99% null density also improved the nullable cases. At 90% nulls, Int32 measured 5.70 to 1.85 us for one column and 16.38 to 2.95 us for three columns. ## What is the testing strategy for this PR? The focused hash tests pass in the normal and `force_hash_collisions` configurations. Workspace formatting, Clippy, and typo checks also pass. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
