mbutrovich opened a new pull request, #7691: URL: https://github.com/apache/datafusion-comet/pull/7691
## Which issue does this PR close? - None. This PR depends on #7687, and the branch includes its commit. ## Rationale for this change Once a fixed-width column has a null, Comet's murmur3 and xxhash64 test the null bit row by row ([`hash_array_primitive!`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/hash_funcs/utils.rs#L108-L112)), so the loop has a branch per row and doesn't vectorize. Shuffle hash partitioning hits this on nullable keys. In TPC-DS, the foreign keys in `store_sales` are 4.5% null, so nearly every batch has a null. ## What changes are included in this PR? Integer and float columns with nulls are hashed 64 rows at a time, adapting apache/datafusion#27559. A chunk with at least 16 valid rows hashes every row with the vectorized loop from #7687 and then restores the previous hashes of its null rows. A sparser chunk hashes only its valid rows. Small decimals, intervals, and booleans keep the per-row loop, because `i64::try_from` must not run on a null slot. ## Benchmark results These are local runs on an Apple M5 Max. They don't predict the Neoverse-N1 and x86-64-v3 builds that Comet releases. Both sides used `-Ctarget-cpu=native` and thin LTO, and they alternated within one run. The aligned layout adds `-C llvm-args=-align-all-functions=7 -C llvm-args=-align-all-nofallthru-blocks=7` to both sides. The baseline is #7687. <img width="1600" height="2148" alt="hash-nulls-local" src="https://github.com/user-attachments/assets/32fd4e29-f351-4072-8820-0a2b6a2c23b3" /> A dense chunk hashes its null slots too and then writes their previous hashes back, so its cost grows with the number of nulls up to the threshold. A chunk just above 16 valid rows pays the most for that. These runs don't include a null rate near that point (about 75%). Could a reviewer run the same comparison on x86 with `-Ctarget-cpu=x86-64-v3` and on Neoverse-N1 with `-Ctarget-cpu=neoverse-n1`, against #7687? On both sides: ```shell cd native RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo bench -p datafusion-comet-spark-expr --bench hash_nulls ``` The `no_nulls` case doesn't reach the changed code, so it shows the noise floor. ## How are these changes tested? `hash_fixed_width_with_nulls_matches_per_row` in `hash_funcs/utils.rs` checks the chunk loop against hashing row by row, for lengths from 1 to 200, offsets 0 and 5, and validity patterns at and one below the threshold, with a null run longer than a chunk, and all null. `create_hashes_with_nulls_matches_per_row` checks int32, int64, and float64 columns, including -0.0 and NaN, through murmur3 and xxhash64. # AI usage disclosure I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
