mbutrovich opened a new pull request, #7687:
URL: https://github.com/apache/datafusion-comet/pull/7687

   ## Which issue does this PR close?
   
   - None.
   
   ## Rationale for this change
   
   The loops that hash fixed-width columns without nulls in Comet's murmur3 and 
xxhash64 index both the values and the hash buffer per row 
([`hash_array_primitive!`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/spark-expr/src/hash_funcs/utils.rs#L99-L105)).
 Inside `create_murmur3_hashes`, LLVM reloads the values pointer and checks 
bounds on every row, and reports `loop not vectorized: cannot identify array 
bounds`. In the 1.2.0 jar, `create_murmur3_hashes` has no vector multiply in 
either the amd64 or the aarch64 `libcomet.so`. Murmur3 is the hash for shuffle 
hash partitioning 
([`multi_partition.rs`](https://github.com/apache/datafusion-comet/blob/d187e6823f9052e7331e8a810b08a1202debb91c/native/shuffle/src/partitioners/multi_partition.rs#L302)).
   
   ## What changes are included in this PR?
   
   The loops for arrays without nulls iterate the hash buffer and the values 
together, so the primitive, float, boolean, interval, and small decimal loops 
can vectorize. With this change, the Int32 and Int64 murmur3 loops vectorize in 
a thin-LTO `libcomet` built with `-Ctarget-cpu=x86-64-v3` and with 
`-Ctarget-cpu=neoverse-n1`. Each column's length is now checked against the 
hash buffer, because a zipped loop would skip rows instead of panicking.
   
   The loops for arrays with nulls are unchanged.
   
   ## How are these changes tested?
   
   The existing hash tests check the hash values. 
`test_hashes_buffer_length_mismatch_is_error` in `murmur3.rs` and `xxhash64.rs` 
checks the length check.
   
   Could a reviewer with an x86 machine benchmark this against `main`? I don't 
have x86 hardware, and the xxhash64 case is the open question: on x86-64-v3, 
LLVM vectorizes it by emulating 64-bit multiplies with `vpmuludq`, and only a 
benchmark shows whether that beats the scalar loop. On both sides:
   
   ```shell
   cd native
   RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo bench -p 
datafusion-comet-spark-expr \
     --bench xxhash64 --bench murmur3 --bench hash -- \
     
'fallback/custom_seed_i64$|compatible/primitive_i64$|^spark_xxhash64/8192/|^spark_murmur3_hash/8192/|^murmur3/int32$'
   ```
   
   `compatible/primitive_i64` runs through datafusion-spark, which this PR 
doesn't change, so it shows the noise floor. A shuffle-heavy TPC-H or TPC-DS 
run with an x86-64-v3 release jar would show the effect on shuffle hash 
partitioning.
   
   # AI usage disclosure
   
   I wrote this PR with help from Claude but understand and support the core 
approach, implementation, and test coverage.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to