andygrove opened a new pull request, #68: URL: https://github.com/apache/datafusion-spark/pull/68
## Summary - Adds Spark's `hash(expr1, expr2, ...)`: 32-bit Murmur3 with seed 42, returning a non-nullable Int32. NULL inputs leave the running hash unchanged. - Ported from DataFusion Comet's `hash_funcs/murmur3.rs`. The Murmur3 core is rewritten without `unsafe`. - Makes `hash_primitive_values`, `hash_bytes_values` and `try_hash_bytes_values` in `hash/utils.rs` generic over the hash state type (previously `u64`). `hash` and `xxhash64` now share the list/struct/map/decimal/dictionary handling. ## Testing - Unit tests in `murmur3.rs` and a new `hash/hash.slt`. - Expected values come from Comet's Spark-verified murmur3 tests. They were not re-checked against a live Spark here. - `cargo clippy --all-targets --all-features -- -D warnings`, `cargo test -p datafusion-spark` and `cargo test -p datafusion-spark-sqllogictest` pass. ## Known gap Non-canonical NaNs (sign bit or payload set) should hash like the canonical NaN in Spark. The shared float macro only normalizes `-0.0`, so `xxhash64` and `hash` both miss this. Not addressed here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
