ump45nose opened a new pull request, #7084: URL: https://github.com/apache/datafusion-comet/pull/7084
## Which issue does this PR close? Closes #7063. ## Rationale for this change The native Murmur3 and xxHash64 kernels had no arm for `Null`, `Interval(YearMonth)` or `Duration(Microsecond)`, so `hash(a, null)`, `xxhash64(a, null)` and native round-robin shuffle over those types failed with `Unsupported data type in hasher`. ## What changes are included in this PR? - `create_hashes_internal!` hashes `Interval(YearMonth)` as an int and `Duration(Microsecond)` as a long, as Spark does for `YearMonthIntervalType` and `DayTimeIntervalType`, and leaves the hash unchanged for `Null`. This covers both the hash expressions and round-robin partitioning, since they share the kernel. - `child_with_parent_nulls` now returns a `Null` child as is. Without that, hashing a null struct that has a `NullType` field still panicked once the `Null` arm was in place, because a `NullArray` can't carry a null buffer (`NullArray data should not contain a null buffer`). I left `hash.scala` alone, so `hash`/`xxhash64` over the ANSI interval types still fall back to Spark. Enabling them there could be a follow-up now that the kernel handles them. ## How are these changes tested? New unit tests in `murmur3.rs` and `xxhash64.rs`: both interval types hash to the same values as the existing `Int32`/`Int64` cases, and a `Null` column or a `NullType` struct field (including under a null struct) leaves the hash unchanged. A unit test in `struct_nulls.rs` covers the `Null` child. I ran the native tests for the two changed crates; I did not run the Spark/JVM suites. AI disclosure: the change, the tests and this description were written with an AI coding assistant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
