andygrove opened a new pull request, #7647:
URL: https://github.com/apache/datafusion-comet/pull/7647

   ## Which issue does this PR close?
   
   Backport of #7084 to `branch-1.2`. Closes #7063 on this branch.
   
   ## Rationale for this change
   
   The native Murmur3 and xxHash64 kernels have no arm for `Null`, 
`Interval(YearMonth)` or `Duration(Microsecond)`, so `hash(a, null)`, 
`xxhash64(a, null)` and native round-robin shuffle over those types fail with 
`Unsupported data type in hasher`. Hashing a null struct that has a `NullType` 
field also panics (`NullArray data should not contain a null buffer`). 
`branch-1.2` has the same kernels. See #7084 and #7063 for the full analysis.
   
   ## What changes are included in this PR?
   
   Cherry-pick of f1778b986b (`git cherry-pick -x`) with no conflicts and no 
adaptations. It adds the missing arms to `create_hashes_internal!`, makes 
`child_with_parent_nulls` return a `Null` child unchanged, and includes the 
source PR's unit tests and SQL file tests. The diff against the source commit 
is identical apart from line numbers and blob hashes.
   
   Other release branches: `branch-1.2` is the newest release branch, so there 
is no newer one to port to.
   
   ## How are these changes tested?
   
   On this branch, with JDK 17 and the default Spark profile only: the 88 
`hash_funcs` tests in `datafusion-comet-spark-expr` and the `struct_nulls` 
tests in `datafusion-comet-common` pass, `cargo check --all-targets` on both 
crates and `cargo fmt --check` are clean, and `./mvnw test` ran 
`CometSqlFileTestSuite` for `hash_null_type.sql`, 
`round_robin_null_and_intervals.sql`, `get_array_struct_fields.sql` and 
`get_struct_field.sql` against a native library built from this branch (4 
tests, 0 failures). With the native sources (`native/common`, 
`native/spark-expr`) restored to `branch-1.2`'s version, all four SQL file 
tests fail (`Unsupported data type in hasher: Null` and the `NullArray` 
null-buffer panic), so the branch has the bug and the tests catch it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to