jadenfix opened a new pull request, #7099:
URL: https://github.com/apache/datafusion-comet/pull/7099
## Which issue does this PR close?
Closes #7062.
## Rationale for this change
A shuffle on the JVM columnar shuffle path fails the query when a column is
a struct with a `NullType` field, such as `named_struct('x', a, 'n', null)`.
The two field-major struct functions in
`native/shuffle/src/spark_unsafe/row.rs` have no `DataType::Null` arm and fall
through to `unreachable!`, while `append_field` in the same file already
handles it. With the default configuration this affects every round-robin
repartition of such a column.
## What changes are included in this PR?
- `append_struct_fields_field_major` (top-level struct column) and
`append_nested_struct_fields_field_major` (struct inside a struct) get a
`DataType::Null` arm that appends one null per row to the field's
`NullBuilder`, the same as `append_field`.
- Rust unit tests that lay out Spark `UnsafeRow`s for a top-level and a
nested struct with a `NullType` field, including null structs. Both panic with
the message from the issue without the fix.
- A regression test in `CometColumnarShuffleSuite` that shuffles both shapes
with round-robin and hash partitioning and compares against Spark.
## How are these changes tested?
- `cd native && cargo test -p datafusion-comet-shuffle`: 189 passed,
including the two new tests (which fail on `main`).
- `cargo fmt --all -- --check` and `cargo clippy --all-targets -p
datafusion-comet-shuffle -- -D warnings` are clean.
- The new Scala test has not been run locally (no JDK on this machine); it
relies on CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
