viirya opened a new pull request, #5564:
URL: https://github.com/apache/datafusion-comet/pull/5564
## Which issue does this PR close?
Closes #5563.
## Rationale for this change
`CometShuffleExchangeExec.supportedSerializableDataType` allows struct data
columns in native shuffle, but `CometNativeShuffleSuite` never exercised one
--
the only nested coverage was `array<array<int>>` and `map<int, null>`, and
the
word `struct` did not appear in the suite at all. `CometColumnarShuffleSuite`
has extensive nested coverage, so the gap was specific to native shuffle.
Nested types in shuffle have been a real source of bugs (e.g. cd4d0e25d,
#5137,
on nested field nullability in `ShuffleScanExec`), so a behavior the type
gate
explicitly claims to support deserves an end-to-end test rather than relying
on
it not regressing.
## What changes are included in this PR?
Test-only. Four tests added to `CometNativeShuffleSuite`:
- `native shuffle on struct data column` -- a plain `struct<int, string>`
- `native shuffle on struct data column including nulls` -- a null struct
and a
null field inside a struct
- `native shuffle on deeply nested data columns` -- `struct<array<int>,
string>`
and `array<struct<int, string>>`, covering the recursive branches of the
gate
- `native shuffle on struct data column with map field`
All of them partition on a primitive column and carry the struct as a data
column, because `supportedHashPartitioningDataType` rejects nested types as a
hash key -- a struct can only reach native shuffle as data. The first test
carries a comment explaining this, so the arrangement is not mistaken for an
oversight.
No product code is changed.
## How are these changes tested?
This PR is the tests. `CometNativeShuffleSuite` passes locally: 40 tests,
40 succeeded, 0 failed (36 pre-existing + 4 new).
I also mutation-tested them to confirm they are not vacuous: temporarily
changing the native gate's `case StructType(...)` to `false` makes exactly
these
4 tests fail while the other 36 keep passing, so they really do assert that
struct data columns go through native shuffle, rather than passing because
the
plan silently fell back to Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
