viirya opened a new pull request, #5562:
URL: https://github.com/apache/datafusion-comet/pull/5562
## Which issue does this PR close?
N/A -- trivial cleanup, no issue filed.
## Rationale for this change
Two small pieces of dead/leftover code noticed while reading through the
shuffle
type gates. Neither has any behavioral effect; removing them makes the
surrounding code slightly easier to read.
## What changes are included in this PR?
- `CometShuffleExchangeExec`: in the columnar shuffle's
`supportedSerializableDataType`, the `StructType` case tested
`fields.nonEmpty` twice -- once at the start of the conjunction and again
as
the final term. The trailing check is dead:
```scala
case StructType(fields) =>
fields.nonEmpty && fields.forall(f =>
supportedSerializableDataType(f.dataType)) &&
// Java Arrow stream reader cannot work on duplicate field name
fields.map(f => f.name).distinct.length == fields.length &&
fields.nonEmpty // <-- removed
```
It appears to have been introduced accidentally in bf18cec52 ("feat:
Implement TimeType support - Infrastructure - shuffle").
- `CometNativeShuffleSuite`: remove a leftover debugging
`println(shuffled.queryExecution.executedPlan)` from the
"native shuffle with Map[_, NullType] column" test.
Net change is `1 insertion(+), 3 deletions(-)`.
## How are these changes tested?
Covered by existing tests -- no behavior changes, so no new tests are added.
The first change is a boolean identity (`A && B && C && A` where `A` is the
leading term, so the trailing `A` cannot change the result); the second only
removes output to stdout from a test.
Verified locally that `mvn spotless:apply` reports no formatting changes and
that `mvn test-compile -pl spark -am` succeeds. I did not run
`CometNativeShuffleSuite` to a green state locally, since my working copy has
no `libcomet` built and the whole suite falls back without it (the same 11
passed / 25 failed before and after this change, so the failures are
unrelated
to it). Relying on CI for the full suite run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
