sunchao commented on code in PR #6361:
URL: https://github.com/apache/datafusion-comet/pull/6361#discussion_r4128863612
##########
spark/src/test/scala/org/apache/spark/shuffle/sort/SpillSorterSuite.scala:
##########
@@ -262,6 +268,43 @@ class SpillSorterSuite extends AnyFunSuite with
BeforeAndAfterEach {
}
}
+ test("write sorted file across partitions with shuffle checksums disabled") {
+ // With spark.shuffle.checksum.enabled=false the sorter gets an empty
checksum array and no
+ // checksum algorithm.
+ val spills = new java.util.LinkedList[SpillInfo]()
Review Comment:
[P1] Alias Comet’s `SpillInfo` to restore Scala 2.12 builds. When compiling
the supported Spark 3.4 or 3.5 profile, Spark’s
`org.apache.spark.shuffle.sort.SpillInfo` in this suite’s package hides the
newly imported Comet class. This allocation therefore produces the wrong
`LinkedList` type, and passing it to `createSpillSorter` fails compilation
instead of running the regression test. The constructor at line 294 has the
same collision. This blocks both profiles’ test compilation and both Celeborn
compatibility jobs. Use an import alias such as `SpillInfo => CometSpillInfo`
at both allocations, or fully qualify the Comet type.
Evidence: Exact-head CI reports `found:
java.util.LinkedList[org.apache.spark.shuffle.sort.SpillInfo]`, `required:
java.util.LinkedList[org.apache.spark.sql.comet.execution.shuffle.SpillInfo]`
at line 276, plus the hidden-import warning. See Spark 3.5 job
https://github.com/apache/datafusion-comet/actions/runs/36499297541/job/109186916881
and Spark 3.4 job
https://github.com/apache/datafusion-comet/actions/runs/36499297541/job/109186916902.
A local JDK 17 compiler probe using Spark 3.5.9 and the checkout’s actual
Comet SpillInfo reproduced both mismatches with Scala 2.12.18. Aliasing both
uses compiled successfully. Original and aliased forms both compiled with Scala
2.13.17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
