haziqishere commented on code in PR #7704:
URL: https://github.com/apache/datafusion-comet/pull/7704#discussion_r4652369704
##########
spark/src/test/scala/org/apache/comet/rules/EliminateRedundantTransitionsSuite.scala:
##########
@@ -64,4 +68,31 @@ class EliminateRedundantTransitionsSuite extends
CometTestBase {
s"Spark-only branch should be left untouched:\n${result.treeString}")
}
}
+
+ // A native shuffle reads Arrow-backed Comet vectors. If its child turns
row-based after Comet
+ // chose native shuffle, Spark add its own `RowToColumnar` under it, whose
vectors are unable
+ // to be read.
+ test("native shuffle over Spark's RowToColumnar moves to the JVM columnar
shuffle") {
+ val rowChild = sparkPlanFor("SELECT id, CAST(id AS STRING) AS s FROM
range(10)")
+ assert(!rowChild.supportsColumnar)
+ val partitioning = HashPartitioning(rowChild.output.take(1), 4)
+ val shuffle = CometShuffleExchangeExec(
+ partitioning,
+ RowToColumnarExec(rowChild),
+ ShuffleExchangeExec(partitioning, rowChild),
+ shuffleType = CometNativeShuffle)
+
+ val result = EliminateRedundantTransitions(spark).apply(shuffle)
+
+ result match {
+ case CometShuffleExchangeExec(_, child, _, _, CometColumnarShuffle, _) =>
+ assert(child == rowChild, s"RowToColumnar should be
removed:\n${result.treeString}")
+ case _ =>
+ fail(s"Expected the JVM columnar shuffle:\n${result.treeString}")
+ }
Review Comment:
Sure, I added a test case in 3adb0057f: under
`spark.comet.shuffle.mode=native` the rule now produces a native exchange over
`CometSparkToColumnarExec(rowChild)`, and the test executes it and checks all
rows come back. Without the new arm it fails with "Expected native shuffle over
CometSparkToColumnar".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]