comphead commented on code in PR #7704:
URL: https://github.com/apache/datafusion-comet/pull/7704#discussion_r4645583312


##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -176,6 +176,15 @@ case class EliminateRedundantTransitions(session: 
SparkSession)
             CometColumnarShuffle,
             _) =>
         s.withNewChildren(Seq(child))
+
+      // Spark also adds `RowToColumnar` even when comet chose native shuffle.
+      // Native shuffle reads only Comet vectors, not Spark's vectors.
+      // Switch to row and drop the transition.

Review Comment:
   This adds a second way for a native exchange to end up as 
`CometColumnarExchange`, and a second producer of the Spark `RowToColumnarExec` 
that native shuffle cannot read. The input-contract paragraph in 
`docs/source/contributor-guide/native_shuffle.md` (just before "When Native 
Shuffle is Used") only describes the `RevertNativeForTransitionHeavyStages` 
case, and the "When JVM Shuffle is Used" list in `jvm_shuffle.md` does not have 
this one. Could you add it to both pages in this PR? Also, "Switch to the JVM 
columnar shuffle, which reads the child's rows" would be clearer here than 
"Switch to row".



##########
spark/src/test/scala/org/apache/comet/rules/EliminateRedundantTransitionsSuite.scala:
##########
@@ -64,4 +68,31 @@ class EliminateRedundantTransitionsSuite extends 
CometTestBase {
         s"Spark-only branch should be left untouched:\n${result.treeString}")
     }
   }
+
+  // A native shuffle reads Arrow-backed Comet vectors. If its child turns 
row-based after Comet
+  // chose native shuffle, Spark add its own `RowToColumnar` under it, whose 
vectors are unable
+  // to be read.
+  test("native shuffle over Spark's RowToColumnar moves to the JVM columnar 
shuffle") {
+    val rowChild = sparkPlanFor("SELECT id, CAST(id AS STRING) AS s FROM 
range(10)")
+    assert(!rowChild.supportsColumnar)
+    val partitioning = HashPartitioning(rowChild.output.take(1), 4)
+    val shuffle = CometShuffleExchangeExec(
+      partitioning,
+      RowToColumnarExec(rowChild),
+      ShuffleExchangeExec(partitioning, rowChild),
+      shuffleType = CometNativeShuffle)
+
+    val result = EliminateRedundantTransitions(spark).apply(shuffle)
+
+    result match {
+      case CometShuffleExchangeExec(_, child, _, _, CometColumnarShuffle, _) =>
+        assert(child == rowChild, s"RowToColumnar should be 
removed:\n${result.treeString}")
+      case _ =>
+        fail(s"Expected the JVM columnar shuffle:\n${result.treeString}")
+    }

Review Comment:
   This test only takes the branch where `columnarShuffleSupports` is true. 
Could you add a case under `withSQLConf(CometConf.COMET_SHUFFLE_MODE.key -> 
"native")` that pins what the rule does when the JVM columnar shuffle is not 
available? Today that is "plan unchanged", or with the bridge suggested above, 
a native exchange over `CometSparkToColumnarExec`.



##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala:
##########
@@ -575,6 +575,15 @@ object CometShuffleExchangeExec
       nativeShuffleFailureReasons(s).isEmpty &&
       !hashesDifferentlyFromSpark(s)
 
+  /**
+   * Whether the JVM columnar shuffle can take `s` with `rowChild`, a 
row-based plan, as its
+   * child. Pure: does not tag the node
+   */
+  def columnarShuffleSupports(s: CometShuffleExchangeExec, rowChild: 
SparkPlan): Boolean =
+    !isCometCelebornShuffleManagerEnabled(s.conf) &&
+      columnarShuffleFailureReasons(
+        ShuffleExchangeExec(s.outputPartitioning, rowChild, 
s.shuffleOrigin)).isEmpty

Review Comment:
   Can the `isCometCelebornShuffleManagerEnabled` check ever decide the result? 
Under Celeborn, `shuffleSupported` declines every exchange unless the mode is 
`native` (through `cometCelebornShuffleFallbackReason`), and for `native` mode 
`columnarShuffleFailureReasons` already returns "Comet columnar shuffle not 
enabled". If that holds, the helper is just 
`columnarShuffleFailureReasons(...).isEmpty`.



##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -176,6 +176,15 @@ case class EliminateRedundantTransitions(session: 
SparkSession)
             CometColumnarShuffle,
             _) =>
         s.withNewChildren(Seq(child))
+
+      // Spark also adds `RowToColumnar` even when comet chose native shuffle.
+      // Native shuffle reads only Comet vectors, not Spark's vectors.
+      // Switch to row and drop the transition.
+      case s @ CometShuffleExchangeExec(_, RowToColumnarExec(child), _, _, 
CometNativeShuffle, _)
+          if CometShuffleExchangeExec.columnarShuffleSupports(s, child) =>

Review Comment:
   When `columnarShuffleSupports` is false this arm does not match. The native 
exchange keeps `RowToColumnarExec` as its child, and every task fails, now with 
the new `IllegalStateException`. That case is not rare. 
`columnarShuffleFailureReasons` rejects `spark.comet.shuffle.mode=native` up 
front, and under the Celeborn manager a Comet exchange only exists in `native` 
mode, so the guard is false for every `native`-mode plan and every Celeborn 
plan. Could the declined case swap the transition for 
`CometSparkToColumnarExec(child)` instead, when 
`CometSparkToColumnarExec.isSchemaSupported` accepts the schema? That is how 
`RevertNativeForTransitionHeavyStages.revertStageIfNeeded` bridges a reverted 
stage under a native shuffle (`outputArrow`), and it keeps native shuffle. The 
JVM columnar shuffle can stay the first choice, since it avoids the string-key 
placement difference that `hashesDifferentlyFromSpark` describes. I have not 
checked whether the Arrow stream input works under Celeborn
 , so that case may need to stay as it is.



##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -176,6 +176,15 @@ case class EliminateRedundantTransitions(session: 
SparkSession)
             CometColumnarShuffle,
             _) =>
         s.withNewChildren(Seq(child))
+
+      // Spark also adds `RowToColumnar` even when comet chose native shuffle.
+      // Native shuffle reads only Comet vectors, not Spark's vectors.
+      // Switch to row and drop the transition.
+      case s @ CometShuffleExchangeExec(_, RowToColumnarExec(child), _, _, 
CometNativeShuffle, _)
+          if CometShuffleExchangeExec.columnarShuffleSupports(s, child) =>
+        val columnar = s.copy(child = child, shuffleType = 
CometColumnarShuffle)
+        columnar.copyTagsFrom(s)
+        columnar

Review Comment:
   The rewrite turns `CometExchange` into `CometColumnarExchange` without 
saying why. Could this tag the new node with `withInfo(columnar, ...)`, saying 
the child became row-based after native shuffle was chosen? The verbose 
extended explain prints that as `[COMET-INFO: ...]`. Today the only plan that 
reaches this arm comes from EMR's nested AQE plan, so the explain output is 
where someone debugging it will look first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to