haziqishere commented on code in PR #7704:
URL: https://github.com/apache/datafusion-comet/pull/7704#discussion_r4652329461


##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -176,6 +176,15 @@ case class EliminateRedundantTransitions(session: 
SparkSession)
             CometColumnarShuffle,
             _) =>
         s.withNewChildren(Seq(child))
+
+      // Spark also adds `RowToColumnar` even when comet chose native shuffle.
+      // Native shuffle reads only Comet vectors, not Spark's vectors.
+      // Switch to row and drop the transition.
+      case s @ CometShuffleExchangeExec(_, RowToColumnarExec(child), _, _, 
CometNativeShuffle, _)
+          if CometShuffleExchangeExec.columnarShuffleSupports(s, child) =>
+        val columnar = s.copy(child = child, shuffleType = 
CometColumnarShuffle)
+        columnar.copyTagsFrom(s)
+        columnar

Review Comment:
   Done in 53fbaeacc. Both arms now tag the new exchange with `withInfo`, 
saying the child became row-based after native shuffle was chosen and which way 
the rule resolved it. `copyTagsFrom(s)` runs afterward so the original tags are 
kept aswell.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to