comphead commented on code in PR #7704:
URL: https://github.com/apache/datafusion-comet/pull/7704#discussion_r4645583312
##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -176,6 +176,15 @@ case class EliminateRedundantTransitions(session:
SparkSession)
CometColumnarShuffle,
_) =>
s.withNewChildren(Seq(child))
+
+ // Spark also adds `RowToColumnar` even when comet chose native shuffle.
+ // Native shuffle reads only Comet vectors, not Spark's vectors.
+ // Switch to row and drop the transition.
Review Comment:
This adds a second way for a native exchange to end up as
`CometColumnarExchange`, and a second producer of the Spark `RowToColumnarExec`
that native shuffle cannot read. The input-contract paragraph in
`docs/source/contributor-guide/native_shuffle.md` (just before "When Native
Shuffle is Used") only describes the `RevertNativeForTransitionHeavyStages`
case, and the "When JVM Shuffle is Used" list in `jvm_shuffle.md` does not have
this one. Could you add it to both pages in this PR? Also, "Switch to the JVM
columnar shuffle, which reads the child's rows" would be clearer here than
"Switch to row".
##########
spark/src/test/scala/org/apache/comet/rules/EliminateRedundantTransitionsSuite.scala:
##########
@@ -64,4 +68,31 @@ class EliminateRedundantTransitionsSuite extends
CometTestBase {
s"Spark-only branch should be left untouched:\n${result.treeString}")
}
}
+
+ // A native shuffle reads Arrow-backed Comet vectors. If its child turns
row-based after Comet
+ // chose native shuffle, Spark add its own `RowToColumnar` under it, whose
vectors are unable
+ // to be read.
+ test("native shuffle over Spark's RowToColumnar moves to the JVM columnar
shuffle") {
+ val rowChild = sparkPlanFor("SELECT id, CAST(id AS STRING) AS s FROM
range(10)")
+ assert(!rowChild.supportsColumnar)
+ val partitioning = HashPartitioning(rowChild.output.take(1), 4)
+ val shuffle = CometShuffleExchangeExec(
+ partitioning,
+ RowToColumnarExec(rowChild),
+ ShuffleExchangeExec(partitioning, rowChild),
+ shuffleType = CometNativeShuffle)
+
+ val result = EliminateRedundantTransitions(spark).apply(shuffle)
+
+ result match {
+ case CometShuffleExchangeExec(_, child, _, _, CometColumnarShuffle, _) =>
+ assert(child == rowChild, s"RowToColumnar should be
removed:\n${result.treeString}")
+ case _ =>
+ fail(s"Expected the JVM columnar shuffle:\n${result.treeString}")
+ }
Review Comment:
This test only takes the branch where `columnarShuffleSupports` is true.
Could you add a case under `withSQLConf(CometConf.COMET_SHUFFLE_MODE.key ->
"native")` that pins what the rule does when the JVM columnar shuffle is not
available? Today that is "plan unchanged", or with the bridge suggested above,
a native exchange over `CometSparkToColumnarExec`.
##########
spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/CometShuffleExchangeExec.scala:
##########
@@ -575,6 +575,15 @@ object CometShuffleExchangeExec
nativeShuffleFailureReasons(s).isEmpty &&
!hashesDifferentlyFromSpark(s)
+ /**
+ * Whether the JVM columnar shuffle can take `s` with `rowChild`, a
row-based plan, as its
+ * child. Pure: does not tag the node
+ */
+ def columnarShuffleSupports(s: CometShuffleExchangeExec, rowChild:
SparkPlan): Boolean =
+ !isCometCelebornShuffleManagerEnabled(s.conf) &&
+ columnarShuffleFailureReasons(
+ ShuffleExchangeExec(s.outputPartitioning, rowChild,
s.shuffleOrigin)).isEmpty
Review Comment:
Can the `isCometCelebornShuffleManagerEnabled` check ever decide the result?
Under Celeborn, `shuffleSupported` declines every exchange unless the mode is
`native` (through `cometCelebornShuffleFallbackReason`), and for `native` mode
`columnarShuffleFailureReasons` already returns "Comet columnar shuffle not
enabled". If that holds, the helper is just
`columnarShuffleFailureReasons(...).isEmpty`.
##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -176,6 +176,15 @@ case class EliminateRedundantTransitions(session:
SparkSession)
CometColumnarShuffle,
_) =>
s.withNewChildren(Seq(child))
+
+ // Spark also adds `RowToColumnar` even when comet chose native shuffle.
+ // Native shuffle reads only Comet vectors, not Spark's vectors.
+ // Switch to row and drop the transition.
+ case s @ CometShuffleExchangeExec(_, RowToColumnarExec(child), _, _,
CometNativeShuffle, _)
+ if CometShuffleExchangeExec.columnarShuffleSupports(s, child) =>
Review Comment:
When `columnarShuffleSupports` is false this arm does not match. The native
exchange keeps `RowToColumnarExec` as its child, and every task fails, now with
the new `IllegalStateException`. That case is not rare.
`columnarShuffleFailureReasons` rejects `spark.comet.shuffle.mode=native` up
front, and under the Celeborn manager a Comet exchange only exists in `native`
mode, so the guard is false for every `native`-mode plan and every Celeborn
plan. Could the declined case swap the transition for
`CometSparkToColumnarExec(child)` instead, when
`CometSparkToColumnarExec.isSchemaSupported` accepts the schema? That is how
`RevertNativeForTransitionHeavyStages.revertStageIfNeeded` bridges a reverted
stage under a native shuffle (`outputArrow`), and it keeps native shuffle. The
JVM columnar shuffle can stay the first choice, since it avoids the string-key
placement difference that `hashesDifferentlyFromSpark` describes. I have not
checked whether the Arrow stream input works under Celeborn
, so that case may need to stay as it is.
##########
spark/src/main/scala/org/apache/comet/rules/EliminateRedundantTransitions.scala:
##########
@@ -176,6 +176,15 @@ case class EliminateRedundantTransitions(session:
SparkSession)
CometColumnarShuffle,
_) =>
s.withNewChildren(Seq(child))
+
+ // Spark also adds `RowToColumnar` even when comet chose native shuffle.
+ // Native shuffle reads only Comet vectors, not Spark's vectors.
+ // Switch to row and drop the transition.
+ case s @ CometShuffleExchangeExec(_, RowToColumnarExec(child), _, _,
CometNativeShuffle, _)
+ if CometShuffleExchangeExec.columnarShuffleSupports(s, child) =>
+ val columnar = s.copy(child = child, shuffleType =
CometColumnarShuffle)
+ columnar.copyTagsFrom(s)
+ columnar
Review Comment:
The rewrite turns `CometExchange` into `CometColumnarExchange` without
saying why. Could this tag the new node with `withInfo(columnar, ...)`, saying
the child became row-based after native shuffle was chosen? The verbose
extended explain prints that as `[COMET-INFO: ...]`. Today the only plan that
reaches this arm comes from EMR's nested AQE plan, so the explain output is
where someone debugging it will look first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]