haziqishere opened a new issue, #7703:
URL: https://github.com/apache/datafusion-comet/issues/7703
### Describe the bug
TL;DR: I got ClassClassException during shuffle stage. This is because
CometNativeShuffle expects type CometVector but receive Spark's
`OffHeapColumnVector` instead. This is caused by Spark inserted RowToColumnar.
This behaviour is seen on Amazon EMR, where AQE wraps a reused exchange's child
in a nested AdaptivesparkPlanExec that reports row-based output, hence Spark
adds RowToColumnar.
### My Observation
A `CometShuffleExchangeExec` with `shuffleType = CometNativeShuffle` whose
child is Spark's `RowToColumnarExec` fails every task, so the stage aborts:
```
java.lang.ClassCastException: class
org.apache.spark.sql.execution.vectorized.OffHeapColumnVector
cannot be cast to class org.apache.comet.vector.CometVector
at
org.apache.spark.sql.comet.execution.arrow.CometArrowStream$.reconcileStreamSchema(CometNativeArrowSource.scala:164)
at
org.apache.spark.sql.comet.execution.arrow.CometArrowStream$.wrapColumnarBatchRDD(CometNativeArrowSource.scala:107)
at
org.apache.spark.sql.comet.CometExecRDD$.resolveInputObjects(CometExecRDD.scala:207)
at
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleInputRDD.compute(CometNativeShuffleInputRDD.scala:120)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(...)
```
Native shuffle reads Arrow-backed `CometVector`s. When the exchange's child
turns row-based after Comet chose native shuffle, Spark's
`ApplyColumnarRulesAndInsertTransitions` puts a `RowToColumnarExec` under it.
`EliminateRedundantTransitions` removes that transition under a JVM columnar
shuffle, but has no case for native shuffle, so it stays. At
runtime`CometShuffleExchangeExec` falls back to `child.executeColumnar()`,
which returns Spark's `OnHeapColumnVector`/`OffHeapColumnVector` batches, and
`reconcileStreamSchema` casts them to `CometVector` without checking.
The plan at failure:
```
CometExchange hashpartitioning(acct, 200), ..., CometNativeShuffle
+- RowToColumnar
+- AdaptiveSparkPlan isFinalPlan=true
+- ResultQueryStage
+- CometColumnarToRow
+- CometHashAggregate [Partial] partial_count(distinct other)
```
The same cast issue came up in the review of #6607, reached through
`transitionRevert` with `maxTransitions=0`. That PR only fixed that path.
### Steps to reproduce
**On Amazon EMR** (Spark 4.1, Comet 1.2.0), with AQE on: a subtree that
contains a `countDistinct` and is used twice in the same query, so Spark reuses
its exchange. A minimal shape:
```python
from pyspark.sql import functions as F
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
spark.conf.set("spark.sql.adaptive.autoBroadcastJoinThreshold", "-1")
spark.range(100000) \
.select((F.col("id") % 1000).alias("k"), (F.col("id") %
37).alias("v")) \
.write.mode("overwrite").parquet("/tmp/comet_r2c")
df = spark.read.parquet("/tmp/comet_r2c")
agg = df.groupBy("k").agg(F.countDistinct("v").alias("n")) # two shuffles
deep
agg.alias("a").join(agg.alias("b"), "k").count() # same subtree
twice -> reused exchange
```
EMR's AQE wraps the reused exchange's child in a nested
`AdaptiveSparkPlanExec`. That nested plan reports row-based output by default
(`spark.sql.adaptive.nested.propagateColumnarSupport=false`), so Spark inserts
`RowToColumnar` under the native exchange. The query passes with AQE off,
without the reuse (`hubs` used
once), or with `count` in place of `countDistinct`.
**On Apache Spark** I haven't found a plan that reaches it: the same query
builds the native exchange and the reused exchange, but never the nested AQE
plan. Building the plan by hand reproduces the same exception on Apache Spark
4.1:
```scala
val rowChild = spark.sql("SELECT id, CAST(id AS STRING) AS s FROM
range(10)")
.queryExecution.executedPlan // any row-based plan, with Comet disabled
val partitioning = HashPartitioning(rowChild.output.take(1), 4)
CometShuffleExchangeExec(
partitioning,
RowToColumnarExec(rowChild),
ShuffleExchangeExec(partitioning, rowChild),
shuffleType = CometNativeShuffle).executeColumnar().count()
```
### Expected behavior
The exchange runs. The JVM columnar shuffle reads rows, so Comet can drop
the `RowToColumnar` and use it instead, the same way #6607 handles its trigger.
When that isn't possible, the error should name the vector class and the cause
instead of a bare `ClassCastException`.
### Additional context
- My setup: Comet 1.2.0 on EMR (Spark 4.1). The code path is unchanged on
`main`.
- Workarounds: `spark.comet.shuffle.mode=jvm`, or on EMR
`spark.sql.adaptive.nested.propagateColumnarSupport=true`, which keeps native
shuffle.
### Willingness to contribute
I would be willing to contribute a fix for this bug with guidance from the
DataFusion community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]