haziqishere opened a new issue, #7703:
URL: https://github.com/apache/datafusion-comet/issues/7703

   ### Describe the bug
   
   TL;DR: I got ClassClassException during shuffle stage. This is because 
CometNativeShuffle expects type CometVector but receive Spark's 
`OffHeapColumnVector` instead. This is caused by Spark inserted RowToColumnar. 
This behaviour is seen on Amazon EMR, where AQE wraps a reused exchange's child 
in a nested AdaptivesparkPlanExec that reports row-based output, hence Spark 
adds RowToColumnar.
   
   ### My Observation
   
   A `CometShuffleExchangeExec` with `shuffleType = CometNativeShuffle` whose 
child is Spark's `RowToColumnarExec` fails every task, so the stage aborts:
   
   ```
     java.lang.ClassCastException: class 
org.apache.spark.sql.execution.vectorized.OffHeapColumnVector
       cannot be cast to class org.apache.comet.vector.CometVector
       at 
org.apache.spark.sql.comet.execution.arrow.CometArrowStream$.reconcileStreamSchema(CometNativeArrowSource.scala:164)
       at 
org.apache.spark.sql.comet.execution.arrow.CometArrowStream$.wrapColumnarBatchRDD(CometNativeArrowSource.scala:107)
       at 
org.apache.spark.sql.comet.CometExecRDD$.resolveInputObjects(CometExecRDD.scala:207)
       at 
org.apache.spark.sql.comet.execution.shuffle.CometNativeShuffleInputRDD.compute(CometNativeShuffleInputRDD.scala:120)
       at org.apache.spark.scheduler.ShuffleMapTask.runTask(...)
     ```
   
   Native shuffle reads Arrow-backed `CometVector`s. When the exchange's child 
turns row-based after Comet chose native shuffle, Spark's 
`ApplyColumnarRulesAndInsertTransitions` puts a `RowToColumnarExec` under it. 
`EliminateRedundantTransitions` removes that transition under a JVM columnar 
shuffle, but has no case for native shuffle, so it stays. At 
runtime`CometShuffleExchangeExec` falls back to `child.executeColumnar()`, 
which returns Spark's `OnHeapColumnVector`/`OffHeapColumnVector` batches, and 
`reconcileStreamSchema` casts them to `CometVector` without checking.
   
   The plan at failure:
   
   ```
     CometExchange hashpartitioning(acct, 200), ..., CometNativeShuffle
     +- RowToColumnar
        +- AdaptiveSparkPlan isFinalPlan=true
           +- ResultQueryStage
              +- CometColumnarToRow
                 +- CometHashAggregate [Partial] partial_count(distinct other)
     ```
   
   The same cast issue came up in the review of #6607, reached through 
`transitionRevert` with `maxTransitions=0`. That PR only fixed that path.
   
   ### Steps to reproduce
   
    **On Amazon EMR** (Spark 4.1, Comet 1.2.0), with AQE on: a subtree that 
contains a `countDistinct` and is used twice in the same query, so Spark reuses 
its exchange. A minimal shape:
   
     ```python
     from pyspark.sql import functions as F
   
     spark.conf.set("spark.sql.adaptive.enabled", "true")
     spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")
     spark.conf.set("spark.sql.adaptive.autoBroadcastJoinThreshold", "-1")
   
     spark.range(100000) \
         .select((F.col("id") % 1000).alias("k"), (F.col("id") % 
37).alias("v")) \
         .write.mode("overwrite").parquet("/tmp/comet_r2c")
     df = spark.read.parquet("/tmp/comet_r2c")
   
     agg = df.groupBy("k").agg(F.countDistinct("v").alias("n"))  # two shuffles 
deep
     agg.alias("a").join(agg.alias("b"), "k").count()            # same subtree 
twice -> reused exchange
     ```
   
   EMR's AQE wraps the reused exchange's child in a nested 
`AdaptiveSparkPlanExec`. That nested plan reports row-based output by default 
(`spark.sql.adaptive.nested.propagateColumnarSupport=false`), so Spark inserts 
`RowToColumnar` under the native exchange. The query passes with AQE off, 
without the reuse (`hubs` used
   once), or with `count` in place of `countDistinct`.
   
   **On Apache Spark** I haven't found a plan that reaches it: the same query 
builds the native exchange and the reused exchange, but never the nested AQE 
plan. Building the plan by hand reproduces the same exception on Apache Spark 
4.1:
   
     ```scala
     val rowChild = spark.sql("SELECT id, CAST(id AS STRING) AS s FROM 
range(10)")
       .queryExecution.executedPlan  // any row-based plan, with Comet disabled
     val partitioning = HashPartitioning(rowChild.output.take(1), 4)
     CometShuffleExchangeExec(
       partitioning,
       RowToColumnarExec(rowChild),
       ShuffleExchangeExec(partitioning, rowChild),
       shuffleType = CometNativeShuffle).executeColumnar().count()
     ```
   
   ### Expected behavior
   
   The exchange runs. The JVM columnar shuffle reads rows, so Comet can drop 
the `RowToColumnar` and use it instead, the same way #6607 handles its trigger. 
When that isn't possible, the error should name the vector class and the cause 
instead of a bare `ClassCastException`.
   
   ### Additional context
   
   - My setup: Comet 1.2.0 on EMR (Spark 4.1). The code path is unchanged on 
`main`.
   - Workarounds: `spark.comet.shuffle.mode=jvm`, or on EMR 
`spark.sql.adaptive.nested.propagateColumnarSupport=true`, which keeps native 
shuffle.
   
   ### Willingness to contribute
   
   I would be willing to contribute a fix for this bug with guidance from the 
DataFusion community


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to