wankunde opened a new issue, #9769:
URL: https://github.com/apache/incubator-gluten/issues/9769
### Backend
VL (Velox)
### Bug description
# Expected behavior
```ini
--conf spark.gluten.sql.columnar.shuffle=true \
--conf
spark.shuffle.manager=org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
\
--conf spark.gluten.sql.columnar.shuffle.sort.partitions.threshold=100 \
--conf spark.celeborn.client.spark.shuffle.writer=hash \
```
When I run a query with gluten + celeborn + columnar shuffle + hash writer ,
when the
https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala#L555-L557
return true,
the `useSortBasedShuffle` in ColumnarShuffleExchangeExec will is true while
the isSort in VeloxCelebornColumnarShuffleWriter will be false
# actual behavior
The isSort in VeloxCelebornColumnarShuffleWriter false while the
`useSortBasedShuffle` in ColumnarShuffleExchangeExec is true.
They are not matched.
```log
Lost task 0.0 in stage 2.0 (TID 2) (datanode-4258.host executor 1):
java.util.NoSuchElementException: key not found: splitTime
at
scala.collection.immutable.BitmapIndexedMapNode.apply(HashMap.scala:674)
at
scala.collection.immutable.BitmapIndexedMapNode.apply(HashMap.scala:672)
at scala.collection.immutable.HashMap.apply(HashMap.scala:132)
at
org.apache.spark.shuffle.VeloxCelebornColumnarShuffleWriter.internalWrite(VeloxCelebornColumnarShuffleWriter.scala:104)
at
org.apache.spark.shuffle.CelebornColumnarShuffleWriter.write(CelebornColumnarShuffleWriter.scala:119)
at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:57)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:112)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:171)
at org.apache.spark.scheduler.Task.run(Task.scala:152)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Executor.scala:739)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:102)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:743)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
--conf spark.gluten.sql.columnar.shuffle=true
--conf
spark.shuffle.manager=org.apache.spark.shuffle.gluten.celeborn.CelebornShuffleManager
--conf spark.gluten.sql.columnar.shuffle.sort.partitions.threshold=100
--conf spark.celeborn.client.spark.shuffle.writer=hash
### System information
_No response_
### Relevant logs
```bash
Lost task 0.0 in stage 2.0 (TID 2) (datanode-4258.host executor 1):
java.util.NoSuchElementException: key not found: splitTime
at
scala.collection.immutable.BitmapIndexedMapNode.apply(HashMap.scala:674)
at
scala.collection.immutable.BitmapIndexedMapNode.apply(HashMap.scala:672)
at scala.collection.immutable.HashMap.apply(HashMap.scala:132)
at
org.apache.spark.shuffle.VeloxCelebornColumnarShuffleWriter.internalWrite(VeloxCelebornColumnarShuffleWriter.scala:104)
at
org.apache.spark.shuffle.CelebornColumnarShuffleWriter.write(CelebornColumnarShuffleWriter.scala:119)
at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:57)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:112)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:171)
at org.apache.spark.scheduler.Task.run(Task.scala:152)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$5(Executor.scala:739)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:102)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:743)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
