fanyi-zhao commented on issue #2386:
URL: https://github.com/apache/auron/issues/2386#issuecomment-4953875597
Hi, I have the same issue here. I can reproduce it using this SQL:
```sql
CREATE TABLE IF NOT EXISTS test USING parquet AS
SELECT id, CAST(id % 100 AS int) AS grp FROM range(0, 1000);
SELECT count(*)
FROM test a
JOIN test b ON a.id = b.id
WHERE b.grp = 1;
```
Save it as `test.sql` and then run:
```
$SPARK_HOME/bin/spark-sql \
--jars auron-spark-4.1_2.13-pre-linux-x86_64-8.0.0-incubating.jar \
--conf
spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=1B \
--conf spark.auron.enable=true \
--conf
spark.sql.extensions=org.apache.spark.sql.auron.AuronSparkSessionExtension \
--conf
spark.shuffle.manager=org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager
\
--conf spark.sql.autoBroadcastJoinThreshold=-1 \
-f test.sql
```
Output:
```
...
thread 'auron-native-stage-4-part-19-tid-60' panicked at
native-engine/datafusion-ext-plans/src/common/execution_context.rs:594:21:
output_with_sender[Shuffle] canceled due to task finished/killed
2026-07-12 19:19:53.568 (+0.621s) [WARN] [auron::rt:214] (stage: 4,
partition: 19, tid: 60) - task completed before native execution done
2026-07-12 19:19:53.568 (+0.621s) [WARN] [auron::rt:214] (stage: 4,
partition: 17, tid: 58) - task completed before native execution done
2026-07-12 19:19:53.568 (+0.621s) [INFO] [auron::rt:303] (stage: 0,
partition: 0, tid: 0) - (partition=6) native execution finalized
2026-07-12 19:19:53.568 (+0.621s) [INFO] [auron::rt:290] (stage: 0,
partition: 0, tid: 0) - (partition=18) native execution finalizing
2026-07-12 19:19:53.569 (+0.622s) [INFO] [auron::rt:290] (stage: 0,
partition: 0, tid: 0) - (partition=17) native execution finalizing
2026-07-12 19:19:53.569 (+0.622s) [INFO] [auron::rt:290] (stage: 0,
partition: 0, tid: 0) - (partition=19) native execution finalizing
2026-07-12 19:19:53.570 (+0.623s) [INFO] [auron::rt:303] (stage: 0,
partition: 0, tid: 0) - (partition=17) native execution finalized
2026-07-12 19:19:53.571 (+0.624s) [INFO] [auron::rt:303] (stage: 0,
partition: 0, tid: 0) - (partition=18) native execution finalized
2026-07-12 19:19:53.571 (+0.624s) [INFO] [auron::rt:303] (stage: 0,
partition: 0, tid: 0) - (partition=19) native execution finalized
Job aborted due to stage failure: Task 4 in stage 4.0 failed 1 times, most
recent failure: Lost task 4.0 in stage 4.0 (TID 45) (10.0.0.136 executor
driver): java.lang.RuntimeException: java.lang.RuntimeException: task panics:
Execution error: Execution error: output_with_sender[Shuffle] error: Execution
error: output_with_sender[Filter] error: Execution error:
output_with_sender[Filter]: output() returns error: External error: Java
exception thrown at
native-engine/datafusion-ext-exprs/src/spark_udf_wrapper.rs:97:
java.lang.ClassCastException: cannot assign instance of
scala.collection.generic.DefaultSerializationProxy to field
org.apache.spark.rdd.RDD.dependencies_ of type scala.collection.immutable.Seq
in instance of org.apache.spark.rdd.MapPartitionsRDD
at
org.apache.auron.jni.AuronCallNativeWrapper.checkError(AuronCallNativeWrapper.java:166)
at
org.apache.auron.jni.AuronCallNativeWrapper.close(AuronCallNativeWrapper.java:190)
at
org.apache.auron.jni.AuronCallNativeWrapper.checkError(AuronCallNativeWrapper.java:165)
at
org.apache.auron.jni.AuronCallNativeWrapper.loadNextBatch(AuronCallNativeWrapper.java:123)
at
org.apache.spark.sql.auron.NativeHelper$$anon$1.hasNext(NativeHelper.scala:133)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:31)
at scala.collection.mutable.Growable.addAll(Growable.scala:61)
at scala.collection.mutable.Growable.addAll$(Growable.scala:57)
at scala.collection.mutable.ArrayBuilder.addAll(ArrayBuilder.scala:75)
at scala.collection.IterableOnceOps.toArray(IterableOnce.scala:1528)
at scala.collection.IterableOnceOps.toArray$(IterableOnce.scala:1521)
at
org.apache.spark.util.CompletionIterator.toArray(CompletionIterator.scala:25)
at
org.apache.spark.sql.execution.auron.shuffle.AuronShuffleWriterBase.nativeShuffleWrite(AuronShuffleWriterBase.scala:68)
at
org.apache.spark.sql.execution.auron.plan.NativeShuffleExchangeExec$$anon$1.internalWrite(NativeShuffleExchangeExec.scala:188)
at
org.apache.spark.sql.execution.auron.plan.NativeShuffleExchangeExec$$anon$1.write(NativeShuffleExchangeExec.scala:152)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:111)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:180)
at org.apache.spark.scheduler.Task.run(Task.scala:147)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:873)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:86)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:83)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:97)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:876)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: java.lang.RuntimeException: task panics: Execution error:
Execution error: output_with_sender[Shuffle] error: Execution error:
output_with_sender[Filter] error: Execution error: output_with_sender[Filter]:
output() returns error: External error: Java exception thrown at
native-engine/datafusion-ext-exprs/src/spark_udf_wrapper.rs:97:
java.lang.ClassCastException: cannot assign instance of
scala.collection.generic.DefaultSerializationProxy to field
org.apache.spark.rdd.RDD.dependencies_ of type scala.collection.immutable.Seq
in instance of org.apache.spark.rdd.MapPartitionsRDD
...
```
Full log file:
[output.log](https://github.com/user-attachments/files/29949519/output.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
