lfdversluis opened a new issue, #2520:
URL: https://github.com/apache/datafusion-comet/issues/2520
### Describe the bug
Following up from #2504, I compiled comet in CentOS 7 to build it against
glibc 2.17. Running a simple query, tasks are getting lost due to some metrics
function not being there.
```
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in
stage 1.0 failed 20 times, most recent failure: Lost task 1.19 in stage 1.0
(TID 274) (<redacted> executor 0): java.lang.NoSuchMethodError:
'java.lang.Object
org.apache.spark.executor.TaskMetrics.withExternalAccums(scala.Function1)'
at
org.apache.spark.sql.comet.shims.ShimTaskMetrics$.getTaskAccumulator(ShimTaskMetrics.scala:30)
at
org.apache.spark.sql.comet.shims.ShimTaskMetrics.getTaskAccumulator(ShimTaskMetrics.scala)
at org.apache.comet.parquet.BatchReader.init(BatchReader.java:356)
at
org.apache.comet.parquet.CometParquetFileFormat.$anonfun$buildReaderWithPartitionValues$1(CometParquetFileFormat.scala:203)
at
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:217)
at
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:279)
at
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:129)
at
org.apache.spark.sql.comet.CometScanExec$$anon$1.hasNext(CometScanExec.scala:271)
at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:576)
at scala.collection.mutable.Growable.addAll(Growable.scala:61)
at scala.collection.mutable.Growable.addAll$(Growable.scala:57)
at scala.collection.mutable.ArrayBuilder.addAll(ArrayBuilder.scala:66)
at scala.collection.IterableOnceOps.toArray(IterableOnce.scala:1282)
at scala.collection.IterableOnceOps.toArray$(IterableOnce.scala:1276)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1293)
at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1049)
at
org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2438)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
```
### Steps to reproduce
Compile comet using glib 2.17 (via CentOS 7) using 0.10.0 and scala 2.13
with openjdk 11. Put the jar on the driver and executors, run a query.
### Expected behavior
Query runs fine
### Additional context
We are running Spark 3.5.1 using JDK 17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
