HuFather opened a new issue, #3725:
URL: https://github.com/apache/celeborn/issues/3725
### What is the bug(with logs or screenshots)?
Deployed via Helm. Worker configuration:
celeborn.storage.availableTypes: S3
celeborn.storage.s3.dir: s3://test
celeborn.storage.s3.endpoint.region: us-east-1
celeborn.hadoop.fs.s3a.impl: org.apache.hadoop.fs.s3a.S3AFileSystem
celeborn.hadoop.fs.s3a.endpoint: http://127.0.0.1:3000
celeborn.hadoop.fs.s3a.access.key: test
celeborn.hadoop.fs.s3a.secret.key: test
celeborn.hadoop.fs.s3a.path.style.access: true
celeborn.hadoop.fs.s3a.connection.ssl.enabled: false
celeborn.hadoop.fs.s3a.aws.credentials.provider:
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
celeborn.worker.storage.storagePolicy.createFilePolicy: S3
Error:
org.apache.spark.shuffle.FetchFailedException: Celeborn FetchFailure
appShuffleId/shuffleId: 3/8
at
org.apache.spark.shuffle.celeborn.CelebornShuffleReader.handleFetchExceptions(CelebornShuffleReader.scala:590)
at
org.apache.spark.shuffle.celeborn.CelebornShuffleReader.checkAndReportFetchFailureForUpdateFileGroupFailure(CelebornShuffleReader.scala:570)
at
org.apache.spark.shuffle.celeborn.CelebornShuffleReader.read(CelebornShuffleReader.scala:184)
at
org.apache.spark.rdd.CoGroupedRDD.$anonfun$compute$2(CoGroupedRDD.scala:148)
at
scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
at scala.collection.immutable.List.foreach(List.scala:431)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:136)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at
com.esri.arcgis.st.spark.GenericFeatureSchemaRDD.compute(GenericFeatureSchemaRDD.scala:25)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54)
at
org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:621)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at
org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:624)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.celeborn.common.exception.CelebornIOException: Failed
to load file group of shuffle 8 partition 11! Request
GetReducerFileGroup(8,false,V1) return SHUFFLE_DATA_LOST for 8.
at
org.apache.celeborn.client.ShuffleClientImpl.updateFileGroup(ShuffleClientImpl.java:1954)
at
org.apache.celeborn.client.ShuffleClientImpl.updateFileGroup(ShuffleClientImpl.java:1919)
at
org.apache.spark.shuffle.celeborn.CelebornShuffleReader.read(CelebornShuffleReader.scala:170)
... 33 more
### How to reproduce the bug?
Steps to reproduce the bug.
/cc @who-need-to-know
/assign @who-can-solve-this-bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
