SteNicholas commented on issue #3725: URL: https://github.com/apache/celeborn/issues/3725#issuecomment-4677461514
`SHUFFLE_DATA_LOST` returned by `GetReducerFileGroup` means the LifecycleManager failed to commit the shuffle files on the workers when the stage ended (the shuffle is marked data-lost in `ReducePartitionCommitHandler` once `CommitFiles` fails). With S3-only storage that almost always means the workers failed to flush/commit data to S3 — the real error is in the **worker logs**, so please attach the worker-side exceptions around stage end (search for `CommitFiles`, `Failed to upload part`, `ClassNotFoundException`, or S3A connectivity errors). > Can this problem be solved by configuration? It depends on which of the following is the actual cause. Given your setup, please check these in order: 1. **Does your image actually contain the S3 jars?** The official Celeborn binary release (and the Docker image built from it) is compiled **without** the `aws` profile, so `hadoop-aws`, `aws-java-sdk-s3`/`aws-java-sdk-sts` and `celeborn-multipart-uploader-s3` are not bundled. Workers write S3 shuffle data through `S3MultipartUploadHandler`, which is loaded reflectively, so if the jars are missing every flush fails and the shuffle is reported as lost. This one is **not** solvable by configuration — you need to build the distribution with the AWS profile, e.g. `./build/make-distribution.sh -Paws`, and bake your own image (or add those jars to `$CELEBORN_HOME/jars` of the workers). You can confirm quickly with `ls $CELEBORN_HOME/jars | grep -E 'aws|multipart'` inside a worker pod. 2. **`celeborn.hadoop.fs.s3a.endpoint = http://127.0.0.1:3000`** — this address is resolved *inside each worker pod*, where `127.0.0.1` is the pod's own loopback. Unless your S3-compatible service runs as a sidecar in every worker pod, workers cannot reach it and all flushes fail. Use the in-cluster service DNS name instead (e.g. `http://<svc>.<namespace>.svc.cluster.local:3000`). Also note that Spark executors read S3-resident shuffle data **directly from S3** (`DfsPartitionReader`), so the same settings must be present on the Spark side as `spark.celeborn.hadoop.fs.s3a.*` (endpoint, keys, path-style access), and the endpoint must be reachable from the executor pods as well. 3. **Your S3-compatible backend must support multipart upload** (`InitiateMultipartUpload` / `UploadPart` / `ListParts` / `CompleteMultipartUpload`). Celeborn writes shuffle files to S3 exclusively via MPU with the AWS SDK; MinIO supports this, but some lightweight S3 mocks don't. Which service is listening on port 3000? If the failure persists after checking 1 and 2, please share the worker logs around the commit time and we can dig further. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
