HuFather commented on issue #3725: URL: https://github.com/apache/celeborn/issues/3725#issuecomment-4686655773
@SteNicholas Thank you for your detailed analysis. Here is my response: 1. yes, build with -Paws and minio indeed have the shuffle files, shuffle directory and files created 2. http://127.0.0.1:3000 is a sample ip and port,every executor can reacheable the endpoint 3. s3 serverice is minio 4. "celeborn.worker.flusher.s3.buffer.size" is not explicitly set, so the default value is used. The worker contains a large number of logs like the following: 26/06/12 00:11:52,103 ERROR [worker-files-committer-306] S3MultipartUploadHandler: bucket geoscene key spark-tmp/celeborn-worker/shuffle_data/spark-4d18ef12bb0d4f78bd6db58cec7cd4cd/8/985-0-0 uploadId YTBmNDIwYWYtNWMwZi00MmZlLWIzYTEtMDU5MzExM2I4MzBjLmRiZTlkMjUzLWJhMzYtNDdjYy1hNGZlLTg1NDFjZjllZTgyNngxNzgxMTk0MTU1ODQ0MzI3NTIy upload failed to complete, will not retry com.amazonaws.services.s3.model.AmazonS3Exception: Your proposed upload is smaller than the minimum allowed object size. (Service: Amazon S3; Status Code: 400; Error Code: EntityTooSmall; Request ID: 18B812CA27128F78; S3 Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8; Proxy: null), S3 Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5520) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5467) at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:3671) at org.apache.celeborn.S3MultipartUploadHandler.complete(S3MultipartUploadHandler.java:207) at org.apache.celeborn.service.deploy.worker.storage.DfsTierWriter.closeStreams(TierWriter.scala:721) at org.apache.celeborn.service.deploy.worker.storage.TierWriterBase.close(TierWriter.scala:114) at org.apache.celeborn.service.deploy.worker.storage.PartitionDataWriter.close(PartitionDataWriter.java:219) at org.apache.celeborn.service.deploy.worker.Controller$$anon$2.run(Controller.scala:405) at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
