sunchao opened a new pull request, #3692: URL: https://github.com/apache/celeborn/pull/3692
## Why are the changes needed? `CelebornShuffleReader` creates data clients serially for each distinct worker before the existing parallel `BATCH_OPEN_STREAM` phase. If one or more worker endpoints are slow or unavailable, a reducer can pay those connection retry costs worker-by-worker before it even starts opening streams. ## What changes were proposed in this PR? - Group batch-open-stream locations by worker before creating data clients. - Create one data client per distinct worker in parallel on the existing stream creator pool. - Keep failed-worker exclusion behavior, then build batch open stream requests only for workers whose client creation succeeded. - Add focused reader tests that verify client creation starts in parallel and that one failed worker does not prevent healthy workers from continuing. ## How was this PR tested? - `build/mvn -Pgoogle-mirror,spark-3.3 -pl client-spark/spark-3 spotless:apply` - `build/mvn -Pgoogle-mirror,spark-3.5 -pl client-spark/spark-3 -am -DskipTests install` - `build/mvn -Pgoogle-mirror,spark-3.5 -pl client-spark/spark-3 -Dtest=CelebornShuffleReaderSuite test` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
