sunchao opened a new pull request, #3687: URL: https://github.com/apache/celeborn/pull/3687
## Why are the changes needed? `ShuffleClientImpl.updateFileGroup` currently performs the blocking `GetReducerFileGroup` RPC from inside `ConcurrentHashMap.compute`. When several reducer tasks touch the same shuffle while that RPC is slow, one task waits in the RPC while peer task threads block on the map reservation node. This turns a single slow file-group lookup into a local executor-side convoy. ## What changes were proposed in this PR? - Replace the blocking `compute` path with a per-shuffle single-flight load using an in-flight `CompletableFuture`. - Keep the existing successful reducer-file-group cache behavior, while ensuring concurrent callers share the same RPC result instead of issuing duplicate lookups. - Clear the in-flight slot during shuffle cleanup and preserve cleanup ordering so a concurrent cleanup cannot republish stale cached state. - Add a regression test that starts two concurrent same-shuffle loads, holds the mocked RPC open, and verifies only one RPC is issued while both callers receive the same loaded result. ## How was this PR tested? - `build/mvn --no-transfer-progress -pl client -Dtest=ShuffleClientSuiteJ test` - `build/mvn --no-transfer-progress -DskipTests spotless:apply` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
