zaynt4606 opened a new pull request, #3881: URL: https://github.com/apache/celeborn/pull/3881
### What changes were proposed in this pull request? Use `allocatedWorkers.containsKey(workerId.toUniqueId)` when checking whether an unknown worker belongs to a shuffle. Add focused tests for an allocated unknown worker, an unrelated unknown worker, disabled detection, enabled replication, a different failure status, and a shuffle without allocated workers. JIRA: https://issues.apache.org/jira/browse/CELEBORN-2489 ### Why are the changes needed? `allocatedWorkers` is a `ConcurrentHashMap[String, ShufflePartitionLocationInfo]`. Its legacy `contains` method checks values, while the lookup argument is a `WorkerInfo`. The condition therefore remains false even when the lost worker is allocated to the shuffle. With `celeborn.client.shuffleDataLostOnUnknownWorker.enabled=true` and replication disabled, the client should report shuffle data loss as soon as an allocated worker becomes `WORKER_UNKNOWN`. The current check misses this and waits for a later failure path. This fix restores the existing early failure behavior and preserves the configuration and replication guards. ### Does this PR resolve a correctness bug? - [x] Yes ### Does this PR introduce _any_ user-facing change? - [x] Yes Restores the configured early failure behavior for a lost allocated worker. ### How was this patch tested? - Before the fix, the new suite runs six tests: the allocated unknown-worker case fails and the other five pass. - After the fix, 19 tests pass across `ReducePartitionCommitHandlerSuite`, `WorkerStatusTrackerSuite`, and `MapPartitionCommitHandlerTest` with Java 17 / Scala 2.12. - `spotless:check` passes for the client and its upstream modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
