sunchao opened a new pull request, #3744: URL: https://github.com/apache/celeborn/pull/3744
## Why are the changes needed? When a Celeborn client loses its cached master connection, the associated Outbox can be stopped before queued or late master RPC messages are sent. Those requests fail with `CelebornException: Message is dropped because Outbox is stopped`. `MasterClient` currently retries I/O and RPC timeout failures, but it does not recognize this stopped-Outbox failure when it is wrapped by the normal `awaitResult` path. In HA mode, a request can therefore fail instead of clearing the stale `RpcEndpointRef` and reconnecting to another available master. The same legacy message is also used when the local `RpcEnv` shuts down permanently. Treating every occurrence as retryable without distinguishing shutdown would cause futile reconnect attempts through an already-stopped environment. JIRA: https://issues.apache.org/jira/browse/CELEBORN-2369 ## What changes were proposed in this PR? - Recognize the exact legacy stopped-Outbox failure through its cause chain and reset the cached master endpoint so the existing HA retry path can reconnect. - Use `RpcEnvStoppedException` for terminal local shutdown so shutdown failures remain non-retryable. - Preserve each Outbox's original stop cause for queued and late messages. - Fail messages submitted after `RpcEnv` shutdown immediately with the terminal cause. - Add focused coverage for HA reconnection, terminal no-reconnect behavior, and transient versus terminal Outbox causes. ### Does this PR resolve a correctness bug? - [x] Yes ### Does this PR introduce _any_ user-facing change? - [x] Yes. Master RPCs can now fail over after a transient stopped-Outbox failure instead of failing immediately. No public API or configuration changes are introduced. ## How was this PR tested? Using JDK 17: - `build/mvn --no-transfer-progress -DskipTests spotless:apply` - `build/mvn --no-transfer-progress -pl common -DskipTests test-compile` - `build/mvn --no-transfer-progress -pl common -DargLine= -Dtest=org.apache.celeborn.common.client.MasterClientSuiteJ surefire:test` — 17 tests passed. - `build/mvn --no-transfer-progress -pl common -DargLine= -DwildcardSuites=org.apache.celeborn.common.rpc.netty.OutboxSuite scalatest:test` — 3 tests passed. - `build/mvn --no-transfer-progress -pl common -DargLine= -DwildcardSuites=org.apache.celeborn.common.rpc.netty.NettyRpcEnvSuite scalatest:test` — 33 tests passed. - `build/mvn --no-transfer-progress -DskipTests spotless:check` — all 10 reactor modules passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
