d-wang-commit opened a new pull request, #18847: URL: https://github.com/apache/iotdb/pull/18847
## Description ### Recover from stale ConfigNode candidates A pooled ConfigNode client created during startup can retain a seed-only address list after its DataNode has learned the full ConfigNode list. If the former leader remains reachable over TCP but returns `REDIRECTION_RECOMMEND` without a redirect address, reconnecting to that node succeeds and retries can remain confined to the stale list instead of reaching the elected majority. Refresh candidates from the local `ConfigNodeInfo` before fallback selection, then continue after the current endpoint in the refreshed list. This also handles reordered lists and removal of the current endpoint. Preserve known candidates when the shared list is empty, and refresh the candidate count before deciding whether to wait after a retry round; a stale singleton list must not trigger an unnecessary full-round wait. Re-enable all nine cases in `IoTDBNodeStatusFailoverIT` by reverting the temporary skip from 454119254263eecd8dc97fa708ee009820a8b6b5. The original test scenarios and timeouts are preserved. ### Tests - Added `AbstractConfigNodeClientTest`: 12 unit tests covering stale and reordered candidates, repeated redirects without a target, explicit redirects, removed and empty candidates, RPC and connection failures, bounded connection attempts, successful RPCs, and leader warm-up. - Added `AbstractConfigNodeClientConnectionTest`: two unit tests covering pool reuse and closure, plus three concurrent reconnect rounds with 100 clients using real loopback TCP connections. These are client lifecycle tests, not a 100-DataNode cluster test. - Restored the existing `IoTDBNodeStatusFailoverIT`; no new IT class was added. - Ran all 14 unit tests and all nine IT cases in one successful Maven reactor build on Windows with JDK 17: zero failures, errors, or skipped tests. Spotless and `git diff --check` passed. - An additional diagnostic run of `testLiveOldLeaderPartitionAndQuorumRecovery` passed. After receiving a redirect without a target, the stale seed-only client refreshed to three candidates and reached the majority leader on its next attempt, before the test healed the network partition. Temporary instrumentation is not included in this PR. ### Side effects and risks Failure paths now perform additional local list copies, endpoint lookup, and read-lock acquisition, with work proportional to the number of ConfigNodes. Updating the shared list can contend with these reads. Refreshing the candidate count can make retries more prompt by avoiding premature backoff. Successful RPCs keep their existing path; no discovery RPC, additional connection pool, protocol change, or configuration change is introduced. An empty refresh retains previously known addresses, which may themselves be stale. Existing retry limits, explicit redirect handling, and known-leader warm-up behavior are preserved; this change does not guarantee recovery within a fixed timeout. <hr> This PR has: - [x] been self-reviewed. - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths. - [x] been tested in a test IoTDB cluster. <hr> ##### Key changed/added classes (or packages if there are too many classes) in this PR - `AbstractConfigNodeClient`: refresh stale candidates and preserve retry progress across list updates. - `AbstractConfigNodeClientTest`: verify candidate selection, retry behavior, and preserved success/redirect paths. - `AbstractConfigNodeClientConnectionTest`: verify connection reuse and release during concurrent reconnects. - `IoTDBNodeStatusFailoverIT`: restore all nine temporarily skipped integration tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
