d-wang-commit opened a new pull request, #18847:
URL: https://github.com/apache/iotdb/pull/18847

   ## Description
   
   ### Recover from stale ConfigNode candidates
   
   A pooled ConfigNode client created during startup can retain a seed-only 
address list after its DataNode has learned the full ConfigNode list. If the 
former leader remains reachable over TCP but returns `REDIRECTION_RECOMMEND` 
without a redirect address, reconnecting to that node succeeds and retries can 
remain confined to the stale list instead of reaching the elected majority.
   
   Refresh candidates from the local `ConfigNodeInfo` before fallback 
selection, then continue after the current endpoint in the refreshed list. This 
also handles reordered lists and removal of the current endpoint. Preserve 
known candidates when the shared list is empty, and refresh the candidate count 
before deciding whether to wait after a retry round; a stale singleton list 
must not trigger an unnecessary full-round wait.
   
   Re-enable all nine cases in `IoTDBNodeStatusFailoverIT` by reverting the 
temporary skip from 454119254263eecd8dc97fa708ee009820a8b6b5. The original test 
scenarios and timeouts are preserved.
   
   ### Tests
   
   - Added `AbstractConfigNodeClientTest`: 12 unit tests covering stale and 
reordered candidates, repeated redirects without a target, explicit redirects, 
removed and empty candidates, RPC and connection failures, bounded connection 
attempts, successful RPCs, and leader warm-up.
   - Added `AbstractConfigNodeClientConnectionTest`: two unit tests covering 
pool reuse and closure, plus three concurrent reconnect rounds with 100 clients 
using real loopback TCP connections. These are client lifecycle tests, not a 
100-DataNode cluster test.
   - Restored the existing `IoTDBNodeStatusFailoverIT`; no new IT class was 
added.
   - Ran all 14 unit tests and all nine IT cases in one successful Maven 
reactor build on Windows with JDK 17: zero failures, errors, or skipped tests. 
Spotless and `git diff --check` passed.
   - An additional diagnostic run of 
`testLiveOldLeaderPartitionAndQuorumRecovery` passed. After receiving a 
redirect without a target, the stale seed-only client refreshed to three 
candidates and reached the majority leader on its next attempt, before the test 
healed the network partition. Temporary instrumentation is not included in this 
PR.
   
   ### Side effects and risks
   
   Failure paths now perform additional local list copies, endpoint lookup, and 
read-lock acquisition, with work proportional to the number of ConfigNodes. 
Updating the shared list can contend with these reads. Refreshing the candidate 
count can make retries more prompt by avoiding premature backoff. Successful 
RPCs keep their existing path; no discovery RPC, additional connection pool, 
protocol change, or configuration change is introduced.
   
   An empty refresh retains previously known addresses, which may themselves be 
stale. Existing retry limits, explicit redirect handling, and known-leader 
warm-up behavior are preserved; this change does not guarantee recovery within 
a fixed timeout.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths.
   - [x] been tested in a test IoTDB cluster.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - `AbstractConfigNodeClient`: refresh stale candidates and preserve retry 
progress across list updates.
   - `AbstractConfigNodeClientTest`: verify candidate selection, retry 
behavior, and preserved success/redirect paths.
   - `AbstractConfigNodeClientConnectionTest`: verify connection reuse and 
release during concurrent reconnects.
   - `IoTDBNodeStatusFailoverIT`: restore all nine temporarily skipped 
integration tests.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to