d-wang-commit opened a new pull request, #18892:
URL: https://github.com/apache/iotdb/pull/18892

   ## Description
   
   ### Restart during IoTConsensus peer removal
   
   A surviving replica can restart after the removed peer's sync channel has 
been deleted but before ConfigNode commits the updated replica set. The restart 
response still includes the removed peer, so IoTConsensus recreates its 
dispatcher. Previously, completing the metadata update did not correct that 
recovered peer list. Replication to the removed peer kept failing and pinned 
the minimum sync index, leaving syncLag above zero.
   
   ### Reconcile before completing removal
   
   For IoTConsensus V1 DataRegions, commit the replica-set removal first, then 
use the existing resetPeerList RPC to send the final membership to every 
retained replica. The existing DataNode handler waits for consensus startup 
before applying the list. Finish the procedure only after every retained 
replica acknowledges success.
   
   Metadata failures, missing or unsuccessful responses, and exceptions keep 
the existing final state pending and retry after a five-second timeout. Offline 
replicas are included. ProcedureExecutor registers the timeout after persisting 
the suspended state, allowing retries to resume automatically and after 
recovery. No procedure state IDs, serialized fields, DataNode production code, 
or RPC definitions are changed.
   
   ### Tests
   
   - **UT added:** RestartDuringRemovePeerTest covers metadata ordering, failed 
or missing responses, exceptions, recovery, idempotent replay, and protocol 
gating. StateMachineTimeoutTest covers persistence-before-wakeup and executor 
recovery. RemovePeerRestartConsensusTest uses real IoTConsensus instances and 
loopback Thrift RPCs to reproduce the stale dispatcher, verify its removal and 
syncLag convergence, and verify subsequent writes and another restart.
   - **UT run:** All 17 tests passed across the three new classes, 
RemoveRegionPeerProcedureTest, and STMProcedureTest. The same restart 
regression fails with the original removal procedure because no final reset is 
sent; with the fix, syncLag falls from 8 to 0 and remains 0 after further 
writes and restart.
   - **IT:** No full cluster IT was added or run. The consensus regression 
restarts instances in one JVM with a test internal-service facade and an 
in-memory state machine; it does not launch full DataNode processes.
   - **Build checks:** ConfigNode Spotless and the full reactor Chinese-locale 
package build passed. The selected UTs ran with the default English locale.
   
   ### Side effects and risks
   
   - An unavailable retained replica now delays completion and causes repeated 
metadata/RPC attempts. The procedure remains pending instead of silently 
leaving stale membership behind.
   - The serialized procedure layout is unchanged, but ConfigNodes taking over 
a suspended retry must contain this fix; older leaders retain the previous 
generic timeout behavior.
   - This reuses existing DataNode dispatcher cleanup, which can wait for its 
stop timeout. The tests verify removal from membership and syncLag accounting, 
not complete dispatcher-thread termination.
   - Already completed removal procedures are not replayed automatically, so 
previously affected Regions need separate reconciliation. The reconciliation is 
limited to IoTConsensus V1 DataRegions.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious for an unfamiliar reader.
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - RemoveRegionPeerProcedure: retain and retry finalization until membership 
reconciliation succeeds.
   - RegionMaintainHandler: verify metadata removal and reset every retained 
replica's peer list.
   - ProcedureExecutor: schedule suspended procedures after persistence.
   - ProcedureMessages (en/zh): add reconciliation and retry diagnostics.
   - RestartDuringRemovePeerTest, RemovePeerRestartConsensusTest, 
StateMachineTimeoutTest: cover removal recovery and timeout execution.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to