d-wang-commit opened a new pull request, #19118:
URL: https://github.com/apache/iotdb/pull/19118

   ## Description
   
   ### Retry rollback cleanup while DataNodes restart
   
   After a whole-cluster restart during region migration, AddRegionPeer 
rollback can reach the target DataNode before its internal RPC service is 
ready. All six immediate RPC attempts can fail, leaving the target consensus 
group directory behind. This occurred in 
[IoTDBRegionMigrateClusterCrashIoTV2StreamIT#clusterCrash2](https://github.com/apache/iotdb/actions/runs/41699501879/job/126138752896).
   
   Retry incomplete or unsuccessful peer-list reset batches up to three 
additional times, waiting five seconds between batches. Stop when all requested 
nodes return success, or stop waiting and preserve the interrupt flag when 
interrupted. Keep the existing rollback flow and completion behavior.
   
   ### Tests
   
   - UT: Added four cases to `AddRegionPeerProcedureTest` for delayed success, 
retry exhaustion, immediate success and interruption. All five tests, including 
the existing serialization test, passed.
   - IT: No integration test changes. The existing 
`IoTDBRegionMigrateClusterCrashIoTV2StreamIT#clusterCrash2` passed with a full 
3-ConfigNode/3-DataNode restart. Reset succeeded in the first batch in this 
run; delayed retries are covered by the unit tests.
   - Spotless checks passed.
   
   ### Side effects and risks
   
   Failed rollback cleanup can occupy a procedure worker for up to 15 
additional seconds of waiting, plus RPC time. Retries repeat the idempotent 
reset on all related nodes. Cleanup remains best effort if retries are 
exhausted; procedure states and serialization are unchanged.
   
   <!--
   In each section, please describe design decisions made, including:
    - Choice of algorithms
    - Behavioral aspects. What configuration values are acceptable? How are 
corner cases and error 
       conditions handled, such as when there are insufficient resources?
    - Class organization and design (how the logic is split between classes, 
inheritance, composition, 
       design patterns)
    - Method organization and design (how the logic is split between methods, 
parameters and return types)
    - Naming (class, method, API, configuration, HTTP endpoint, names of 
emitted metrics)
   -->
   
   
   <!-- It's good to describe an alternative design (or mention an alternative 
name) for every design 
   (or naming) decision point and compare the alternatives with the designs 
that you've implemented 
   (or the names you've chosen) to highlight the advantages of the chosen 
designs and names. -->
   
   <!-- If there was a discussion of the design of the feature implemented in 
this PR elsewhere 
   (e. g. a "Proposal" issue, any other issue, or a thread in the development 
mailing list), 
   link to that discussion from this PR description and explain what have 
changed in your final design 
   compared to your original proposal or the consensus version in the end of 
the discussion. 
   If something hasn't changed since the original discussion, you can omit a 
detailed discussion of 
   those aspects of the design here, perhaps apart from brief mentioning for 
the sake of readability 
   of this PR description. -->
   
   <!-- Some of the aspects mentioned above may be omitted for simple and small 
changes. -->
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added unit tests or modified existing tests to cover new code paths.
   - [x] been tested in a test IoTDB cluster.
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not 
all of these items 
   apply to every PR. Remove the items which are not done or not relevant to 
the PR. None of the items 
   from the checklist above are strictly necessary, but it would be very 
helpful if you at least 
   self-review the PR. -->
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - `AddRegionPeerProcedure`: add bounded delayed retries around rollback 
peer-list reset RPCs.
   - `AddRegionPeerProcedureTest`: cover retry timing, attempt limits, early 
success and interruption.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to