GrantPSpencer opened a new pull request, #3012: URL: https://github.com/apache/helix/pull/3012
### Issues - [ ] My PR addresses the following Helix issues and references them in the PR description: ParticipantDeregistrationStage can remove nodes as they join due to a stale _instanceOfflineTimeMap. This map is currently only updated when there is a LIVEINSTANCE change in the cluster, this can lead entries in the map for instances that have left the cluster. This can cause the following behavior to occur: 1. Node A goes offline at Time T. - _instanceOfflineTimeMap now contains node A --> time T mapping 2. Node A is removed 3. Node attempts to join the cluster after it has been removed by deregistration, Time = T + (deregistrationTimeout + n) 4. Node creates its CONFIGS/PARTICIPANT/* and INSTANCES/* nodes, this triggers an InstanceConfigChange event and pipeline is ran 5. ParticipantDeregistrationStage loops over _instanceOfflineTimeMap to determine which nodes to remove. It sees: node A --> time T while currrent time is T + (deregistrationTimeout + n), so deregisters it. 6. The node then calls createLiveInstance(..), which creates the ephemeral node under LIVEINSTANCES, and calls persistHistory(..), which will create the instance path and a HISTORY child node 7. This causes the node to be in a bad state where it is not properly "set up" (instance path does not have all children, no config znode) so it will not be considered assignable. ### Description - [ ] Here are some details about my PR, including screenshots of any UI changes: This PR includes a change to always clean up stale nodes from the _instanceOfflineTimeMap, by ensuring that the _instanceOfflineTimeMap never has any instances that are also not in the _allInstanceConfigCache. This issue could have been previously caught with more sufficient testing, so I have added a new test case to cover an instance rejoining and successfully regaining assignments `testParticipantDeregisteredAndRejoins` ### Tests - [ ] The following tests are written for this issue: (List the names of added unit/integration tests) - The following is the result of the "mvn test" command on the appropriate module: `TestParticipantDeregistrationStage.testParticipantDeregisteredAndRejoins()` ``` mvn test -o -Dtest=TestParticipantDeregistrationStage -pl=helix-core [INFO] Results: [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:30 min [INFO] Finished at: 2025-03-25T16:29:08-07:00 [INFO] ------------------------------------------------------------------------ ``` ### Commits - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Code Quality - My diff has been formatted using helix-style.xml (helix-style-intellij.xml if IntelliJ IDE is used) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
