xBis7 commented on PR #4362: URL: https://github.com/apache/ozone/pull/4362#issuecomment-1495948016
@adoroszlai The latest changes fix all the issues that I encountered while looking into this ticket. To sum everything up, `testOMHAMetrics` was failing most of the time while waiting for a new leader to be elected. This was due to the leader not being elected on time or because there weren’t enough pipelines allocated and while waiting for that we were encountering a time out. By moving the test to a new class the timeout issue was resolved. There is no interference and every time we start with 3 available OMs. Three latest workflows, running `TestOzoneManagerHAWithData` on repeat, without `testOMHAMetrics` under it. All passing. https://github.com/xBis7/ozone/actions/runs/4596538780 https://github.com/xBis7/ozone/actions/runs/4596542019 https://github.com/xBis7/ozone/actions/runs/4596958030 After resolving the timeout issue, another error was uncovered. For 30% of the repetitions, during a leader change, the metrics weren’t getting updated. The old leader was now a follower but its metrics were still registered with the old state. To fix this, `void notifyLeaderChanged()` from `StateMachine.EventApi` was overridden and used to initialize the metrics. This method gets called every time there is a leader change unlike `updatePeerList()` and it also wasn’t used in `OzoneManagerStateMachine`. Two latest workflows, running the new class `TestOzoneManagerHAMetrics` on repeat. All passing. https://github.com/xBis7/ozone/actions/runs/4607890850 https://github.com/xBis7/ozone/actions/runs/4607671480 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
