Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/24694 )
Change subject: KUDU-3791 add metrics for auto-leader-rebalancer ...................................................................... Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/24694/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24694/1//COMMIT_MSG@13 PS1, Line 13: - auto_leader_rebalancer_moves_scheduled : - auto_leader_rebalancer_moves_failed : - auto_leader_rebalancer_rounds_completed : - auto_leader_rebalancer_global_pass_skipped If building an automated monitoring/alerting system based on this set of metrics, is there a way to put together a simple recipe to spot when the leader rebalancing in a multi-master cluster falters up to the point when it needs an operator's attention? A very basic ask is to provide a warning flag that isn't prone to false positives and is easy to reason about given multiple master instances. I guess it might be something like 'every run of the rebalancing task yields an increment of auto_leader_rebalancer_rounds_completed for a leader master'. There is 'auto_leader_rebalancer_rounds_completed' metric, but it's a bit cumbersome to reason about not knowing (a) whether automatic leader rebalancing is enabled (b) the rebalancing task interval, and whether there are some delays because of synchronous RPCs (c) the metric is sourced from multiple masters. Maybe, it makes sense to add one such metric of 'warning' severity to allow for simpler and more robust alerting rules? What do you think? http://gerrit.cloudera.org:8080/#/c/24694/1//COMMIT_MSG@18 PS1, Line 18: mirror the counters AutoRebalancerTask already : exposes Yeah, and that is as confusing to me as this one. I'd expect to see information on both the scheduled and the completed moves, so there would be an invariant like moves_scheduled = moves_completed + moves_failed That would allow to reason about the progress of the planned moves. http://gerrit.cloudera.org:8080/#/c/24694/1/src/kudu/master/auto_leader_rebalancer.cc File src/kudu/master/auto_leader_rebalancer.cc: http://gerrit.cloudera.org:8080/#/c/24694/1/src/kudu/master/auto_leader_rebalancer.cc@457 PS1, Line 457: an async : // refactor What's 'async refactor'? http://gerrit.cloudera.org:8080/#/c/24694/1/src/kudu/master/auto_leader_rebalancer.cc@479 PS1, Line 479: continue; It seems we are incrementing the counter of failed moves at line 490 when we cannot resolve server name into IP address, but we don't care incrementing the metric if we cannot find tablet server by its UUID? I think it would be nice to add a comment to explain why we ignore this condition. http://gerrit.cloudera.org:8080/#/c/24694/1/src/kudu/master/auto_leader_rebalancer.cc@905 PS1, Line 905: WARN_NOT_OK(RunLeaderRebalancer(), : Substitute("the master instance isn't leader")); IIUC, this is misleading: RunLeaderRebalancer() might return a non-OK status not only when this instance of catalog manager isn't a leader. If we are introducing metrics here, probably it's a good venue to verify that the particular status code is what's expected from a non-leader catalog manager. If that's something else, it's a good idea to increment an umbrella-type metric of 'warning' severity that would mean "leader rebalancing task is failing". That could be used for a very basic monitoring of whether leader rebalancer has issues. -- To view, visit http://gerrit.cloudera.org:8080/24694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I49b86e59043a7f652d4f556b94582bb74c5698ad Gerrit-Change-Number: 24694 Gerrit-PatchSet: 1 Gerrit-Owner: Gabriella Lotz <[email protected]> Gerrit-Reviewer: Abhishek Chennaka <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Gabriella Lotz <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Martonka <[email protected]> Gerrit-Comment-Date: Thu, 03 Sep 2026 02:28:41 +0000 Gerrit-HasComments: Yes
