Gabriella Lotz has posted comments on this change. ( http://gerrit.cloudera.org:8080/24694 )
Change subject: KUDU-3791 add metrics for auto-leader-rebalancer ...................................................................... Patch Set 2: (5 comments) http://gerrit.cloudera.org:8080/#/c/24694/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/24694/1//COMMIT_MSG@13 PS1, Line 13: - auto_leader_rebalancer_moves_completed : - auto_leader_rebalancer_moves_failed : - auto_leader_rebalancer_rounds_completed : - auto_leader_rebalancer_global_pass_skipped > If building an automated monitoring/alerting system based on this set of me Good idea. I've added auto_leader_rebalancer_task_errors, incremented in RunLoop only when RunLeaderRebalancer() returns an unexpected non-OK. The "lost leadership mid-round" case is now swallowed. I think a monotonic increase there is a reasonable alerting signal. http://gerrit.cloudera.org:8080/#/c/24694/1//COMMIT_MSG@18 PS1, Line 18: : Naming > Yeah, and that is as confusing to me as this one. I'd expect to see inform I renamed the old moves_scheduled to moves_completed, and added a new moves_scheduled that's incremented once per attempted transfer. The updated RebalancerMetrics test asserts it. http://gerrit.cloudera.org:8080/#/c/24694/1/src/kudu/master/auto_leader_rebalancer.cc File src/kudu/master/auto_leader_rebalancer.cc: http://gerrit.cloudera.org:8080/#/c/24694/1/src/kudu/master/auto_leader_rebalancer.cc@457 PS1, Line 457: : } > What's 'async refactor'? Sorry, that was shorthand for a follow-up I have in mind: the LeaderStepDown RPCs in this loop are issued synchronously, so a round with many transfers can be slow. I've rewritten the TODO to spell that out. http://gerrit.cloudera.org:8080/#/c/24694/1/src/kudu/master/auto_leader_rebalancer.cc@479 PS1, Line 479: (1) << Su > It seems we are incrementing the counter of failed moves at line 490 when w The pre-flight skips are cases where we can't actually run the transfer anymore by the time we get to executing, like the source tserver deregistered or the destination is now in maintenance mode, so we never send the RPC. The line I've drawn is that moves_failed counts actual attempts (RPCs we committed to sending). If we counted the pre-flight skips too, moves_failed would climb during a totally normal maintenance window, which is exactly the kind of false positive that makes a warn severity signal useless. I've added a comment above the skips saying so. http://gerrit.cloudera.org:8080/#/c/24694/1/src/kudu/master/auto_leader_rebalancer.cc@905 PS1, Line 905: } else { : catalog_manager_->GetAllTables(&table_infos); > IIUC, this is misleading: RunLeaderRebalancer() might return a non-OK statu That's right, that log line was misleading. Now the second ScopedLeaderSharedLock in RunLeaderRebalancer now returns OK on non-leader, like the first check, and RunLoop now increments the new auto_leader_rebalancer_task_errors counter on any non-OK, with a clearer log message. -- To view, visit http://gerrit.cloudera.org:8080/24694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I49b86e59043a7f652d4f556b94582bb74c5698ad Gerrit-Change-Number: 24694 Gerrit-PatchSet: 2 Gerrit-Owner: Gabriella Lotz <[email protected]> Gerrit-Reviewer: Abhishek Chennaka <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Gabriella Lotz <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Zoltan Martonka <[email protected]> Gerrit-Comment-Date: Wed, 16 Sep 2026 12:31:31 +0000 Gerrit-HasComments: Yes
