Yifan Zhang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24630
Change subject: [tools/master] make move of a leader replica recoverable ...................................................................... [tools/master] make move of a leader replica recoverable Moving a replica via 'kudu tablet change_config move_replica' or the 'kudu cluster rebalance' tool is client-driven: the source is marked with the 'replace' attribute and, when it is the leader, the client asks it to step down so it can be evicted (Raft cannot evict a leader). Both the step-down and the wait-for-completion happen while the client keeps polling, so if the client is interrupted (e.g. Ctrl+C, or an aborted/timed-out rebalance run) after the config change but before the leader steps down, the tablet is left over-replicated with a stuck, un-evictable leader. Add two complementary recovery paths: * Master-side self-healing: ShouldStepDownLeaderForReplacement() plus an AsyncLeaderStepDownForReplacementTask scheduled from ProcessTabletReport() once a leader marked with 'replace' has its replacement promoted to voter (guarded by --catalog_manager_step_down_leader_for_replacement, on by default, 3-4-3 only). It targets the captured leader and no-ops if that replica is no longer the leader. This is driven by tablet state rather than by whoever initiated the move, so it recovers an interrupted move_replica as well as an interrupted 'cluster rebalance' run, with no client involved. * CLI idempotency (move_replica only): IsReplicaMoveScheduled() lets MoveReplica() detect an already-scheduled move and just wait for it to finish, so re-running the command resumes instead of failing on the over-replicated tablet. Add a unit test for ShouldStepDownLeaderForReplacement() and an integration test (TestMoveReplicaIdempotent) covering the interrupted-move resume. Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc Co-authored-by: Cursor <[email protected]> --- M src/kudu/consensus/quorum_util-test.cc M src/kudu/consensus/quorum_util.cc M src/kudu/consensus/quorum_util.h M src/kudu/master/catalog_manager.cc M src/kudu/tools/kudu-admin-test.cc M src/kudu/tools/tool_action_tablet.cc M src/kudu/tools/tool_replica_util.cc M src/kudu/tools/tool_replica_util.h 8 files changed, 415 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/30/24630/1 -- To view, visit http://gerrit.cloudera.org:8080/24630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc Gerrit-Change-Number: 24630 Gerrit-PatchSet: 1 Gerrit-Owner: Yifan Zhang <[email protected]>
