Alexey Serbin has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/24630 )
Change subject: [tools/master] make move of a leader replica recoverable
......................................................................
[tools/master] make move of a leader replica recoverable
Moving a replica ('kudu tablet change_config move_replica' or 'kudu cluster
rebalance') is client-driven: the source is marked 'replace' and, if it is
the leader, the client asks it to step down so it can be evicted (Raft can't
evict a leader). If the client is interrupted (Ctrl+C, or an aborted/timed-out
rebalance) after the config change but before the leader steps down, the
tablet is left over-replicated with a stuck, un-evictable leader.
Add two recovery paths:
* Master-side self-healing: once a leader marked 'replace' has its replacement
promoted to voter, ProcessTabletReport() schedules an
AsyncLeaderStepDownForReplacementTask to step it down so it can be evicted
(see ShouldStepDownLeaderForReplacement(); guarded by
--catalog_manager_step_down_leader_for_replacement, on by default, 3-4-3
only). Driven by tablet state, so it recovers an interrupted move_replica or
'cluster rebalance' with no client involved.
* CLI idempotency (move_replica): IsReplicaMoveScheduled() lets MoveReplica()
detect an already-scheduled move and wait for it, so re-running resumes
instead of failing on the over-replicated tablet.
Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Co-authored-by: Cursor <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/24630
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Alexey Serbin <[email protected]>
---
M src/kudu/consensus/quorum_util-test.cc
M src/kudu/consensus/quorum_util.cc
M src/kudu/consensus/quorum_util.h
M src/kudu/integration-tests/raft_consensus_nonvoter-itest.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/tools/kudu-admin-test.cc
M src/kudu/tools/tool_action_tablet.cc
M src/kudu/tools/tool_replica_util.cc
M src/kudu/tools/tool_replica_util.h
9 files changed, 522 insertions(+), 9 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24630
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic7b28ca4e41355bd82c76d50e137f06197b402bc
Gerrit-Change-Number: 24630
Gerrit-PatchSet: 5
Gerrit-Owner: Yifan Zhang <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Gabriella Lotz <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
Gerrit-Reviewer: Yifan Zhang <[email protected]>
