Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11251 )
Change subject: KUDU-2245 Graceful leadership transfer ...................................................................... KUDU-2245 Graceful leadership transfer This patch implements graceful leadership transfer, as described in the original Raft thesis. It has the following steps: 1. An admin client sends a request to the tablet leader for it to transfer leadership. The client can indicate a specific voter that it wants to become the leader, or it can allow the current leader to choose its successor. 2. The leader receives the request and begins a leader transfer period. During a leader transfer period, the leader does not accept writes or config change requests. This allows followers to catch up to the leader. A background timer expires the transfer period after one election timeout, since clients should be able to ride over interruptions in service lasting at least that long. If another request to transfer leadership is received during a transfer period, it will be rejected. 3. During the transfer period, the leader continues to update peers. When it receives a response from a peer, it checks if that peer is a voter and fully caught up to the leader's log. If it is, and if it is the designated successor if one was provided, the leader signals the peer to start an election, which it should win. If no eligible successor appears, the transfer period expires and the leader resumes normal operation. This is an improvement over the current leader step down method, which causes the leader to simply relinquish leadership and snooze its election timer for an extra long period, so another voter will likely become leader. Leadership transfer should usually be much faster and it allows the client to select the new leader among current voters. However, note that it does not provide strictly better guarantees- it is still possible that leadership will not be transferred. I ran TestRepeatLeaderStepDown and TestGracefulLeaderStepDown 1000 times and 200 times each, in debug and TSAN modes, with 4 stress threads, and saw no failures. Change-Id: Ic97343af9eb349556424c999799ed5e2941f0083 Reviewed-on: http://gerrit.cloudera.org:8080/11251 Tested-by: Kudu Jenkins Reviewed-by: Mike Percy <[email protected]> --- M src/kudu/consensus/consensus-test-util.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_peers.cc M src/kudu/consensus/consensus_peers.h M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/consensus_queue.h M src/kudu/consensus/peer_manager.cc M src/kudu/consensus/peer_manager.h M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h M src/kudu/integration-tests/raft_consensus-itest.cc M src/kudu/tools/kudu-admin-test.cc M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_tablet.cc M src/kudu/tools/tool_replica_util.cc M src/kudu/tools/tool_replica_util.h M src/kudu/tserver/tablet_service.cc 17 files changed, 839 insertions(+), 67 deletions(-) Approvals: Kudu Jenkins: Verified Mike Percy: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11251 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic97343af9eb349556424c999799ed5e2941f0083 Gerrit-Change-Number: 11251 Gerrit-PatchSet: 19 Gerrit-Owner: Will Berkeley <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Fengling Wang <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <[email protected]> Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Will Berkeley <[email protected]>
