Ivan Andika created HDDS-16259:
----------------------------------
Summary: Ratis transfer leadership fails with DIFFERENT_LEADER
Key: HDDS-16259
URL: https://issues.apache.org/jira/browse/HDDS-16259
Project: Apache Ozone
Issue Type: Bug
Reporter: Ivan Andika
Assignee: Ivan Andika
We encountered an issue where transfer leadership failed with DIFFERENT_LEADER.
After looking at the audit logs, it was identified that it was due to the
METADATA log created in the old leader. The old leader should block
non-METADATA Raft log append through the use of RaftServerImpl#isSteppingDown.
The sequence looks like, say the Raft group consists of peer [A,B,C]
# User sends a transfer leadership request from Raft peer A to Raft peer B
# Raft peer A checks that Raft peer B log is up to date through
TransferLeadership#isFollowUpToDate, say the Raft peer B log index matched with
the Raft peer A for index 5
# Raft peer A sends a StartLeaderElectionRequest to Raft peer B
# {*}(Important{*}) Raft peer A creates a METADATA log entry at index 6 due to
onFollowerSuccessAppendEntries (note that unlike normal log, METADATA log is
not pretected by RaftServerImpl#isSteppingDown
# {*}(Important{*}) Raft peer A replicates the a METADATA log entry at index 6
to Raft peer C (another follower)
# Raft peer B starts an election, but Raft peer A and C rejected the election
because Raft peer B entry is still at 5 (i.e. not up-to-date)
# Raft peer A or Raft peer C election timeout is hit and then starts an
election, say Raft peer C becomes the new leader instead
# Raft peer A TransferLeadership#transfer finished, but it recorded
DIFFERENT_LEADER (expect Raft peer B, but found Raft peer C to be the new
leader)
We can disable METADATA log to prevent this issue, but we should also try to
think how to prevent METADATA log from impacting the transfer leadership
process.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
