[
https://issues.apache.org/jira/browse/HDDS-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-16259:
-------------------------------
Description:
We encountered an issue where transfer leadership failed with DIFFERENT_LEADER.
After looking at the audit logs, it was identified that it was due to the
METADATA log created in the old leader. The old leader should block
non-METADATA Raft log append through the use of RaftServerImpl#isSteppingDown.
The sequence looks like, say the Raft group consists of peer [A,B,C]
# User sends a transfer leadership request from Raft peer A to Raft peer B
# Raft peer A checks that Raft peer B log is up to date through
TransferLeadership#isFollowUpToDate, say the Raft peer B log index matched with
the Raft peer A for index 5
# Raft peer A sends a StartLeaderElectionRequest to Raft peer B
# {*}(Important{*}) Raft peer A creates a METADATA log entry at index 6 due to
onFollowerSuccessAppendEntries (note that unlike normal log, METADATA log is
not pretected by RaftServerImpl#isSteppingDown
# {*}(Important{*}) Raft peer A replicates the a METADATA log entry at index 6
to Raft peer C (another follower)
# Raft peer B starts an election, but Raft peer A and C rejected the election
because Raft peer B entry is still at 5 (i.e. not up-to-date)
# Raft peer A or Raft peer C election timeout is hit and then starts an
election, say Raft peer C becomes the new leader instead
# Raft peer A TransferLeadership#transfer finished, but it recorded
DIFFERENT_LEADER (expect Raft peer B, but found Raft peer C to be the new
leader)
We can disable METADATA log to prevent this issue, but we should also try to
think how to prevent METADATA log from impacting the transfer leadership
process. One easy way is that we ignore the METADATA log creation when transfer
leadership is going on, but we need to double check whether this is safe
(METADATA log is not in the original Raft paper)
was:
We encountered an issue where transfer leadership failed with DIFFERENT_LEADER.
After looking at the audit logs, it was identified that it was due to the
METADATA log created in the old leader. The old leader should block
non-METADATA Raft log append through the use of RaftServerImpl#isSteppingDown.
The sequence looks like, say the Raft group consists of peer [A,B,C]
# User sends a transfer leadership request from Raft peer A to Raft peer B
# Raft peer A checks that Raft peer B log is up to date through
TransferLeadership#isFollowUpToDate, say the Raft peer B log index matched with
the Raft peer A for index 5
# Raft peer A sends a StartLeaderElectionRequest to Raft peer B
# {*}(Important{*}) Raft peer A creates a METADATA log entry at index 6 due to
onFollowerSuccessAppendEntries (note that unlike normal log, METADATA log is
not pretected by RaftServerImpl#isSteppingDown
# {*}(Important{*}) Raft peer A replicates the a METADATA log entry at index 6
to Raft peer C (another follower)
# Raft peer B starts an election, but Raft peer A and C rejected the election
because Raft peer B entry is still at 5 (i.e. not up-to-date)
# Raft peer A or Raft peer C election timeout is hit and then starts an
election, say Raft peer C becomes the new leader instead
# Raft peer A TransferLeadership#transfer finished, but it recorded
DIFFERENT_LEADER (expect Raft peer B, but found Raft peer C to be the new
leader)
We can disable METADATA log to prevent this issue, but we should also try to
think how to prevent METADATA log from impacting the transfer leadership
process.
> Ratis transfer leadership fails with DIFFERENT_LEADER
> -----------------------------------------------------
>
> Key: HDDS-16259
> URL: https://issues.apache.org/jira/browse/HDDS-16259
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> We encountered an issue where transfer leadership failed with
> DIFFERENT_LEADER.
> After looking at the audit logs, it was identified that it was due to the
> METADATA log created in the old leader. The old leader should block
> non-METADATA Raft log append through the use of RaftServerImpl#isSteppingDown.
> The sequence looks like, say the Raft group consists of peer [A,B,C]
> # User sends a transfer leadership request from Raft peer A to Raft peer B
> # Raft peer A checks that Raft peer B log is up to date through
> TransferLeadership#isFollowUpToDate, say the Raft peer B log index matched
> with the Raft peer A for index 5
> # Raft peer A sends a StartLeaderElectionRequest to Raft peer B
> # {*}(Important{*}) Raft peer A creates a METADATA log entry at index 6 due
> to onFollowerSuccessAppendEntries (note that unlike normal log, METADATA log
> is not pretected by RaftServerImpl#isSteppingDown
> # {*}(Important{*}) Raft peer A replicates the a METADATA log entry at index
> 6 to Raft peer C (another follower)
> # Raft peer B starts an election, but Raft peer A and C rejected the
> election because Raft peer B entry is still at 5 (i.e. not up-to-date)
> # Raft peer A or Raft peer C election timeout is hit and then starts an
> election, say Raft peer C becomes the new leader instead
> # Raft peer A TransferLeadership#transfer finished, but it recorded
> DIFFERENT_LEADER (expect Raft peer B, but found Raft peer C to be the new
> leader)
> We can disable METADATA log to prevent this issue, but we should also try to
> think how to prevent METADATA log from impacting the transfer leadership
> process. One easy way is that we ignore the METADATA log creation when
> transfer leadership is going on, but we need to double check whether this is
> safe (METADATA log is not in the original Raft paper)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
