[
https://issues.apache.org/jira/browse/KAFKA-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105011#comment-18105011
]
Suman Pal commented on KAFKA-20895:
-----------------------------------
Hi [~mimaison] , added a \{{hasLeaderResigned}} flag on \{{FollowerState}},
whose purpose is to tell a follower that its leader has stepped down. It is set
from the EndQuorumEpoch the leader sends on a clean shutdown, and
\{{canGrantVote()}} then grants a pre-vote once it is set, since granting is no
longer disruptive at that point. The first preferred successor now wins the
election instead of losing it.
Covered by a new simulation test over 4 to 9 voters, which fails without the
change with the winner at exactly the position you predicted.
PR: https://github.com/apache/kafka/pull/23168
could you take a look when you get a chance?
> Different controller election behavior depending on node count
> --------------------------------------------------------------
>
> Key: KAFKA-20895
> URL: https://issues.apache.org/jira/browse/KAFKA-20895
> Project: Kafka
> Issue Type: Improvement
> Reporter: Mickael Maison
> Assignee: Suman Pal
> Priority: Major
>
> On a clean shutdown, the leader computes a preferred successors list. This
> list is order by log end offsets and by node.id for followers with the same
> log end offsets.
> With a 3 node quorum, when the leader enters the resigned state, each
> follower will wait a specific time based on its position in the successors
> list before initiating a vote. As the first entry waits the least amount of
> time, it starts a vote first. It will receive its own vote as well as the
> vote from the resigned leader. This is 2 votes out of 3, a majority, thus
> becomes the new leader. The 3rd node won't vote for it however. It reaches
> FollowerState.canGrantVote(), but will return false because
> hasFetchedFromLeader will be true.
> With a 4 node quorum, it turns out that the 1st entry in the preferred
> successors list cannot win the vote. Like in the 3 node scenario, it will get
> its own vote as well as the vote from the resigned leader, but this isn't a
> majority. The other 2 nodes won't vote for it because
> FollowerState.canGrantVote() will return false because hasFetchedFromLeader
> will be true. So the 1st node in the list will fail the vote and return to
> Follower state. The 2nd node in the list will then start a new vote, it will
> receive its own vote, the vote from the resigned leader. The first node will
> also vote for it, as it's FollowerState.canGrantVote() will return true
> because hasFetchedFromLeader is false. This is a majority (3 out of 4), so
> this node will become the new leader. Again the 4th node won't vote for it as
> FollowerState.canGrantVote() will return false because hasFetchedFromLeader
> will be true.
> With a 5 node quorum, we get a similar behavior than 4 nodes, the 2nd node in
> the preferred successors typically win the vote as the first node cannot win.
> The patterns seems to repeat. With 6 or 7 nodes, the 3rd entry in the
> preferred successors should win, with 8 or 9, the 4th entry, etc.
> I would expect the first entry in the preferred successors list to typically
> win, but with more than 3 nodes it's not possible. Even if there's logic to
> avoid electing a node with a shorter metadata log, I think it still means
> more election rounds so potentially degraded availability for quorum with
> more than 3 controllers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
