Mickael Maison created KAFKA-20895:
--------------------------------------
Summary: Different controller election behavior depending on node
count
Key: KAFKA-20895
URL: https://issues.apache.org/jira/browse/KAFKA-20895
Project: Kafka
Issue Type: Improvement
Reporter: Mickael Maison
On a clean shutdown, the leader computes a preferred successors list. This list
is order by log end offsets and by node.id for followers with the same log end
offsets.
With a 3 node quorum, when the leader enters the resigned state, each follower
will wait a specific time based on its position in the successors list before
initiating a vote. As the first entry waits the least amount of time, it starts
a vote first. It will receive its own vote as well as the vote from the
resigned leader. This is 2 votes out of 3, a majority, thus becomes the new
leader. The 3rd node won't vote for it however. It reaches
FollowerState.canGrantVote(), but will return false because
hasFetchedFromLeader will be true.
With a 4 node quorum, it turns out that the 1st entry in the preferred
successors list cannot win the vote. Like in the 3 node scenario, it will get
its own vote as well as the vote from the resigned leader, but this isn't a
majority. The other 2 nodes won't vote for it because
FollowerState.canGrantVote() will return false because hasFetchedFromLeader
will be true. So the 1st node in the list will fail the vote and return to
Follower state. The 2nd node in the list will then start a new vote, it will
receive its own vote, the vote from the resigned leader. The first node will
also vote for it, as it's FollowerState.canGrantVote() will return true because
hasFetchedFromLeader is false. This is a majority (3 out of 4), so this node
will become the new leader. Again the 4th node won't vote for it as
FollowerState.canGrantVote() will return false because hasFetchedFromLeader
will be true.
With a 5 node quorum, we get a similar behavior than 4 nodes, the 2nd node in
the preferred successors typically win the vote as the first node cannot win.
The patterns seems to repeat. With 6 or 7 nodes, the 3rd entry in the preferred
successors should win, with 8 or 9, the 4th entry, etc.
I would expect the first entry in the preferred successors list to typically
win, but with more than 3 nodes it's not possible so in some case it seems we
may elect a node with a smaller log end offset.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
