[
https://issues.apache.org/jira/browse/KAFKA-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mickael Maison updated KAFKA-20895:
-----------------------------------
Description:
On a clean shutdown, the leader computes a preferred successors list. This list
is order by log end offsets and by node.id for followers with the same log end
offsets.
With a 3 node quorum, when the leader enters the resigned state, each follower
will wait a specific time based on its position in the successors list before
initiating a vote. As the first entry waits the least amount of time, it starts
a vote first. It will receive its own vote as well as the vote from the
resigned leader. This is 2 votes out of 3, a majority, thus becomes the new
leader. The 3rd node won't vote for it however. It reaches
FollowerState.canGrantVote(), but will return false because
hasFetchedFromLeader will be true.
With a 4 node quorum, it turns out that the 1st entry in the preferred
successors list cannot win the vote. Like in the 3 node scenario, it will get
its own vote as well as the vote from the resigned leader, but this isn't a
majority. The other 2 nodes won't vote for it because
FollowerState.canGrantVote() will return false because hasFetchedFromLeader
will be true. So the 1st node in the list will fail the vote and return to
Follower state. The 2nd node in the list will then start a new vote, it will
receive its own vote, the vote from the resigned leader. The first node will
also vote for it, as it's FollowerState.canGrantVote() will return true because
hasFetchedFromLeader is false. This is a majority (3 out of 4), so this node
will become the new leader. Again the 4th node won't vote for it as
FollowerState.canGrantVote() will return false because hasFetchedFromLeader
will be true.
With a 5 node quorum, we get a similar behavior than 4 nodes, the 2nd node in
the preferred successors typically win the vote as the first node cannot win.
The patterns seems to repeat. With 6 or 7 nodes, the 3rd entry in the preferred
successors should win, with 8 or 9, the 4th entry, etc.
I would expect the first entry in the preferred successors list to typically
win, but with more than 3 nodes it's not possible. Even if there's logic to
avoid electing a node with a shorter metadata log, I think it still means more
election rounds so potentially degraded availability for quorum with more than
3 controllers.
was:
On a clean shutdown, the leader computes a preferred successors list. This list
is order by log end offsets and by node.id for followers with the same log end
offsets.
With a 3 node quorum, when the leader enters the resigned state, each follower
will wait a specific time based on its position in the successors list before
initiating a vote. As the first entry waits the least amount of time, it starts
a vote first. It will receive its own vote as well as the vote from the
resigned leader. This is 2 votes out of 3, a majority, thus becomes the new
leader. The 3rd node won't vote for it however. It reaches
FollowerState.canGrantVote(), but will return false because
hasFetchedFromLeader will be true.
With a 4 node quorum, it turns out that the 1st entry in the preferred
successors list cannot win the vote. Like in the 3 node scenario, it will get
its own vote as well as the vote from the resigned leader, but this isn't a
majority. The other 2 nodes won't vote for it because
FollowerState.canGrantVote() will return false because hasFetchedFromLeader
will be true. So the 1st node in the list will fail the vote and return to
Follower state. The 2nd node in the list will then start a new vote, it will
receive its own vote, the vote from the resigned leader. The first node will
also vote for it, as it's FollowerState.canGrantVote() will return true because
hasFetchedFromLeader is false. This is a majority (3 out of 4), so this node
will become the new leader. Again the 4th node won't vote for it as
FollowerState.canGrantVote() will return false because hasFetchedFromLeader
will be true.
With a 5 node quorum, we get a similar behavior than 4 nodes, the 2nd node in
the preferred successors typically win the vote as the first node cannot win.
The patterns seems to repeat. With 6 or 7 nodes, the 3rd entry in the preferred
successors should win, with 8 or 9, the 4th entry, etc.
I would expect the first entry in the preferred successors list to typically
win, but with more than 3 nodes it's not possible so in some case it seems we
may elect a node with a smaller log end offset.
> Different controller election behavior depending on node count
> --------------------------------------------------------------
>
> Key: KAFKA-20895
> URL: https://issues.apache.org/jira/browse/KAFKA-20895
> Project: Kafka
> Issue Type: Improvement
> Reporter: Mickael Maison
> Priority: Major
>
> On a clean shutdown, the leader computes a preferred successors list. This
> list is order by log end offsets and by node.id for followers with the same
> log end offsets.
> With a 3 node quorum, when the leader enters the resigned state, each
> follower will wait a specific time based on its position in the successors
> list before initiating a vote. As the first entry waits the least amount of
> time, it starts a vote first. It will receive its own vote as well as the
> vote from the resigned leader. This is 2 votes out of 3, a majority, thus
> becomes the new leader. The 3rd node won't vote for it however. It reaches
> FollowerState.canGrantVote(), but will return false because
> hasFetchedFromLeader will be true.
> With a 4 node quorum, it turns out that the 1st entry in the preferred
> successors list cannot win the vote. Like in the 3 node scenario, it will get
> its own vote as well as the vote from the resigned leader, but this isn't a
> majority. The other 2 nodes won't vote for it because
> FollowerState.canGrantVote() will return false because hasFetchedFromLeader
> will be true. So the 1st node in the list will fail the vote and return to
> Follower state. The 2nd node in the list will then start a new vote, it will
> receive its own vote, the vote from the resigned leader. The first node will
> also vote for it, as it's FollowerState.canGrantVote() will return true
> because hasFetchedFromLeader is false. This is a majority (3 out of 4), so
> this node will become the new leader. Again the 4th node won't vote for it as
> FollowerState.canGrantVote() will return false because hasFetchedFromLeader
> will be true.
> With a 5 node quorum, we get a similar behavior than 4 nodes, the 2nd node in
> the preferred successors typically win the vote as the first node cannot win.
> The patterns seems to repeat. With 6 or 7 nodes, the 3rd entry in the
> preferred successors should win, with 8 or 9, the 4th entry, etc.
> I would expect the first entry in the preferred successors list to typically
> win, but with more than 3 nodes it's not possible. Even if there's logic to
> avoid electing a node with a shorter metadata log, I think it still means
> more election rounds so potentially degraded availability for quorum with
> more than 3 controllers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
