[
https://issues.apache.org/jira/browse/KAFKA-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jack Vanlightly updated KAFKA-16281:
------------------------------------
Description:
I have a TLA+ model of KIP-996 (pre-vote) and I have identified an IllegalState
exception that would occur with the existing MaybeHandleCommonResponse behavior.
The issue stems from the fact that a leader, let's call it r1, can resign
(either due to a restart or check quorum) and then later initiate a pre-vote
where it ends up in the same epoch as before. When r1 receives a response from
r2 who believes that r1 is still the leader, the logic in
MaybeHandleCommonResponse tries to transition r1 to follower of itself, causing
an IllegalState exception to be raised.
This is an example history:
# r1 is the leader in epoch 1.
# r1 quorum resigns, or restarts and resigns.
# r1 experiences an election timeout and transitions to Prospective.
# r1 sends a pre vote request to its peers.
# r2 thinks r1 is still the leader, sends a vote response, not granting its
vote and setting leaderId=r1 and epoch=1.
# r1 receives the vote response and executes MaybeHandleCommonResponse which
tries to transition r1 to Follower of itself and an illegal state occurs.
The relevant else if statement in MaybeHandleCommonResponse is here:
[https://github.com/apache/kafka/blob/a26a1d847f1884a519561e7a4fb4cd13e051c824/raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java#L1538]
In the TLA+ specification, I fixed this issue by adding a fourth condition to
this statement, that replica must not be in the Prospective state.
[https://github.com/Vanlightly/kafka-tlaplus/blob/9b2600d1cd5c65930d666b12792d47362b64c015/kraft/kip_996/kraft_kip_996_functions.tla#L336|https://github.com/Vanlightly/kafka-tlaplus/blob/421f170ba4bd8c5eceb36b88b47901ee3d9c3d2a/kraft/kip_996/kraft_kip_996_functions.tla#L336]
Note, that I also had to implement the sending of the BeginQuorumEpoch request
by the leader to prevent a replica getting stuck in Prospective. If the replica
r2 has an election timeout but due to a transient connectivity issue with the
leader, but has also fallen behind slightly, then r2 will remain stuck as a
Prospective because none of its peers, who have connectivity to the leader,
will grant it a pre-vote. To enable r2 to become a functional member again, the
leader must give it a nudge with a BeginQuorumEpoch request. The alternative
(which I have also modeled) is for a Prospective to transition to Follower when
it receives a negative pre-vote response with a non-null leaderId. This comes
with a separate liveness issue which I can discuss if this "transition to
Follower" approach is interesting. Either way, a stuck Prospective needs a way
to transition to follower eventually, if all other members have a stable leader.
was:
I have a TLA+ model of KIP-966 and I have identified an IllegalState exception
that would occur with the existing MaybeHandleCommonResponse behavior.
The issue stems from the fact that a leader, let's call it r1, can resign
(either due to a restart or check quorum) and then later initiate a pre-vote
where it ends up in the same epoch as before, but a cleared local leader id.
When r1 transitions to Prospective it clears its local leader id. When r1
receives a response from r2 who believes that r1 is still the leader, the logic
in MaybeHandleCommonResponse tries to transition r1 to follower of itself,
causing an IllegalState exception to be raised.
This is an example history:
# r1 is the leader in epoch 1.
# r1 quorum resigns, or restarts and resigns.
# r1 experiences an election timeout and transitions to Prospective clearing
its local leader id.
# r1 sends a pre vote request to its peers.
# r2 thinks r1 is still the leader, sends a vote response, not granting its
vote and setting leaderId=r1 and epoch=1.
# r1 receives the vote response and executes MaybeHandleCommonResponse which
tries to transition r1 to Follower of itself and an illegal state occurs.
The relevant else if statement in MaybeHandleCommonResponse is here:
https://github.com/apache/kafka/blob/a26a1d847f1884a519561e7a4fb4cd13e051c824/raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java#L1538
In the TLA+ specification, I fixed this issue by adding a fourth condition to
this statement, that the leaderId also does not equal this server's id.
[https://github.com/Vanlightly/kafka-tlaplus/blob/9b2600d1cd5c65930d666b12792d47362b64c015/kraft/kip_996/kraft_kip_996_functions.tla#L336]
We should probably create a test to confirm the issue first and then look at
using the fix I made in the TLA+, though there may be other options.
> Possible IllegalState with KIP-996
> ----------------------------------
>
> Key: KAFKA-16281
> URL: https://issues.apache.org/jira/browse/KAFKA-16281
> Project: Kafka
> Issue Type: Task
> Components: kraft
> Reporter: Jack Vanlightly
> Priority: Major
>
> I have a TLA+ model of KIP-996 (pre-vote) and I have identified an
> IllegalState exception that would occur with the existing
> MaybeHandleCommonResponse behavior.
> The issue stems from the fact that a leader, let's call it r1, can resign
> (either due to a restart or check quorum) and then later initiate a pre-vote
> where it ends up in the same epoch as before. When r1 receives a response
> from r2 who believes that r1 is still the leader, the logic in
> MaybeHandleCommonResponse tries to transition r1 to follower of itself,
> causing an IllegalState exception to be raised.
> This is an example history:
> # r1 is the leader in epoch 1.
> # r1 quorum resigns, or restarts and resigns.
> # r1 experiences an election timeout and transitions to Prospective.
> # r1 sends a pre vote request to its peers.
> # r2 thinks r1 is still the leader, sends a vote response, not granting its
> vote and setting leaderId=r1 and epoch=1.
> # r1 receives the vote response and executes MaybeHandleCommonResponse which
> tries to transition r1 to Follower of itself and an illegal state occurs.
> The relevant else if statement in MaybeHandleCommonResponse is here:
> [https://github.com/apache/kafka/blob/a26a1d847f1884a519561e7a4fb4cd13e051c824/raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java#L1538]
> In the TLA+ specification, I fixed this issue by adding a fourth condition to
> this statement, that replica must not be in the Prospective state.
> [https://github.com/Vanlightly/kafka-tlaplus/blob/9b2600d1cd5c65930d666b12792d47362b64c015/kraft/kip_996/kraft_kip_996_functions.tla#L336|https://github.com/Vanlightly/kafka-tlaplus/blob/421f170ba4bd8c5eceb36b88b47901ee3d9c3d2a/kraft/kip_996/kraft_kip_996_functions.tla#L336]
>
> Note, that I also had to implement the sending of the BeginQuorumEpoch
> request by the leader to prevent a replica getting stuck in Prospective. If
> the replica r2 has an election timeout but due to a transient connectivity
> issue with the leader, but has also fallen behind slightly, then r2 will
> remain stuck as a Prospective because none of its peers, who have
> connectivity to the leader, will grant it a pre-vote. To enable r2 to become
> a functional member again, the leader must give it a nudge with a
> BeginQuorumEpoch request. The alternative (which I have also modeled) is for
> a Prospective to transition to Follower when it receives a negative pre-vote
> response with a non-null leaderId. This comes with a separate liveness issue
> which I can discuss if this "transition to Follower" approach is interesting.
> Either way, a stuck Prospective needs a way to transition to follower
> eventually, if all other members have a stable leader.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
