Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #21025: URL: https://github.com/apache/kafka/pull/21025#issuecomment-3604818647 > > Thanks for the feature @TaiJuWu but I don't think we should change the semantic of the KRaft protocol without a KIP. > > I understand that the auto join feature doesn't meet your requirements. Have you considered not using the auto join feature and instead implementing controller addition and removal in the orchestration layer using Admin:addRaftVoter, Admin::removeRaftVoter and Admin::describeMetadataQuorum? > > Thanks for the comments @jsancio ! I agree if we can't come up a good solution for this issue, it's also fine to keep what we have so far. But, I think we should at least improve the documentation and mention about this known UX issue. @TaiJuWu , could you help on this? Thanks. Sure. Will update this PR for documentation only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #21025: URL: https://github.com/apache/kafka/pull/21025#issuecomment-3604763210 > Thanks for the feature @TaiJuWu but I don't think we should change the semantic of the KRaft protocol without a KIP. > > I understand that the auto join feature doesn't meet your requirements. Have you considered not using the auto join feature and instead implementing controller addition and removal in the orchestration layer using Admin:addRaftVoter, Admin::removeRaftVoter and Admin::describeMetadataQuorum? Thanks for the comments @jsancio ! I agree if we can't come up a good solution for this issue, it's also fine to keep what we have so far. But, I think we should at least improve the documentation and mention about this known UX issue. @TaiJuWu , could you help on this? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
jsancio commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2582197596
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2335,9 +2337,12 @@ private boolean handleAddVoterResponse(
/* These error codes indicate the replica was successfully added or
the leader is unable to
* process the request. In either case, reset the update voter set
timer to back off.
*/
-if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
-error == Errors.DUPLICATE_VOTER) {
+if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT) {
+
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+return true;
+} else if (error == Errors.DUPLICATE_VOTER) {
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasJoined = true;
Review Comment:
This doesn't seem correct. You want to have the has joined boolean because
true when the voter is successfully added to the voter set.
##
raft/src/main/java/org/apache/kafka/raft/internals/KRaftControlRecordStateMachine.java:
##
@@ -302,6 +310,7 @@ private void handleBatch(Batch batch, OptionalLong
overrideOffset) {
case KRAFT_VOTERS:
VoterSet voters = VoterSet.fromVotersRecord((VotersRecord)
record.message());
kafkaRaftMetrics.updateNumVoters(voters.size());
+seenReplicaKey.addAll(voters.voterKeys());
Review Comment:
Kafka should not have data structures that only grow in size. E.g. there is
no remove operation against `seenReplicaKey`. This means that the size of this
data structure is unbounded and will continue to grow even if the voter set is
as small as 3 to 5 voters.
##
raft/src/main/java/org/apache/kafka/raft/internals/AddVoterHandler.java:
##
@@ -156,6 +156,23 @@ public CompletableFuture
handleAddVoterRequest(
);
}
+
+if (!ackWhenCommitted && partitionState.hasSeenReplicaKey(voterKey)) {
+// ackWhenCommitted is used to distinguish the request from admin
+// or broker. If it comes from broker and had been in cluster, we
should
+// reject it.
Review Comment:
This is peculiar association. Why should "ack when committed" requests fail
if the voter key is in the log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #21025: URL: https://github.com/apache/kafka/pull/21025#issuecomment-3600652601 @showuon Thanks for your sharing. Do we consider to apply the other solution which is adding a configuration between two auto-join? We can make it a private at 4.2 and public at 4.3. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #21025: URL: https://github.com/apache/kafka/pull/21025#issuecomment-3600448983 @jsancio @kevin-wu24 , I found the current solution has one drawback, which is that the snapshot only contains `the latest voters at the time the snapshot taken`. That means we lost the voters change history before the snapshot. This could cause the issue: 1. controller 1, 2, 3 are the voters 2. controller 3 is removed, so the current voters are [1, 2] 3. controller 3 cannot be auto-joined because the leader controller found 3 has ever been in the voter. 4. The leader controller takes a snapshot for metadata log 5. The leader controller restarts, and load the snapshot + records after snapshot, which gets the voter history contains only [1, 2] 6. controller 3 restarts, addVoterRequest is sent because auto-join is enabled, but this time it will succeed because the leader controller doesn't know controller 3 has ever been in the voters. My suggestion: 1. Keep the same semantic: `new controllers (node id + directory UUID tuples) will automatically join the KRaft voter set once if they have not been a voter before.`, but add a note in doc to notify users to shutdown the removed controller soon to avoid the node being auto-joined again in some edge cases. 2. Change the semantic to: `Only do the auto-join when a node is start up first time`. The start up first time is defined by: LEO of metadata_log is 0. The use case of auto-join is original designed for new formatted controller node, so it should not be a problem. The drawback of this issue is that the auto-join can only have one shot if the first auto-join failed and restarted. For example: a. controller A is formatted, with auto-join enabled b. controller A startup, addVoterRequest is sent (We can initialize the updateVoter timer as expired state, and reset after response received) c. The leader controller responded with error d. controller A fetches metadata log from the leader controller (metadata log LEO > 0) e. Before controller A retrying the addVoter request, it crashes f. After restart, controller A will not auto send addVoter request anymore due to (metadata log LEO > 0) I think the suggestion (2) is better since it has no edge cases at all. We just need to make it clear that the auto-join can only happen when a node has empty metadata log. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2579479629
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2339,6 +2341,9 @@ private boolean handleAddVoterResponse(
error == Errors.DUPLICATE_VOTER) {
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
return true;
+} else if (error == Errors.INVALID_VOTER_KEY) {
+hasJoined = true;
+return true;
Review Comment:
It was fixed, I made a mistake at that moment :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #21025: URL: https://github.com/apache/kafka/pull/21025#issuecomment-3600031084 > Thanks for the PR @TaiJuWu . > > This solution has the unfortunate side effect that nodes cannot leave and join the quorum. The user cannot do the following operations: > > 1. First add voter A. > 2. Remove voter A, on purpose or by accident. > 3. Add voter A again. This should succeed but it will fail with this PR. Our integration test also cover such case, it test follow: 1. observers can join cluster with auto-join 2. it be removed by manual and does not send addVoterRequest twice again 3. admin can add this node to the cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2579429259
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2338,6 +2340,7 @@ private boolean handleAddVoterResponse(
if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
error == Errors.DUPLICATE_VOTER) {
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasJoined = true;
Review Comment:
Error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2579434322
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2339,6 +2341,9 @@ private boolean handleAddVoterResponse(
error == Errors.DUPLICATE_VOTER) {
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
return true;
+} else if (error == Errors.INVALID_VOTER_KEY) {
+hasJoined = true;
+return true;
Review Comment:
The `DUPLICATE_VOTER` case should be its own `else if` clause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2579429259
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2338,6 +2340,7 @@ private boolean handleAddVoterResponse(
if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
error == Errors.DUPLICATE_VOTER) {
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasJoined = true;
Review Comment:
Error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #21025: URL: https://github.com/apache/kafka/pull/21025#issuecomment-3599930001 > Thanks for the PR @TaiJuWu . > > This solution has the unfortunate side effect that nodes cannot leave and join the quorum. The user cannot do the following operations: > > 1. First add voter A. > > 2. Remove voter A, on purpose or by accident. > > 3. Add voter A again. This should succeed but it will fail with this PR. @jsancio , basically, the current solution only block the `auto-join` add voter request if it's not the 1st time joined the voters. So in your operation example, the step (3) will succeed if it's `manual join` (with `AckWhenCommitted=true` in KIP-1186). And if in step (3), it's `auto-join` (with `AckWhenCommitted=false`), it will be rejected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2578058942
##
raft/src/main/java/org/apache/kafka/raft/internals/AddVoterHandler.java:
##
@@ -156,6 +156,23 @@ public CompletableFuture
handleAddVoterRequest(
);
}
+
+if (!ackWhenCommitted && partitionState.hasSeenReplicaKey(voterKey)) {
+// ackWhenCommitted is used to distinguish the request from admin
+// or broker. If it comes from broker and had been in cluster, we
should
+// reject it.
+return CompletableFuture.completedFuture(
+RaftUtil.addVoterResponse(
+Errors.INVALID_VOTER_KEY,
Review Comment:
Return `Errors.DUPLICATE_VOTER`.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2339,6 +2341,9 @@ private boolean handleAddVoterResponse(
error == Errors.DUPLICATE_VOTER) {
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
return true;
+} else if (error == Errors.INVALID_VOTER_KEY) {
+hasJoined = true;
+return true;
Review Comment:
We want to use `Errors.DUPLICATE_VOTER` not `Errors.INVALID_VOTER_KEY`,
since the former error code means the requested node is already a part of the
voter set.
Whenever the requesting node receives a `DUPLICATE_VOTER` response, it knows
for sure it can turn off auto-join code.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2667,8 +2672,6 @@ private Optional maybeHandleCommonResponse(
} else if (error == Errors.BROKER_NOT_AVAILABLE) {
return Optional.of(false);
} else if (error == Errors.INVALID_VOTER_KEY) {
-// The voter key in the request for VOTE and BEGIN_QUORUM_EPOCH
doesn't match the
Review Comment:
Please revert this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
ppatierno commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2576492118
##
raft/src/main/java/org/apache/kafka/raft/internals/AddVoterHandler.java:
##
@@ -156,6 +156,23 @@ public CompletableFuture
handleAddVoterRequest(
);
}
+
+if (!ackWhenCommitted && partitionState.hasSeenReplicaKey(voterKey)) {
+// ackWhenCommitted is used to distinguish the request from admin
+// or broker. If it comes from broker and had been in cluster, we
should
+// reject it.
+return CompletableFuture.completedFuture(
+RaftUtil.addVoterResponse(
+Errors.INVALID_VOTER_KEY,
+String.format(
+"The voter key for %s has joined the set of voters.",
Review Comment:
```suggestion
"The voter key for %s has already joined the set of
voters.",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2576407012
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2339,6 +2341,9 @@ private boolean handleAddVoterResponse(
error == Errors.DUPLICATE_VOTER) {
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
return true;
+} else if (error == Errors.INVALID_VOTER_KEY) {
+ hasJoined = true;
+ return true;
Review Comment:
Indention is wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #21025: URL: https://github.com/apache/kafka/pull/21025#issuecomment-3595624376 @ppatierno @kevin-wu24 , FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3595464034 new PR https://github.com/apache/kafka/pull/21025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594746804 I will open a new PR for this, thank! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu closed pull request #20859: KAFKA-19850: KRaft voter auto join will add a removed voter immediately URL: https://github.com/apache/kafka/pull/20859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594464927 > Kevin and I had an offline discussion about this, we agree: > > 1. Go with server-side `ReplicaKey` idempotency approach. Even if the `UpdateVoterSetTimer` is not required in the client side, and the `LEO==0` check should work, but the client-side gating is still as not reliable as in the server-side. > 2. We can get the implementation ready first (without test), and waiting for other reviewers' comments, especially the KIP-853 author @jsancio 's comments. > 3. After everyone is onboard with this solution, we add tests and review and merge it in v4.2.0. > > @TaiJuWu , let us know if you have any other thoughts. Thank you. I also agree your approach, that is really better than current design, will close this PR, thanks for your participation! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594454885 Kevin and I had an offline discussion about this, we agree: 1. Go with server-side `ReplicaKey` idempotency approach. Even if the `UpdateVoterSetTimer` is not required in the client side, and the `LEO==0` check should work, but the client-side gating is still as not reliable as in the server-side. 2. We can get the implementation ready first (without test), and waiting for other reviewers' comments, especially the KIP-853 author @jsancio 's comments. 3. After everyone is onboard with this solution, we add tests and review and merge it in v4.2.0. @TaiJuWu , let us know if you have any other thoughts. Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594416314 > Then, we can easily check the log end offset == 0 in metadata log in kakfaRaftClient#initialize() in the client side. If LEO is 0, we know this node is a brand new node, we will auto-join it if auto-join is enabled. Otherwise, we won't do that. Is this much clear to resolve this issue? Here is a case where this client side implementation would run into poor UX. New ReplicaKey X formats and starts up. It is in the "auto-joining" mode, performs a fetch first, writes the records to disk, and then restarts. If the above occurs before the `UpdateVoterSetTimer` expires and the client polls, the client has no way of turning auto-join on during subsequent restarts. In this case, the user expects a node to auto-join but it doesn't because of the process restarting. I believe the server-side `ReplicaKey` idempotency approach is best. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594404590 > @showuon, the active controller can distinguish between a auto-join AddVoter request and a manual AddVoter request because of https://cwiki.apache.org/confluence/display/KAFKA/KIP-1186%3A+Update+AddRaftVoterRequest+RPC+to+support+auto-join. This KIP was needed as part of the initial implementation to make auto-join fully available. When a ReplicaKey auto-joins and is then manually removed, it will no longer auto-join again, but you can still add it manually. Hopefully that clears things up. Thanks for the update, @kevin-wu24 ! That looks good. I have another thought. How about the `auto-join` is only allowed when the node _firstly_ start up. Basically, the purpose of the auto-join is to allow a node to easily join the voters after storage formatting, right? Then, we can easily check the `log end offset == 0 in metadata log` in `kakfaRaftClient#initialize()` in the client side. If LEO is 0, we know this node is a brand new node, we will auto-join it if `auto-join is enabled`. Otherwise, we won't do that. Is this much clear to resolve this solution? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594356326 > I think the the semantic will change to Once a node(ReplicaKey) is removed it will never be able to join the voter again. However, we already allow one node join/remove multiple times before v4.2, so I don't think we can do that now without a KIP. @showuon, the active controller can distinguish between a auto-join `AddVoter` request and a manual `AddVoter` request because of https://cwiki.apache.org/confluence/display/KAFKA/KIP-1186%3A+Update+AddRaftVoterRequest+RPC+to+support+auto-join. This KIP was needed as part of the initial implementation to make auto-join fully available. When a ReplicaKey auto-joins and is then manually removed, it will no longer auto-join again, but you can still add it manually. Hopefully that clears things up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594348819 > Maybe a better documentation is that new controllers (node id + directory UUID tuples) will automatically join the KRaft voter set once if they have not been a voter before. @kevin-wu24 , I think the biggest problem is that the server side (active controller) doesn't know if an observer is doing `auto-join` or `manual join`. I think the the semantic will change to `Once a node(ReplicaKey) is removed it will never be able to join the voter again`. However, we already allow one node join/remove multiple times before v4.2, so I don't think we can do that now. I think the suggestion sounds good as long as we can provide a way to distinguish `auto-join` or `manual join` in the addVoter request, but that needs a KIP... :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3589283023 > Does it count for the initial controllers or not? Does it count for manually join? Yes for both. When handling an auto-join `AddVoterRPC`, the leader will check its voter set history if the requesting `ReplicaKey` was part of it. Maybe a better documentation is that new controllers (node id + directory UUID tuples) will automatically join the KRaft voter set once if they are not already a voter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587892645 > Why make the proposed semantic idempotent per (ReplicaKey + process incarnation) tuple? I would argue that having the node try to auto-join on restart is just as confusing UX. In the original use case, if the node just gets restarted, we end up in the same place of the controller rejoining the voter set when we don't "want" it to (i.e. after an explicit removal). Yes, I know what you mean. I don't have a good answer on this TBH. I just know it's better than "auto-joining a just removed voter". > Another approach is we can make auto-join idempotent per ReplicaKey and handle this server-side. I think this approach best captures the intent of the feature and maintains a good UX. Basically, when I startup a new observer controller with auto-join, it will join the quorum automatically at most once for the lifetime of its ReplicaKey. After an explicit removal, the leader can return a special error for a subsequent auto-join request from the same ReplicaKey to tell the node to stop sending AddVoter. If the controller's disk fails and it restarts, it will have a new ReplicaKey, and will auto-join again (this is the intended behavior because it is essentially a completely new controller). This does not require a KIP, and would be simpler to implement than either of the discussed approaches. 1. `make auto-join idempotent per ReplicaKey ` -> +1 2. `it will join the quorum automatically at most once for the lifetime of its ReplicaKey` -> I don't know how we define the `automatically join`. Does it count for the initial controllers or not? Does it count for `manually join`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587856896 > What I'm afraid is that we shipped in v4.2 with "the aggressive auto-join semantic even if one node is just removed from the voters set". Then in v4.3, the semantic suddenly changes to "auto-join only when startup". It will confuse users and even surprise users. I think we should decide which semantic we want to deliver first, instead of deliver it then change it. Yeah, that makes sense to me. One thing that confuses me with the proposed semantic from the JIRA is: Why make the proposed semantic idempotent per (`ReplicaKey` + process incarnation) tuple? I would argue that having the node try to auto-join on restart is just as confusing UX. In the original use case, if the node just gets restarted, we end up in the same place of the controller rejoining the voter set when we don't "want" it to (i.e. after an explicit removal). Another approach is we can make auto-join idempotent per `ReplicaKey` and handle this server-side. I think this approach best captures the intent of the feature and maintains a good UX. Basically, when I startup a new observer controller with auto-join, it will join the quorum automatically at most once for the lifetime of its `ReplicaKey`. If the controller's disk fails and it restarts, it will have a new `ReplicaKey`, and will auto-join again. This does not require a KIP, and would be simpler to implement than either of the discussed approaches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587821188 Thanks for the comment @kevin-wu24 ! > My opinion is we should go with the KIP-design of server-side idempotency for auto-join additions if we are going to implement this semantic change at all. We should not try to hastily implement this change just to land it in 4.2, especially if we intend to just revert this change and do the server-side idempotency in 4.3. Remember that bugs with this implementation will also lead to confusing/bad UX. If we want to change the semantic, we should do it just once, with what we believe to be the best design + implementation. What I'm afraid is that we shipped in v4.2 with "the aggressive auto-join semantic even if one node is just removed from the voters set". Then in v4.3, the semantic suddenly changes to "auto-join only when startup". It will confuse users and even surprise users. I think we should decide which semantic we want to deliver first, instead of deliver it then change it. > I believe we have gotten in KIPs after KIP-freeze before right? This KIP would be self-contained within KRaft module and only affect the auto-join feature, so maybe we can make this an exception and get this in for 4.2. I'm not sure if it will be accepted or not. Need the release manager's decision. > I don't see the harm in this waiting until 4.3, so we can give this change in semantic the proper time for review. Let's also wait and see what @jsancio thinks about all this too. I'd also like to hear @jsancio 's thought on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587811957 > I believe we have gotten in KIPs after KIP-freeze before right? This KIP would be self-contained within KRaft module and only affect the auto-join feature, so maybe we can make this an exception and get this in for 4.2. I am happy to see there is a solution with KIP, actually it is difficult to resolve in client side (as you mentioned edge cases) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2570363560
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
kafkaRaftMetrics,
externalKRaftMetrics
);
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added
to voter set at offset {}",
+ localReplicaKey, offset);
+hasJoined = Optional.of(true);
+}
+}
+});
+
// Read the entire log
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent() && canBecomeVoter && quorumConfig.autoJoin()
+&& isVoter(ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)))
{
+hasJoined = Optional.of(true);
+}
+
Review Comment:
You need to wait for a fetch to complete, so that the starting voters can
complete a fetch loop and transition from `UNKNOWN -> HAS_JOINED`.
In `testBootstrapVoterSetDoesNotSendAddVoterAfterRemove`, I am pretty sure
that `advanceTimeAndCompleteFetch` just does an empty fetch. You need to
actually perform a fetch that contains the starting voters in the response. The
code you currently have sets up a state where the initial local voter set state
defined by `withStartingVoters` is "out-of-date" with the first fetch loop
performed, so the node thinks it is in `HAS_NOT_JOINED` and tries to auto-join,
which is the intended behavior of the design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587775523 Thanks for the discussion @TaiJuWu and @showuon. > But for v4.2, I think we still need to have a temporary solution for it, which should be this PR. My imagination is after the improved KIP merged, we can revert the solutions in this PR. And in users point of view, there is no any change when using v4.2 and v4.3. But this implementation will still exist in the released 4.2 branch + binary. Again, although I've mentioned some cases, it is difficult to reason about the correctness of this client-side design. My opinion is we should go with the KIP-design of server-side idempotency for auto-join additions if we are going to implement this semantic change at all. We should not try to hastily implement this change just to land it in 4.2, especially if we intend to just revert this change and do the server-side idempotency in 4.3. Remember that bugs with this implementation will also lead to confusing/bad UX. If we want to change the semantic, we should do it just once, with what we believe to be the best design + implementation. I believe we have gotten in KIPs after KIP-freeze before right? This KIP would be self-contained within KRaft module and only affect the auto-join feature, so maybe we can make this an exception and get this in for 4.2. I hope my previous comments have shown that trying to implement this functionality client-side is not straightforward at all. Just because I have identified edge cases that the current design handles does not mean the current design handles all possible cases. I also will not have as much time to review this change like I had this week as we come up on 4.2 code freeze. I don't see the harm in this waiting until 4.3, so we can give this change in semantic the proper time for review. Let's also wait and see what @jsancio thinks about all this too. About your question @showuon: > We might need to add another clause in shouldSendAddOrRemoveVoter: && local node LEO >= leader HWM. > > I can understand and agree with this check, the network partition issue will be resolved. But alternatively, I think we can also fix the issue by only improving KAFKA-19933 without this shouldSendAddOrRemoveVoter: && local node LEO >= leader HWM change. WDYT? Maybe? I can offer the following but I still need to think about this more. I am pretty sure that we need to implement KAFKA-19933 for the current design, but I do not know what that new "is caught up" check should be at the moment. I think the check on the leader is `auto-joing node's fetch offset >= leader's previous HWM`, but what is "previous HWM" here? This is yet another reason to do just the server-side approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2570324985
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
kafkaRaftMetrics,
externalKRaftMetrics
);
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added
to voter set at offset {}",
+ localReplicaKey, offset);
+hasJoined = Optional.of(true);
+}
+}
+});
+
// Read the entire log
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent() && canBecomeVoter && quorumConfig.autoJoin()
+&& isVoter(ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)))
{
+hasJoined = Optional.of(true);
+}
+
Review Comment:
@TaiJuWu , about your comment:
1. Even if this is the temporary solution for v4.2, we still need to make it
right, otherwise, this is not a `good solution` at all.
2. If the change will fail the tests or make them flaky, we should:
a. figure it out if it is expected or not
b. update the tests to make it work as what we expected
I appreciate you stay with us for this long time. Please let us know if you
need any help. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2570289277
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
kafkaRaftMetrics,
externalKRaftMetrics
);
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added
to voter set at offset {}",
+ localReplicaKey, offset);
+hasJoined = Optional.of(true);
+}
+}
+});
+
// Read the entire log
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent() && canBecomeVoter && quorumConfig.autoJoin()
+&& isVoter(ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)))
{
+hasJoined = Optional.of(true);
+}
+
Review Comment:
You are right here but if this is just quick fix for 4.2, I think this is ok
for temporary solution, otherwise the new add test from me
(`testRemovedControllerWontJoinAgain` and
`testBootstrapVoterSetDoesNotSendAddVoterAfterRemove`) will flaky / fail which
is not I expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859: URL: https://github.com/apache/kafka/pull/20859#discussion_r2570278282 ## raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java: ## @@ -3372,8 +3393,8 @@ private boolean shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr * and are configured to auto join should attempt to automatically join the voter * set for the configured topic partition. */ -return !hasJoined && partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter -&& quorumConfig.autoJoin() && state.hasUpdateVoterSetPeriodExpired(currentTimeMs); +return hasJoined.isPresent() && !hasJoined.get() && partitionState.lastKraftVersion().isReconfigSupported() Review Comment: `hasJoined.isPresent` is also mean the local node catch hight watermark, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859: URL: https://github.com/apache/kafka/pull/20859#discussion_r2570278282 ## raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java: ## @@ -3372,8 +3393,8 @@ private boolean shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr * and are configured to auto join should attempt to automatically join the voter * set for the configured topic partition. */ -return !hasJoined && partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter -&& quorumConfig.autoJoin() && state.hasUpdateVoterSetPeriodExpired(currentTimeMs); +return hasJoined.isPresent() && !hasJoined.get() && partitionState.lastKraftVersion().isReconfigSupported() Review Comment: `hasJoined.isPresent` is also mean, the local node catch hight watermark, right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587572361 > Thanks for the changes @TaiJuWu. Leaving a review of the current approach in `src/main`, but please consider the below as an alternative approach (I think the below is simpler, but I would like @jsancio and @showuon 's input on it). In the interest of not duplicating effort, I think we should all agree on a design first. > > Sorry for the back and forth, but I think another, potentially easier way to go about this is actually to make the `ADD_RAFT_VOTER` sent by auto-join idempotent for the lifetime of the controller process. Basically, the leader will look through its voter set history and return a successful response if the requested node + incarnation ID was ever part of its voter set history. > > This approach requires a KIP, because we need to send the incarnation ID as part of the request, and the VotersRecord schema needs to be updated (meaning we need a kraft.version=2 as well I think), but it is much easier to prove this approach's correctness. > > You can think of our current approach as trying to handle every single edge case on the client-side, which is difficult to reason about because of partitioning + async nature of replication, whereas this approach makes our state change operation idempotent server-side, so the leader does not care if the client (a given controller process incarnation + `ReplicaKey` tuple) sends additional add voter requests. @kevin-wu24 , I'm +1 on having a KIP to have the gate in the leader controller side. After all, the controllers addition/removal are all handling in the leader controller side. I agree that will be much reliable and clean way to handle it. But for v4.2, I think we still need to have a temporary solution for it, which should be this PR. My imagination is after the improved KIP merged, we can revert the solutions in this PR. And in users point of view, there is no any change when using v4.2 and v4.3. So my suggestion is: 1. in v4.2, we adopt the temporary solution in this PR 2. in v4.3 or later, we adopt the new way which gating in the leader controller side, which requires a KIP, and then revert this PR after KIP implemented. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3585878159 > Thanks for the changes @TaiJuWu. Leaving a review of the current approach in `src/main`, but please consider the below as an alternative approach (I think the below is simpler, but I would like @jsancio and @showuon 's input on it). In the interest of not duplicating effort, I think we should all agree on a design first. We agree that we need to reach a design consensus first. > > Sorry for the back and forth, but I think another, potentially easier way to go about this is actually to make the `ADD_RAFT_VOTER` sent by auto-join idempotent for the lifetime of the controller process. Basically, the leader will look through its voter set history and return a successful response if the requested node + incarnation ID was ever part of its voter set history. > > This approach requires a KIP, because we need to send the incarnation ID as part of the request, and the VotersRecord schema needs to be updated (meaning we need a kraft.version=2 as well I think), but it is much easier to prove this approach's correctness. If we want to do this change, I think using epoch is more reasonable. When the voter set change , we bump epoch but not to elect new leader, is it a possible solution? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2568505975
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -497,11 +501,31 @@ public void initialize(
kafkaRaftMetrics,
externalKRaftMetrics
);
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added
to voter set at offset {}",
Review Comment:
need to change level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2568445112
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
kafkaRaftMetrics,
externalKRaftMetrics
);
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added
to voter set at offset {}",
+ localReplicaKey, offset);
+hasJoined = Optional.of(true);
+}
+}
+});
+
// Read the entire log
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent() && canBecomeVoter && quorumConfig.autoJoin()
+&& isVoter(ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)))
{
+hasJoined = Optional.of(true);
+}
+
Review Comment:
It is not super obvious, but I explained here why this code is incorrect:
https://github.com/apache/kafka/pull/20859#pullrequestreview-3506989795
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3372,8 +3393,8 @@ private boolean
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
* and are configured to auto join should attempt to automatically
join the voter
* set for the configured topic partition.
*/
-return !hasJoined &&
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter
-&& quorumConfig.autoJoin() &&
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
+return hasJoined.isPresent() && !hasJoined.get() &&
partitionState.lastKraftVersion().isReconfigSupported()
Review Comment:
We need to add `!hasFetchTimeoutExpired() && local node LEO >= leader HWM`
to this check.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -1810,18 +1836,16 @@ private boolean handleFetchResponse(
OptionalLong.empty() :
OptionalLong.of(partitionResponse.highWatermark());
updateFollowerHighWatermark(state, highWatermark);
-if (initHighWatermark < 0 && partitionResponse.highWatermark()
>= 0) {
-initHighWatermark = partitionResponse.highWatermark();
-
-state.highWatermark().ifPresent(hw -> {
-if (hw.offset() >= initHighWatermark &&
nodeId.isPresent()) {
-hasJoined =
partitionState.lastVoterSet().isVoter(
-ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId));
-}
-}
-);
+if (nodeId.isPresent() && logEndOffset() >=
partitionResponse.highWatermark()) {
Review Comment:
We should only make these state changes when auto join is enabled.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
kafkaRaftMetrics,
externalKRaftMetrics
);
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
Review Comment:
We should only set this listener when autojoin is enabled and we are on a
controller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3585479651 Rewrite code from design by @kevin-wu24 , if this change is acceptable, I will add test tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3584352866 > The follower can only move to HAS_JOINED via processing a FETCH_RESPONSE, which is too late (i.e. the local node has already sent an "incorrect" auto-join). > IMO this is the harder case to handle, and handling this on the leader side via KAFKA-19933 is kind of "wrong" too. Yes, so I think as long as KAFKA-19933 is implemented, we can make sure the observer cannot join the voters successfully due to `leaderState.isReplicaCaughtUp()` is false. And with the addVoter request sent, the UpdateVoterSet timer will be reset and fetch request will be sent next. > We might need to add another clause in `shouldSendAddOrRemoveVoter: && local node LEO >= leader HWM`. I can understand and agree with this check, the network partition issue will be resolved. But alternatively, I think we can also fix the issue by only improving `KAFKA-19933` without this `shouldSendAddOrRemoveVoter: && local node LEO >= leader HWM` change. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2567200843
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the
node id of this node,
+// mark it as already joined because it is already in the voter
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}
Review Comment:
Alternatively, you can set `hasJoined = Optional.of(true)` when the leader
is elected and `hasJoined.isEmpty()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2567198996
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the
node id of this node,
+// mark it as already joined because it is already in the voter
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}
Review Comment:
```
if (canBecomeVoter && quorumConfig.autoJoin() &&
partitionState.lastVoterSet().size() == 1
&&
partitionState.lastVoterSet().isVoter(ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId)) {
hasJoined = Optional.of(true);
```
is the correct check. The code you have always evaluates to true because the
left hand side of the || is always true for a controller in `intialize()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2567198996
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the
node id of this node,
+// mark it as already joined because it is already in the voter
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}
Review Comment:
```
if (quorumConfig.autoJoin() && partitionState.lastVoterSet().size() == 1
&&
partitionState.lastVoterSet().isVoter(ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId)) {
hasJoined = Optional.of(true);
```
is the correct check. The code you have always evaluates to true because the
left hand side of the || is always true for a controller in `intialize()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2567177319
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the
node id of this node,
+// mark it as already joined because it is already in the voter
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}
Review Comment:
Sorry, from my understanding, this code should like below but after this
change it always true for controller.
```
// Q: After this condition, hasJoin always true
if (nodeId.isPresent() && (hasJoined.isEmpty()) ||
partitionState.lastVoterSet().size() == 1
&&
partitionState.lastVoterSet().isVoter(ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId)))
{
// if voter set is a singleton set contain only local ReplicaKey
// it is the standalone bootstrap node
hasJoined = Optional.of(true);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3584172984 > we can resolve this issue because the state will move to HAS_JOINED when finding the voter contains itself. But we can handle the issue separately since it's not directly related to this issue. Opened [KAFKA-19933](https://issues.apache.org/jira/browse/KAFKA-19933). The follower can only move to HAS_JOINED via processing a FETCH_RESPONSE, which is too late (i.e. the local node has already sent an "incorrect" auto-join). In the first scenario, based on whether the `UpdateVoterSet` timer is expired or not, the follower will either send an "incorrect" AddVoterRequest, or send a FetchRequest (this will lead to the correct state assuming we get a response). IMO this is the harder case to handle, and handling this on the leader side via `KAFKA-19933` is kind of "wrong" too. I think the observer needs to look at its fetch timer, and if that is expired, sending a FETCH takes priority over running the auto-join algorithm. That way, long-partitioned followers will fetch first, and avoid the above case. The proposed fix here is to add a `&& !hasFetchTimeoutExpired()` to `shouldSendAddOrRemoveVoter`. I think my previous assertion about the conditions of transitioning from `HAS_NOT_JOINED -> HAS_JOINED` are incorrect. If the local node C, which currently believes the voter set as (A,B) and is in `HAS_NOT_JOINED` state, is so far behind it requires two fetches, and the first fetch does not contain any new VotersRecords, but the second fetch would contain the records (A,B,C) and (A,B), local node C would still try to auto-join "incorrectly"... We might need to add another clause in `shouldSendAddOrRemoveVoter`: `&& local node LEO >= leader HWM`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3584117480 > Voter set is (A,B) with leader A. Node C is a caught up observer and tries to add itself. The new voter set (A,B,C) is committed by A and B, but C is partitioned and never learns its "auto-join" succeeded. The user removes C, and the voter set is back to (A,B). When the partition goes away, if updateVoterSet timer is expired, node C's shouldSendAddOrRemoveVoter evaluates to true and it will try to auto-join again. > I think in AddVoterHandler we check that the node being added is "caught up," but that check currently returns true if the node has fetched within the last hour (we can make this more strict obviously). Nice find! And I agree we can make the "caught up" criteria more strict to make sure the new added voter already knows the voters change, and then adopt the voter sets track in partitionState.updateState() solution below, we can resolve this issue because the state will move to `HAS_JOINED` when finding the voter contains itself. But we can handle the issue separately since it's not directly related to this issue. Opened [KAFKA-19933](https://issues.apache.org/jira/browse/KAFKA-19933). > Voter set is (A,B) with leader A. Node C is a caught up observer and tries to add itself. The new voter set (A,B,C) is committed by A and B, but C is partitioned and never learns its "auto-join" succeeded. The user removes C, and the voter set is back to (A,B). This is similar to the case above, but when the partition goes away, suppose node C's shouldSendAddOrRemoveVoter evaluates to false, and node C fetches instead of trying to auto-join. If this fetch contains both voter set changes in the same response (the latest voter set is (A,B)), the current code will still believe node C is in the HAS_NOT_JOINED state, and will attempt to auto-join again because we are only checking partitionState.lastVoterSet(). > I still think we can handle this properly in the code, and it means we need transition HAS_NOT_JOINED -> HAS_JOINED if any VotersRecord from the FETCH_RESPONSE contains the local ReplicaKey. We go through the new voter sets in partitionState.updateState(), and remember we do not need the HWM here because to get to the HAS_NOT_JOINED state we had to be "caught up" at that point in time. Another good find! (How could you come up these edge cases :) ) Yes, I think tracking the voters set change in `partitionState.updateState()` should work. @TaiJuWu , So in short, we have to do: 1. When a node is the leader and the state is `UNKNOWN`, we make it to `HAS_JOINED` 2. When a node is the follower (both voter and observer) and state is `UNKNOWN`, wait until the HWM updated, then check if the state is `HAS_JOINED` or `HAS_NOT_JOINED`. 3. Implement the tracking the voters set change in `partitionState.updateState()` solution 4. If in (3), we found the state is `HAS_NOT_JOINED` and local ReplicaKey appear in the voters set, we should change the state to `HAS_JOINED`. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3584110211 @showuon regarding: > This will not work because if the node lag a lot. The 1st fetch might not be able to catch up with the leader. Where in the code can this situation happen? I was under the impression that when a lagging follower node sends a fetch request to the leader, the leader does one of two things if there is no diverging log: 1. If the leader's log start offset is greater than the followers fetch offset, the leader sends a response telling the follower to fetch a snapshot. 2. Otherwise, send all the records until the end of the log in the fetch response: https://github.com/apache/kafka/blob/trunk/raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java#L1625. This means when a node receives a fetch response, they get the uncommitted state (if it exists) on the leader. Am I missing something? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3583469356 Hmmm, the more I think about this change the more complicated I find this new proposed semantic to be. Here is another case I thought about: Voter set is (A,B) with leader A. Node C is a caught up observer and tries to add itself. The new voter set (A,B,C) is committed by A and B, but C is partitioned and never learns its "auto-join" succeeded. The user removes C, and the voter set is back to (A,B). When the partition goes away, if `updateVoterSet` timer is expired, node C's `shouldSendAddOrRemoveVoter` evaluates to `true` and it will try to auto-join again. I think in `AddVoterHandler` we check that the node being added is "caught up," but that check currently returns true if the node has fetched within the last hour (we can make this more strict obviously). Another thing we need to consider is that multiple voter set changes may occur in the same `FetchRequestResponse`. For example, we could have the following: Voter set is (A,B) with leader A. Node C is a caught up observer and tries to add itself. The new voter set (A,B,C) is committed by A and B, but C is partitioned and never learns its "auto-join" succeeded. The user removes C, and the voter set is back to (A,B). This is similar to the case above, but when the partition goes away, suppose node C's `shouldSendAddOrRemoveVoter` evaluates to `false`, and node C fetches instead of trying to auto-join. If this fetch contains both voter set changes in the same response (the latest voter set is (A,B)), the current code will still believe node C is in the `HAS_NOT_JOINED` state, and will attempt to auto-join again because we are only checking `partitionState.lastVoterSet()`. I still think we can handle this properly in the code, and it means we need transition `HAS_NOT_JOINED -> HAS_JOINED` if any `VotersRecord` from the `FETCH_RESPONSE` contains the local `ReplicaKey`. We go through the new voter sets in `partitionState.updateState()`, and remember we do not need the HWM here because to get to the `HAS_NOT_JOINED` state we had to be "caught up" at that point in time. I want to highlight the above two cases to say that despite the desired semantic seeming very simple (auto-join at most once per process lifetime), implementing this properly is not straightforward at all, since there are a ton of edge cases because our local node's voter set can be out-of-date by an arbitrary number of VotersRecords. Additionally, at least for me, trying to verify that this implementation does indeed match our desired semantic is just as difficult (I do think I am out of edge cases now... no promises though 😆 ). Let me know your thoughts @TaiJuWu @showuon @jsancio. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565548226
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the
node id of this node,
+// mark it as already joined because it is already in the voter
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}
Review Comment:
H. Actually, we need to set `hasJoined = Optional.of(true)` IFF
`hasJoined.isEmpty()` for the first leader elected, even if it is not a
standalone leader. Otherwise, if we remove the leader, it can auto-join back
because the state can go from UNKNOWN -> HAS_NOT_JOINED if the user manually
removes the leader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean hasJoined = false;
Review Comment:
I have an idea how we can model this state properly. We can use an
Optional. Now we can represent 3 distinct "hasJoined" states:
1. `UNKNOWN`: We'll use Optional.empty(), which applies to all nodes on
startup, including brokers. Brokers, static controller quorums, or dynamic
controllers without auto-join enabled will always remain in this state.
2. `HAS_NOT_JOINED`: We'll use Optional.of(false), which only controllers
with auto-join can transition to.
3. `HAS_JOINED`: We'll use Optional.of(true), which only controllers with
auto-join can transition to.
The possible state transitions for controllers who have auto-join enabled
are:
1. `UNKNOWN -> HAS_JOINED`: This occurs when the controller handles a fetch
response, is "up-to-date" with the HWM, and discovers it IS in the voter set.
This transition can also occur when the controller is the only member of the
voter set when starting up (standalone controller), or if the controller wins
the initial election.
2. `UNKNOWN -> HAS_NOT_JOINED`: This occurs when the controller handles a
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the
voter set.
3. `HAS_NOT_JOINED -> HAS_JOINED`: This also occurs when the controller
handles a fetch response, and discovers it is in the voter set. However, we do
not need to do the HWM check here because the transition from `UNKNOWN ->
HAS_NOT_JOINED` has already made the check (i.e. we know we are up to date).
Importantly, state transition 2 can only occur once in the lifetime of the
controller, because we cannot transition to UNKNOWN from any state during the
lifetime of the process. Additionally, we cannot transition from `HAS_JOINED ->
HAS_NOT_JOINED` ever.
`shouldSendAddOrRemoveVoter` will be conditioned on the state being
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its
process lifetime, a controller will not try to auto-join again after being
removed from the voter set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean hasJoined = false;
Review Comment:
I have an idea how we can model this state properly. We can use an
Optional. Now we can represent 3 distinct "hasJoined" states:
1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on
startup, including brokers. Brokers, static controller quorums, or dynamic
controllers without auto-join enabled will always remain in this state.
2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers with
auto-join can transition to.
3. HAS_JOINED: We'll use Optional.of(true), which only controllers with
auto-join can transition to.
The possible state transitions for controllers who have auto-join enabled
are:
1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch
response, is "up-to-date" with the HWM, and discovers it IS in the voter set.
This transition can also occur when the controller is the only member of the
voter set when starting up (standalone controller), or if the controller wins
the initial election.
2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the
voter set.
3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller
handles a fetch response, and discovers it is in the voter set. However, we do
not need to do the HWM check here because the transition from UNKNOWN ->
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
Importantly, state transition 2 can only occur once in the lifetime of the
controller, because we cannot transition to UNKNOWN during the lifetime of the
process. Additionally, the controller process cannot go from HAS_JOINED ->
HAS_NOT_JOINED ever.
`shouldSendAddOrRemoveVoter` will be conditioned on the state being
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its
process lifetime, a controller will not try to auto-join again after being
removed from the voter set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean hasJoined = false;
Review Comment:
I have an idea how we can model this state properly. We can use an
Optional. Now we can represent 3 distinct "hasJoined" states:
1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on
startup, including brokers
2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers with
auto-join can transition to.
3. HAS_JOINED: We'll use Optional.of(true), which only controllers with
auto-join can transition to.
The possible state transitions for controllers who have auto-join enabled
are:
1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch
response, is "up-to-date" with the HWM, and discovers it IS in the voter set.
This transition can also occur when the controller is the only member of the
voter set when starting up (standalone controller), or if the controller wins
the initial election.
2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the
voter set.
3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller
handles a fetch response, and discovers it is in the voter set. However, we do
not need to do the HWM check here because the transition from UNKNOWN ->
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
Importantly, state transition 2 can only occur once in the lifetime of the
controller, because we cannot transition to UNKNOWN during the lifetime of the
process. Additionally, the controller process cannot go from HAS_JOINED ->
HAS_NOT_JOINED ever.
`shouldSendAddOrRemoveVoter` will be conditioned on the state being
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its
process lifetime, a controller will not try to auto-join again after being
removed from the voter set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean hasJoined = false;
Review Comment:
I have an idea how we can model this state properly. We can use an
Optional. Now we can represent 3 distinct "hasJoined" states:
1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on
startup, including brokers
2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers with
auto-join can transition to.
3. HAS_JOINED: We'll use Optional.of(true), which only controllers with
auto-join can transition to.
The possible state transitions for controllers who have auto-join enabled
are:
1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch
response, is "up-to-date" with the HWM, and discovers it IS in the voter set,
or if the controller is the only member of the voter set when starting up.
2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the
voter set.
3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller
handles a fetch response, and discovers it is in the voter set. However, we do
not need to do the HWM check here because the transition from UNKNOWN ->
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
Importantly, state transition 2 can only occur once in the lifetime of the
controller, because we cannot transition to UNKNOWN during the lifetime of the
process. Additionally, the controller process cannot go from HAS_JOINED ->
HAS_NOT_JOINED ever.
`shouldSendAddOrRemoveVoter` will be conditioned on the state being
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its
process lifetime, a controller will not try to auto-join again after being
removed from the voter set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565548226
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the
node id of this node,
+// mark it as already joined because it is already in the voter
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}
Review Comment:
H. Actually, we need to set `hasJoined = Optional.of(true)` IFF
`hasJoined.isEmpty()` for the first leader elected, even if it is not a
standalone leader. Otherwise, if we remove the leader, it can auto-join back
because the state can go from UNKNOWN -> HAS_NOT_JOINED.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565548226
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the
node id of this node,
+// mark it as already joined because it is already in the voter
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}
Review Comment:
H. Actually, we need to set `hasJoined = Optional.of(true)` for the
first leader elected, even if it is not a standalone leader. Otherwise, if we
remove the leader, it can auto-join back because the state can go from UNKNOWN
-> HAS_NOT_JOINED.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean hasJoined = false;
Review Comment:
I have an idea how we can model this state properly. We can use an
Optional. Now we can represent 3 distinct "hasJoined" states:
1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on
startup, including brokers
2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers can
transition to.
3. HAS_JOINED: We'll use Optional.of(true), which only controllers can
transition to.
The possible state transitions for controllers who have auto-join enabled
are:
1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch
response, is "up-to-date" with the HWM, and discovers it IS in the voter set,
or if the controller is the only member of the voter set when starting up.
2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the
voter set.
3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller
handles a fetch response, and discovers it is in the voter set. However, we do
not need to do the HWM check here because the transition from UNKNOWN ->
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
Importantly, state transition 2 can only occur once in the lifetime of the
controller, because we cannot transition to UNKNOWN during the lifetime of the
process. Additionally, the controller process cannot go from HAS_JOINED ->
HAS_NOT_JOINED ever.
`shouldSendAddOrRemoveVoter` will be conditioned on the state being
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its
process lifetime, a controller will not try to auto-join again after being
removed from the voter set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean hasJoined = false;
Review Comment:
I have an idea how we can model this state properly. We can use an
Optional. Now we can represent 3 distinct "hasJoined" states:
1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on
startup, including brokers
2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers can
transition to.
3. HAS_JOINED: We'll use Optional.of(true), which only controllers can
transition to.
The possible state transitions for controllers are:
1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch
response, is "up-to-date" with the HWM, and discovers it IS in the voter set,
or if the controller is the only member of the voter set when starting up.
2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the
voter set.
3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller
handles a fetch response, and discovers it is in the voter set. However, we do
not need to do the HWM check here because the transition from UNKNOWN ->
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
Importantly, state transition 2 can only occur once in the lifetime of the
controller, because we cannot transition to UNKNOWN during the lifetime of the
process. Additionally, the controller process cannot go from HAS_JOINED ->
HAS_NOT_JOINED ever.
`shouldSendAddOrRemoveVoter` will be conditioned on the state being
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its
process lifetime, a controller will not try to auto-join again after being
removed from the voter set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565460133
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -1805,6 +1809,19 @@ private boolean handleFetchResponse(
OptionalLong highWatermark = partitionResponse.highWatermark()
< 0 ?
OptionalLong.empty() :
OptionalLong.of(partitionResponse.highWatermark());
updateFollowerHighWatermark(state, highWatermark);
+
+if (initHighWatermark < 0 && partitionResponse.highWatermark()
>= 0) {
+initHighWatermark = partitionResponse.highWatermark();
+
+state.highWatermark().ifPresent(hw -> {
+if (hw.offset() >= initHighWatermark &&
nodeId.isPresent()) {
+hasJoined =
partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId));
+}
+}
+);
Review Comment:
hmmm, this if check seems incorrect. Doesn't this mean we won't ever enter
this method again if the leader sends us a partitionResponse.highWatermark() >=
0?
We should be checking the `highWatermark` against our local LEO, and if our
LEO >= HWM, the local node considers itself "caught up" and can set the
`hasJoined` state to the proper value. I believe this is correct because we can
only have at most one uncommitted voters record at a time.
Importantly, we need to enforce the state transition requirements in my
comment above here.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -1805,6 +1809,19 @@ private boolean handleFetchResponse(
OptionalLong highWatermark = partitionResponse.highWatermark()
< 0 ?
OptionalLong.empty() :
OptionalLong.of(partitionResponse.highWatermark());
updateFollowerHighWatermark(state, highWatermark);
+
+if (initHighWatermark < 0 && partitionResponse.highWatermark()
>= 0) {
+initHighWatermark = partitionResponse.highWatermark();
+
+state.highWatermark().ifPresent(hw -> {
+if (hw.offset() >= initHighWatermark &&
nodeId.isPresent()) {
+hasJoined =
partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId));
+}
+}
+);
Review Comment:
hmmm, this if check seems incorrect. Doesn't this mean we won't ever enter
this method again if the leader sends us a `partitionResponse.highWatermark()
>= 0`?
We should be checking the `highWatermark` against our local LEO, and if our
LEO >= HWM, the local node considers itself "caught up" and can set the
`hasJoined` state to the proper value. I believe this is correct because we can
only have at most one uncommitted voters record at a time.
Importantly, we need to enforce the state transition requirements in my
comment above here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean hasJoined = false;
Review Comment:
I have an idea how we can model this state properly. We can use an
Optional. Now we can represent 3 distinct "hasJoined" states:
1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on
startup, including brokers
2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers can
transition to.
3. HAS_JOINED: We'll use Optional.of(true), which only controllers can
transition to.
The possible state transitions for controllers are:
1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch
response, is "up-to-date" with the HWM, and discovers it IS in the voter set,
or if the controller is the only member of the voter set when starting up.
2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the
voter set.
3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller
handles a fetch response is "up-to-date" with the HWM, and discovers it is in
the voter set.
Importantly, state transition 2 can only occur once in the lifetime of the
controller, because we cannot transition to UNKNOWN during the lifetime of the
process. Additionally, the controller process cannot go from HAS_JOINED ->
HAS_NOT_JOINED ever.
`shouldSendAddOrRemoveVoter` will be conditioned on the state being
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its
process lifetime, a controller will not try to auto-join again after being
removed from the voter set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565395726
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3285,6 +3301,9 @@ private void
prospectiveTransitionAfterElectionLoss(ProspectiveState prospective
private long pollFollower(long currentTimeMs) {
FollowerState state = quorum.followerStateOrThrow();
if (quorum.isVoter()) {
+if (quorumConfig.autoJoin()) {
+hasJoined = true;
+}
Review Comment:
We can remove this code.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the
node id of this node,
+// mark it as already joined because it is already in the voter
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}
Review Comment:
Here we should only set `hasJoined = Optional.of(true)` IFF the
partitionState.lastVoterSet() is a singleton set containing only the local
`ReplicaKey`.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3352,8 +3371,8 @@ private boolean
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
* and are configured to auto join should attempt to automatically
join the voter
* set for the configured topic partition.
*/
-return partitionState.lastKraftVersion().isReconfigSupported() &&
canBecomeVoter &&
-quorumConfig.autoJoin() &&
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
+return !hasJoined &&
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter
Review Comment:
```suggestion
return hasJoined.isPresent() && !hasJoined.get() &&
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter
```
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean hasJoined = false;
Review Comment:
I have an idea how we can model this state properly. We can use an
Optional. Now we can represent 3 distinct "hasJoined" states:
1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on
startup, including brokers
2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers can
transition to.
3. HAS_JOINED: We'll use Optional.of(true), which only controllers can
transition to.
The possible state transitions for controllers are:
1. UNKNOWN -> HAS_JOINED: This occurs when the controller fetches, is
"up-to-date" with the HWM, and discovers it IS in the voter set, or if the
controller is the only member of the voter set when starting up.
2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller fetches, is
"up-to-date" with the HWM, and discovers it is NOT in the voter set.
3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller
fetches, is "up-to-date" with the HWM, and discovers it is in the voter set.
Importantly, state transition 2 can only occur once in the lifetime of the
controller, because we cannot transition to UNKNOWN during the lifetime of the
process. Additionally, the controller process cannot go from HAS_JOINED ->
HAS_NOT_JOINED ever.
`shouldSendAddOrRemoveVoter` will be conditioned on the state being
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its
process lifetime, a controller will not try to auto-join again after being
removed from the voter set.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -1805,6 +1809,19 @@ private boolean handleFetchResponse(
OptionalLong highWatermark = partitionResponse.highWatermark()
< 0 ?
OptionalLong.empty() :
OptionalLong.of(partitionResponse.highWatermark());
updateFollowerHighWatermark(state, highWatermark);
+
+if (initHighWatermark < 0 && partitionResponse.highWatermark()
>= 0) {
+initHighWatermark = partitionResponse.highWatermark();
+
+state.highWatermark().ifPresent(hw -> {
+if (hw.offset() >= initHighWatermark &&
nodeId.isPresent()) {
+
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3581768804 > > Consider the following case, which IMO results in equally confusing UX: > > We have a voter set of (A,B,C) with leader A, KIP-853, and auto-join enabled. We remove node C via RemoveRaftVoter, and the new (A,B) VotersRecord is committed (i.e. A and B both persist it). However, C could be partitioned and not replicate the (A,B) VotersRecord, meaning its partitionState.lastVoterSet() == (A,B,C). Node C restarts and enters initialize(), and it will set hasAutoJoined = true. In this case, the user "expects" C to auto-join because when they look at the quorum status, the voter set is (A,B) and C just restarted with auto-join enabled, but C will not attempt to auto-join. The point here is that A, B, and the user believe the voter set is (A,B), whereas C believes the voter set is (A,B,C) and changes its state based on it. > > This is a really good point @kevin-wu24 ! Yes, the state in the `KafkaRaftClient#initialize` is just the `the VotersRecord with the highest offset in local node`. So in the case of network partition above, the state will not be update-to-date. Actually, not only the network partition case, suppose the node C is down, and during the period, C is removed from the voters set. After node C startup, the initial voters set is also [A,B,C]. So I think we should fix this issue. > > > I'm not sure how to fix this case at the moment, and need to think about how to set hasAutoJoined properly. Maybe this value starts as true and the local node can set it to false after successfully fetching and finding out it is not a voter? > > This will not work because if the node lag a lot. The 1st fetch might not be able to catch up with the leader. > > I think what we need to do is to make sure the node is caught up with the leader controller. By making sure it is caught up, the network partition case above can be resolved because once we catch up with the leader, we know the `latest voter state on startup`. And we can identify the node caught up or not by checking the highwatermark in fetch response from leader. The flow is like this (using the situation above): > > 1. node C starts up with network partition (voters = [ABC]) > 2. network partition resolved > 3. node C sends the 1st fetch request to the leader > 4. leader responds with highwatermark=100 > 5. After appending with the fetch response data, the LEO in the node C is 80 (not caught up yet) > 6. Node C sends 2nd fetch request > 7. The LEO moves to 102 (suppose the leader has new data appended), which is >= 100 > 8. Node C updates the `hasJoin` value with the latest voters set at this moment (i.e. voters = [A, B]), and `hasJoined=false`) > 9. Node C auto joined the voters set due to `hasJoined=false` > > @TaiJuWu @kevin-wu24 , does this make sense to you? It make sense to me, address it by https://github.com/apache/kafka/pull/20859/commits/61984cf3d8119f0953d04abfd9b582d21734d015 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
ppatierno commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3580795668 @kevin-wu24 thank you very much for looking into this! So to answer your questions ... > Question for @ppatierno regarding their use case: Just to confirm my understanding on the issue: when you remove the controller role from these nodes, is the operator "watching" the config file, and then sends the RemoveRaftVoter based on that? And then you end up in this state where the combined "controllers" are continuously being removed + added back because of auto-join? I am not sure what you mean by "is the operator "watching" the config file" but let me explain how it works in practice. What the operator detects (because the user has update the corresponding `KafkaNodePool` custom resource, which hosts info for the broker/controller nodes) is that it has to remove the controllers from the combined-nodes. So it knows what should be the new configuration for these nodes (i.e. removing "controller" from the `process.roles` property) and then it rolls them to allow restarting as broker only with the new configuration. Before rolling, it's going to remove them from voters list by using the RemoveRaftVoter Admin API. You can see here that with auto-join enabled, these controllers re-join immediately right after the RemoveRaftVoter Admin API and before rolling. So they would restart as broker-only but seeing them still in the voters list and try to send the UpdateVoteRequest (see the https://issues.apache.org/jira/browse/KAFKA-19867). > I think what you want is actually to first disable auto-join on the combined controller, restart it, and then remove it from the voter set. Then you are free to make it broker-only. This would make the controllers scaling more complex and longer, even because the operator should: * disable auto-join, restart the nodes still as "controller" (as combined nodes) * remove them from the voters set * re-configuring them by removing the "controller" (making broker only), restarting them. Two rollings while just one would be sufficient, and for a big cluster it's really a lot imho other than the complexity being added to the operator which would be not necessary thanks to this fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3580486460 > Consider the following case, which IMO results in equally confusing UX: > We have a voter set of (A,B,C) with leader A, KIP-853, and auto-join enabled. We remove node C via RemoveRaftVoter, and the new (A,B) VotersRecord is committed (i.e. A and B both persist it). However, C could be partitioned and not replicate the (A,B) VotersRecord, meaning its partitionState.lastVoterSet() == (A,B,C). Node C restarts and enters initialize(), and it will set hasAutoJoined = true. In this case, the user "expects" C to auto-join because when they look at the quorum status, the voter set is (A,B) and C just restarted with auto-join enabled, but C will not attempt to auto-join. The point here is that A, B, and the user believe the voter set is (A,B), whereas C believes the voter set is (A,B,C) and changes its state based on it. This is a really good point @kevin-wu24 ! Yes, the state in the `KafkaRaftClient#initialize` is just the `the VotersRecord with the highest offset in local node`. So in the case of network partition above, the state will not be update-to-date. Actually, not only the network partition case, suppose the node C is down, and during the period, C is removed from the voters set. After node C startup, the initial voters set is also [A,B,C]. So I think we should fix this issue. > I'm not sure how to fix this case at the moment, and need to think about how to set hasAutoJoined properly. Maybe this value starts as true and the local node can set it to false after successfully fetching and finding out it is not a voter? This will not work because if the node lag a lot. The 1st fetch might not be able to catch up with the leader. I think what we need to do is to make sure the node is caught up with the leader controller. By making sure it is caught up, the network partition case above can be resolved because once we catch up with the leader, we know the `latest voter state on startup`. And we can identify the node caught up or not by checking the highwatermark in fetch response from leader. The flow is like this (using the situation above): 1. node C starts up with network partition (voters = [ABC]) 5. network partition resolved 6. node C sends the 1st fetch request to the leader 7. leader responds with highwatermark=100 8. After appending with the fetch response data, the LEO in the node C is 80 (not caught up yet) 9. Node C sends 2nd fetch request 10. The LEO moves to 102 (suppose the leader has new data appended), which is >= 100 11. Node C updates the `hasJoin` value with the latest voters set at this moment (i.e. voters = [A, B]), and `hasJoined=false`) 12. Node C auto joined the voters set due to `hasJoined=false` @TaiJuWu @kevin-wu24 , does this make sense to you? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562686075
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark
it as already joined
Review Comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562640152
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2335,8 +2344,11 @@ private boolean handleAddVoterResponse(
/* These error codes indicate the replica was successfully added or
the leader is unable to
* process the request. In either case, reset the update voter set
timer to back off.
*/
-if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
-error == Errors.DUPLICATE_VOTER) {
+if (error == Errors.NONE) {
+
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasAutoJoined = true;
Review Comment:
Thanks your detailed explanation, I missed this part indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562519856
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark
it as already joined
Review Comment:
If we have a static voters set, controllers cannot auto-join because they
are on `kraft.verison=0`, not `kraft.version=1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562507354
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark
it as already joined
Review Comment:
If we use static voters for initialization, I believe this should not be
treated as a VotersRecord but rather as an internal variable within
partitionState. From that perspective, I think the comment is acceptable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562507354
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark
it as already joined
Review Comment:
If we use static voter for initialization , I think it is not voter record
just a variable in partitionState and I think this comment is ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2561593244
##
raft/src/main/java/org/apache/kafka/raft/QuorumConfig.java:
##
@@ -105,7 +105,7 @@ public class QuorumConfig {
public static final String QUORUM_AUTO_JOIN_ENABLE_CONFIG = QUORUM_PREFIX
+ "auto.join.enable";
public static final String QUORUM_AUTO_JOIN_ENABLE_DOC = "Controls whether
a KRaft controller should automatically " +
-"join the cluster metadata partition for its cluster id.";
+"join the cluster metadata partition for its cluster id when the node
startup.";
Review Comment:
"Controls whether a KRaft controller ( tuple) should
automatically join the cluster metadata partition as a voter during node
startup if the controller is not already a voter"
There is probably a better way to word the above.
This most accurate describes what we are changing the semantic too. I think
I am okay with this definition of the semantic from a UX POV. I guess we do not
really care if the `ReplicaKey` was added by auto-join or not, just that it was
added as a voter.
Our proposed semantic is essentially to make the auto-join code "at most
once" for the lifetime of a given controller process execution, conditioned on
the local ReplicaKey not being part of our voter set. However, this semantic
can run into UX issues too.
##
raft/src/test/java/org/apache/kafka/raft/KafkaRaftClientAutoJoinTest.java:
##
@@ -317,6 +321,131 @@ private void pollAndDeliverFetchToUpdateVoterSet(
context.client.poll();
}
+@Test
+public void testBootstrapVoterSetDoesNotSendAddVoterAfterRemove() throws
Exception {
+// Test that a bootstrap voter node, after being removed by
RemoveVoter RPC,
+// will NOT automatically rejoin (canAutoJoin remains false)
+final var leader = replicaKey(randomReplicaId(), true);
+final var follower = replicaKey(leader.id() + 1, true);
+final var thisVoter = replicaKey(follower.id() + 1, true);
+
+final int epoch = 1;
+final var context = new RaftClientTestContext.Builder(
+thisVoter.id(),
+thisVoter.directoryId().get()) // Same directory ID -> node is
voter
+.withRaftProtocol(KIP_853_PROTOCOL)
+.withStartingVoters(
+VoterSetTest.voterSet(Stream.of(leader, follower, thisVoter)),
KRaftVersion.KRAFT_VERSION_1
+)
+.withElectedLeader(epoch, leader.id())
+.withAutoJoin(true)
+.withCanBecomeVoter(true)
+.build();
+
+// Node should be a follower (voter) initially
+assertTrue(context.client.quorum().isFollower());
+
+// Complete initial fetch
+context.advanceTimeAndCompleteFetch(epoch, leader.id(), true);
+
+// Simulate this node being removed by RemoveVoter RPC (initiated by
another node)
+// Update voter set via fetch - this node is now removed from voter set
+pollAndDeliverFetchToUpdateVoterSet(
+context,
+epoch,
+VoterSetTest.voterSet(Stream.of(leader, follower)) // This node
is no longer in voter set
+);
+
+// Node should now be an observer after being removed
+assertTrue(context.client.quorum().isObserver());
+
+// Advance time to expire update voter set timer
+context.time.sleep(context.fetchTimeoutMs);
+
+// Poll and verify that NO AddVoter request is sent
+// With canAutoJoin=false (set during init because ID was in voter
set),
+// node should NOT automatically rejoin
+context.time.sleep(1);
+context.pollUntilRequest();
+
+// Should not include AddVoter
+assertEquals(0,
context.channel.drainSentRequests(Optional.of(ApiKeys.ADD_RAFT_VOTER)).size());
+
+}
+
+@Test
+public void testSuccessfulAddVoterSetsCanAutoJoinAgain() throws Exception {
+// Test scenario:
+// 1. Node initializes as observer
+// 2. Node joins as follower via AddVoter
+// 3. Node is removed via RemoveVoter and becomes observer again
+// 4. Node should NOT send AddVoter request again
+final var leader = replicaKey(randomReplicaId(), true);
+final var follower = replicaKey(leader.id() + 1, true);
+final var newVoter = replicaKey(follower.id() + 1, true);
+final int epoch = 1;
+final var context = new RaftClientTestContext.Builder(
+newVoter.id(),
+newVoter.directoryId().get())
+.withRaftProtocol(KIP_853_PROTOCOL)
+.withStartingVoters(
+VoterSetTest.voterSet(Stream.of(leader, follower)),
KRaftVersion.KRAFT_VERSION_1
+)
+.withElectedLeader(epoch, leader.id())
+.withAutoJoin(true)
+.withCanBecomeVoter(true)
+.build();
+
+// 1.
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2561465070
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2335,8 +2344,11 @@ private boolean handleAddVoterResponse(
/* These error codes indicate the replica was successfully added or
the leader is unable to
* process the request. In either case, reset the update voter set
timer to back off.
*/
-if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
-error == Errors.DUPLICATE_VOTER) {
+if (error == Errors.NONE) {
+
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasAutoJoined = true;
Review Comment:
Take a look at this code from `AddVoterHandler#handleApiVersionsResponse`:
```
current.setLastOffset(leaderState.appendVotersRecord(newVoters,
currentTimeMs));
if (!current.ackWhenCommitted()) { ***we execute the below when
auto-join is enabled***
// complete the future to send response, but do not reset the
state,
// since the new voter set is not yet committed
current.future().complete(RaftUtil.addVoterResponse(Errors.NONE,
null));
}
```
When auto-join is enabled, we send the RPC response BEFORE the new voters
record is committed. The motivation behind this is detailed in:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1186%3A+Update+AddRaftVoterRequest+RPC+to+support+auto-join,
and is necessary for a fully available auto-join feature.
What this means here is that our local node may not actually have
auto-joined, but we set `hasAutoJoined = true`. For example, what if the remote
leader fails over before the VotersRecord adding the local node is committed?
The local node has not added itself via auto-join, and will not until it
restarts, despite the user expecting it too.
The "more correct" place to set `hasAutoJoined = true` is in `pollFollower`
if we are in the `quorum.isVoter()` case AND `quorumConfig.autoJoin() == true`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2561429495
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark
it as already joined
+// because it is already in the voter set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853
properly
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(),
nodeDirectoryId);
+hasAutoJoined =
partitionState.lastVoterSet().isVoter(localReplicaKey);
Review Comment:
```suggestion
hasAutoJoined = partitionState.lastVoterSet().isVoter(
ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
);
```
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2335,8 +2344,11 @@ private boolean handleAddVoterResponse(
/* These error codes indicate the replica was successfully added or
the leader is unable to
* process the request. In either case, reset the update voter set
timer to back off.
*/
-if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
-error == Errors.DUPLICATE_VOTER) {
+if (error == Errors.NONE) {
+
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasAutoJoined = true;
Review Comment:
Take a look at this code from `AddVoterHandler#handleApiVersionsResponse`:
```
current.setLastOffset(leaderState.appendVotersRecord(newVoters,
currentTimeMs));
if (!current.ackWhenCommitted()) { ***we execute the below when
auto-join is enabled***
// complete the future to send response, but do not reset the
state,
// since the new voter set is not yet committed
current.future().complete(RaftUtil.addVoterResponse(Errors.NONE,
null));
}
```
When auto-join is enabled, we send the RPC response BEFORE the new voters
record is committed. The motivation behind this is detailed in:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1186%3A+Update+AddRaftVoterRequest+RPC+to+support+auto-join,
and is necessary for a fully available auto-join feature.
What this means here is that our local node may not actually have
auto-joined, but we set `hasAutoJoined = true`. For example, what if the remote
leader fails over before the VotersRecord adding the local node is committed?
The "more correct" place to set `hasAutoJoined = true` is in `pollFollower`
if we are in the `quorum.isVoter()` case AND `quorumConfig.autoJoin() == true`.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3352,8 +3364,13 @@ private boolean
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
* and are configured to auto join should attempt to automatically
join the voter
* set for the configured topic partition.
*/
-return partitionState.lastKraftVersion().isReconfigSupported() &&
canBecomeVoter &&
-quorumConfig.autoJoin() &&
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
+if (!partitionState.lastKraftVersion().isReconfigSupported() ||
!canBecomeVoter ||
+!quorumConfig.autoJoin() ||
!state.hasUpdateVoterSetPeriodExpired(currentTimeMs)) {
+return false;
+}
+
+// Only attempt auto-join if we haven't already auto-joined
+return !hasAutoJoined;
Review Comment:
```suggestion
return !hasAutoJoined &&
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter &&
quorumConfig.autoJoin() && state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
```
IMO this reads cleaner. It will also short circuit after a node successfully
auto-joins and sets `hasAutoJoined = true`, whereas this code evaluates all
boolean expressions always.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark
it as already joined
Review Comment:
> if the starting voters contain the node id of this node, mark it as
already joined
This comment is inaccurate. We are checking `lastVoterSet()`, which is the
VotersRecord with the highest
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3577632887 Hi @kevin-wu24 , thanks for your detailed review and suggestions, all comments are addressed, PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859: URL: https://github.com/apache/kafka/pull/20859#discussion_r2561448757 ## docs/upgrade.html: ## @@ -171,8 +171,8 @@ Notable changes in 4 -The controller.quorum.auto.join.enable has been added to QuorumConfig, enabling KRaft controllers to automatically join the cluster's voter set, -and defaults to false. +The controller.quorum.auto.join.enable has been added to QuorumConfig, enabling KRaft controllers to automatically join the cluster's voter set +when the node startup, and defaults to false. Review Comment: Update it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2560048766
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean canAutoJoin = true;
Review Comment:
It seems to me that the intention of the change is the following semantic:
For each lifetime of the controller process on a KRaft node, the node should
be in the "auto-joining" state at most once (i.e. Observer state &&
`shouldSendAddOrRemoveVoterRequest(state, currentTimeMs)`). This means when we
remove a node and it goes back to Observer state, the removed node will not
automatically join back via `AddVoterRPC`. However, if the user removes the
node as a voter and then the node restarts, the node will still try to
auto-join the voter set.
I think a better name for this boolean is `hasAutoJoined` with a value on
startup of `false`. In `initialize()`, each controller node checks their
`partitionState.lastVoterSet()` and sets `hasAutoJoined = true` if they are a
part of the voter set already. Whenever an observer that has auto-join enabled
successfully joins the voter set, we can set `hasAutoJoined = true`.
One issue with the proposed semantic is that we don't know if previous
additions to the voter set were a result of auto-join or manually adding the
voter. Remember that the `auto.join.enable` config can also change when a node
is restarted. I'm not sure how big of an issue that is, but there are a lot of
edge cases around this feature that lead to confusing UX (i.e. this node is
auto-joining even though I don't "expect" it to) if we try to implement a
semantic like the above. That is one reason why we wanted to keep the
pre-requisite for executing the auto-join code simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2560048766
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean canAutoJoin = true;
Review Comment:
It seems to me that the intention of the change is the following semantic:
For each lifetime of the controller process on a KRaft node, the node should
be in the "auto-joining" state at most once (i.e. Observer state &&
`shouldSendAddOrRemoveVoterRequest(state, currentTimeMs)`). This means when we
remove a node and it goes back to Observer state, the removed node will not
automatically join back via `AddVoterRPC`. However, if the user removes the
node as a voter and then the node restarts, the node will still try to
auto-join the voter set.
I think a better name for this boolean is `hasAutoJoined` with a value on
startup of `false`. In `initialize()`, each controller node checks their
`partitionState.lastVoterSet()` and sets `hasAutoJoined = true` if they are a
part of the voter set already. Whenever an observer that has auto-join enabled
successfully joins the voter set, we can set `hasAutoJoined = true`.
The main problem with this functionality is that we don't know if previous
additions to the voter set were a result of auto-join or manually adding the
voter. Remember that the `auto.join.enable` config can also change when a node
is restarted. I'm not sure how big of an issue that is, but there are a lot of
edge cases around this feature that lead to confusing UX (i.e. this node is
auto-joining even though I don't "expect" it to) if we try to implement a
semantic like the above. That is one reason why we wanted to keep the
pre-requisite for executing the auto-join code simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2560048766
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean canAutoJoin = true;
Review Comment:
It seems to me that the intention of the change is the following semantic:
For each lifetime of the controller process on a KRaft node, the node should
be in the "auto-joining" state at most once (i.e. Observer state &&
`shouldSendAddOrRemoveVoterRequest(state, currentTimeMs)`). This means when we
remove a node and it goes back to Observer state, the removed node will not
automatically join back via `AddVoterRPC`. However, if the user removes the
node as a voter and then the node restarts, the node will still try to
auto-join the voter set.
I think a better name for this boolean is `hasAutoJoined` with a value on
startup of `false`. In `initialize()`, each controller node checks their
`partitionState.lastVoterSet()` and sets `hasAutoJoined = true` if they are a
part of the voter set already. Whenever an observer that has auto-join enabled
successfully joins the voter set, we can set `hasAutoJoined = true`.
The main problem with this functionality is that we don't know if previous
additions to the voter set were a result of auto-join or manually adding the
voter. I'm not sure how big of an issue that is, but there are a lot of edge
cases around this feature that lead to confusing UX (i.e. this node is
auto-joining even though I don't "expect" it to) if we have a semantic like the
above. That is one reason why we wanted to keep the pre-requisite for executing
the auto-join code simple.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2560073825
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3356,30 +3366,56 @@ private boolean
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
quorumConfig.autoJoin() &&
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
}
+private boolean shouldSendAddVoterRequest(FollowerState state, long
currentTimeMs) {
+return canAutoJoin && shouldAutoJoin(state, currentTimeMs);
+}
+
+private boolean shouldSendRemoveVoterRequest(FollowerState state, long
currentTimeMs) {
+final var localReplicaKey = quorum.localReplicaKeyOrThrow();
+final var voters = partitionState.lastVoterSet();
+
Review Comment:
In my opinion, these are too many boolean states and they are very difficult
to reason about. If we add an additional `&& !hasAutoJoined` to the
`shouldSendAddOrRemoveVoterRequest` predicate and ensure `hasAutoJoined` is set
properly, we can keep all of the existing auto-join code the same.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3347,7 +3357,7 @@ private long pollFollowerAsVoter(FollowerState state,
long currentTimeMs) {
);
}
-private boolean shouldSendAddOrRemoveVoterRequest(FollowerState state,
long currentTimeMs) {
+private boolean shouldAutoJoin(FollowerState state, long currentTimeMs) {
Review Comment:
Let's keep this method name as is.
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,11 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, it
can't auto join to the cluster
+// because it is already in.
+canAutoJoin =
!partitionState.lastVoterSet().voterIds().contains(nodeId.getAsInt());
+}
Review Comment:
I think I understand what this is trying to do. Basically, if the local node
is already part of the voter set we should avoid sending `AddVoterRPC`.
However, there is an issue with this code:
With KIP-853 enabled, a voter is no longer just a single Integer ID, but
rather a tuple of `` represented by the `ReplicaKey` class, so
checking the ID is not sufficient. Look at `QuorumState#isVoter()` for the
right way to do this check.
##
docs/upgrade.html:
##
@@ -171,8 +171,8 @@ Notable changes in 4
-The controller.quorum.auto.join.enable has been added to
QuorumConfig, enabling KRaft controllers to automatically join the
cluster's voter set,
-and defaults to false.
+The controller.quorum.auto.join.enable has been added to
QuorumConfig, enabling KRaft controllers to automatically join the
cluster's voter set
+when the node startup, and defaults to false.
Review Comment:
We should also add the documentation from the KIP here to make this more
clear:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217391519#KIP853:KRaftControllerMembershipChanges-controller.quorum.auto.join.enable
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements
RaftClient {
private volatile RemoveVoterHandler removeVoterHandler;
private volatile UpdateVoterHandler updateVoterHandler;
+private volatile boolean canAutoJoin = true;
Review Comment:
It seems to me that the intention of the change is the following semantic:
For each lifetime of the controller process on a KRaft node, the node should
be in the "auto-joining" state at most once (i.e. Observer state &&
`shouldSendAddOrRemoveVoterRequest(state, currentTimeMs)`). This means when we
remove a node and it goes back to Observer state, the removed node will not
automatically join back via `AddVoterRPC`. However, if the user removes the
node as a voter and then the node restarts, the node will still try to
auto-join the voter set.
I think a better name for this boolean is `hasAutoJoined` with a value on
startup of `false`. In `initialize()`, each controller node checks their
`partitionState.lastVoterSet()` and sets `hasAutoJoined = true` if they are a
part of the voter set already. Whenever an observer that has auto-join enabled
successfully joins the voter set, we can set `hasAutoJoined = true`.
The main problem with this functionality is that we don't know if previous
additions to the voter set were a result of auto-join or manually adding the
voter. I'm not sure how big of an issue that is, but there are a lot of edge
cases around this feature if we have
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
kevin-wu24 commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3573689124 @TaiJuWu I can take a look tomorrow. Thanks for the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3573683953 @kevin-wu24 @jsancio , do you want to have another look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on PR #20859: URL: https://github.com/apache/kafka/pull/20859#issuecomment-3570844088 > LGTM! @TaiJuWu , please update the PR description. Thanks. Updated description and merge trunk, thanks for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2549453089
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId =
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send
AddVoter request automatically.
+Thread.sleep(2000);
+
+// Verify 3002 is already fetch and does not send add voter
request
+long removedAtHighWatermark =
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong();
+TestUtils.waitForCondition(() ->
+
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
> removedAtHighWatermark,
+30_000, 100, () -> "High watermark is not advanced in
3ms"
+);
+
+// 3002 does not join the voter set after high watermark
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+}
Review Comment:
Nice suggestion, add it to the end of this test but the Set\
comes from line 253 to avoiding NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2549453089
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId =
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send
AddVoter request automatically.
+Thread.sleep(2000);
+
+// Verify 3002 is already fetch and does not send add voter
request
+long removedAtHighWatermark =
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong();
+TestUtils.waitForCondition(() ->
+
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
> removedAtHighWatermark,
+30_000, 100, () -> "High watermark is not advanced in
3ms"
+);
+
+// 3002 does not join the voter set after high watermark
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+}
Review Comment:
Nice suggestion, add it to the end of this test but the Set\
comes from line 3002 to avoiding NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2549453089
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId =
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send
AddVoter request automatically.
+Thread.sleep(2000);
+
+// Verify 3002 is already fetch and does not send add voter
request
+long removedAtHighWatermark =
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong();
+TestUtils.waitForCondition(() ->
+
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
> removedAtHighWatermark,
+30_000, 100, () -> "High watermark is not advanced in
3ms"
+);
+
+// 3002 does not join the voter set after high watermark
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+}
Review Comment:
Nice suggestion, add it to the end of this test but the Set
comes from line 3002 to avoiding NPE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2549453089
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId =
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send
AddVoter request automatically.
+Thread.sleep(2000);
+
+// Verify 3002 is already fetch and does not send add voter
request
+long removedAtHighWatermark =
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong();
+TestUtils.waitForCondition(() ->
+
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
> removedAtHighWatermark,
+30_000, 100, () -> "High watermark is not advanced in
3ms"
+);
+
+// 3002 does not join the voter set after high watermark
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+}
Review Comment:
Nice suggestion, add it to the end of this test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548678971
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId =
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send
AddVoter request automatically.
+Thread.sleep(2000);
Review Comment:
1. We can add a variable for the 2 sec to explain it. Ex:
`defaultUpdateVoterSetPeriodTimeout = 2000`.
2. It's "update voter set timer".
3. `We need to wait for the update voter set timer expiring to allow the
addVoter request to be sent.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548669325
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId =
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
Review Comment:
This can be removed now because we already do the assertion below.
##
raft/src/test/java/org/apache/kafka/raft/KafkaRaftClientAutoJoinTest.java:
##
@@ -317,6 +321,131 @@ private void pollAndDeliverFetchToUpdateVoterSet(
context.client.poll();
}
+@Test
+public void testBootstrapVoterSetDoesNotSendAddVoterAfterRemove() throws
Exception {
Review Comment:
Nice tests added in KafkaRaftClientAutoJoinTest. Thanks.
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId =
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000,
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548605894
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,74 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
Review Comment:
Good catch, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548531651
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,74 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
Review Comment:
This work, though, it's not clear because if the voters update takes some
time, you could retry many times. And on each retry, you'll try to remove voter
3002 again and again. We can do something similar like
[this](https://github.com/apache/kafka/blob/trunk/server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java#L179-L186).
It will remove voter once, and retry the validation until the voter updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548463582
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() ->
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in
3ms"
+);
+
+// 3002 does not join the voter set after high watermark
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
Review Comment:
Sorry, I was wrong. In L277, we already verify that the voters are only
[3000, 3001], so it is fine now. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548357630
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() ->
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in
3ms"
+);
+
+// 3002 does not join the voter set after high watermark
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
Review Comment:
This is not right. Here, we want to verify `3002 does not join the voter set
after high watermark advance`, but you're testing if [3000, 3001] is in the
voters set. So, if 3002 is joined the voters, the test can still pass, right?
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546323712
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different
directory ID.
+// This simulates the case where the controller failed and is brought
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+// Remove 3002 from voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be removed
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// do not join the voter set in the next twenty seconds
+for (int i = 0; i < 20; ++i) {
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, ()
-> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be
removed
+return;
+}
+
+admin.removeRaftVoter(3002,
voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+});
+Thread.sleep(1000);
+}
Review Comment:
> This test doesn't make sense to me.
>
> 1. `do not join the voter set in the next twenty seconds`: The 20 seconds
is not true because the `retryOnExceptionWithTimeout` could take up to 30
seconds one time.
It make sense to me and time-base test is not a good thing as you say bellow.
> 2. Why should we use 2 `retryOnExceptionWithTimeout` here?
My mistake.
> 3. The test is not testing what we expected.
>3.1. What are you asserting here? I can only see you assert the node
3000, 3001 are in the voters, but we want to test 3002, right? If 3002 is in
the voters, the test can still be passed!?
From my understanding, only the voter keys (3000 and 3001) are in the
cluster is the same meaning as 3002 is removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546716203
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different
directory ID.
+// This simulates the case where the controller failed and is brought
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
Review Comment:
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546691075
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() ->
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in
3ms"
+);
Review Comment:
@showuon Your suggestion make sense to me but there is an issue which is the
high watermark is tuned base on different compute.
If I use
```
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
> removedAtHighWatermark.get()
```
The original code is still passed, you can use
b0136c1f8c8beb3ce853a83f9b43f9d1ed623e9b to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546691075
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() ->
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in
3ms"
+);
Review Comment:
@showuon Your suggestion make sense to me but there is an issue which is the
high watermark is tuned base on different compute.
If I use
```
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
> removedAtHighWatermark.get()
```
The original code is still passed, you can use
b0136c1f8c8beb3ce853a83f9b43f9d1ed623e9b to check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546222726
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3356,30 +3366,56 @@ private boolean
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
quorumConfig.autoJoin() &&
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
}
+private boolean shouldSendAddVoterRequest(FollowerState state, long
currentTimeMs) {
+return canAutoJoin && maybeAutoJoin(state, currentTimeMs);
+}
+
+private boolean shouldSendRemoveVoterRequest(FollowerState state, long
currentTimeMs) {
+final var localReplicaKey = quorum.localReplicaKeyOrThrow();
+final var voters = partitionState.lastVoterSet();
+
+if (voters.voterIds().contains(localReplicaKey.id())) {
+if (maybeAutoJoin(state, currentTimeMs)) {
+// When the bootstrap controller needs to update directory id,
+// it should be removed and then rejoining to cluster
+// In such a case, we should set canAutoJoin to true, and it
will
+// be removed from the cluster, update directory id and rejoin
+// to the cluster.
Review Comment:
The disk failure is not only the case, one case is at line 3403.
```
/* The replica's id is in the voter set but the replica is not a voter
because
* the directory id of the voter set entry is different. Remove the old
voter.
* Local replica is not in the voter set because the replica is an observer.
*/
```
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() ->
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in
3ms"
+);
Review Comment:
@showuon Your suggestion make sense to me but there is an issue which is the
high watermark is tuned base on different compute.
If I use
```
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
> removedAtHighWatermark.get()
```
The original code is still passed.
##
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2544638061
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different
directory ID.
+// This simulates the case where the controller failed and is brought
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+// Remove 3002 from voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be removed
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+
+// do not join the voter set in the next twenty seconds
+for (int i = 0; i < 20; ++i) {
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, ()
-> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be
removed
+return;
+}
+
+admin.removeRaftVoter(3002,
voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+});
+Thread.sleep(1000);
+}
Review Comment:
We can test won't auto join the removed node by:
1. Remove a controller node, ex: 3002.
2. Verify the highwatermark in controller metadata log is advancing by:
`cluster.raftManagers().get(3002)..client().highWatermark()`
The high watermark increasing means the follower is fetching the data from
the leader, and because we auto join the controller node before fetching from
the leader, if the node can be auto joined, it'll be already added.
3. Verify the voters are still 3000, 3001, without 3002.
So, we don't need the mysterious 20 times loop to do the voter removal.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2544580605
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different
directory ID.
+// This simulates the case where the controller failed and is brought
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
Review Comment:
What is this for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2544517164
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different
directory ID.
+// This simulates the case where the controller failed and is brought
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(),
voters.get(replicaId));
+}
+});
+
+// Remove 3002 from voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be removed
+return;
+}
Review Comment:
We should fail the test in this unexpected case, right?
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,11 @@ public void initialize(
logger.info("Reading KRaft snapshot and log as part of the
initialization");
partitionState.updateState();
logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if starting voters contain node id, it can't join to the cluster
+// because it is already in.
Review Comment:
nit: if [the] starting voters contain [the node id of this node], it can't
[auto] join to the cluster ...
##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter()
throws Exception {
}
}
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different
directory ID.
+// This simulates the case where the controller failed and is brought
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG,
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+
Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]
ppatierno commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2542831283
##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3356,30 +3366,56 @@ private boolean
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
quorumConfig.autoJoin() &&
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
}
+private boolean shouldSendAddVoterRequest(FollowerState state, long
currentTimeMs) {
+return canAutoJoin && maybeAutoJoin(state, currentTimeMs);
+}
+
+private boolean shouldSendRemoveVoterRequest(FollowerState state, long
currentTimeMs) {
+final var localReplicaKey = quorum.localReplicaKeyOrThrow();
+final var voters = partitionState.lastVoterSet();
+
+if (voters.voterIds().contains(localReplicaKey.id())) {
+if (maybeAutoJoin(state, currentTimeMs)) {
+// When the bootstrap controller needs to update directory id,
+// it should be removed and then rejoining to cluster
+// In such a case, we should set canAutoJoin to true, and it
will
+// be removed from the cluster, update directory id and rejoin
+// to the cluster.
Review Comment:
if I got it correctly, it can happen when, for example, for any reasons the
controller has a new directory ID (i.e. disk failure, ...) but it's registered
within the voters with the old directory ID. For this reason, it needs to
remove itself with the old directory ID and re-join with the new directory ID.
Can we improve the comment by specify the "disk failure" as an example of a new
directory id and why an update is needed (because of voters still include the
old directory ID)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
