Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-02 Thread via GitHub


TaiJuWu commented on PR #21025:
URL: https://github.com/apache/kafka/pull/21025#issuecomment-3604818647

   > > Thanks for the feature @TaiJuWu but I don't think we should change the 
semantic of the KRaft protocol without a KIP.
   > > I understand that the auto join feature doesn't meet your requirements. 
Have you considered not using the auto join feature and instead implementing 
controller addition and removal in the orchestration layer using 
Admin:addRaftVoter, Admin::removeRaftVoter and Admin::describeMetadataQuorum?
   > 
   > Thanks for the comments @jsancio ! I agree if we can't come up a good 
solution for this issue, it's also fine to keep what we have so far. But, I 
think we should at least improve the documentation and mention about this known 
UX issue. @TaiJuWu , could you help on this? Thanks.
   
   Sure. Will update this PR for documentation only.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-02 Thread via GitHub


showuon commented on PR #21025:
URL: https://github.com/apache/kafka/pull/21025#issuecomment-3604763210

   > Thanks for the feature @TaiJuWu but I don't think we should change the 
semantic of the KRaft protocol without a KIP.
   > 
   > I understand that the auto join feature doesn't meet your requirements. 
Have you considered not using the auto join feature and instead implementing 
controller addition and removal in the orchestration layer using 
Admin:addRaftVoter, Admin::removeRaftVoter and Admin::describeMetadataQuorum?
   
   Thanks for the comments @jsancio ! I agree if we can't come up a good 
solution for this issue, it's also fine to keep what we have so far. But, I 
think we should at least improve the documentation and mention about this known 
UX issue. @TaiJuWu , could you help on this? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-02 Thread via GitHub


jsancio commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2582197596


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2335,9 +2337,12 @@ private boolean handleAddVoterResponse(
 /* These error codes indicate the replica was successfully added or 
the leader is unable to
  * process the request. In either case, reset the update voter set 
timer to back off.
  */
-if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
-error == Errors.DUPLICATE_VOTER) {
+if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT) {
+
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+return true;
+} else if (error == Errors.DUPLICATE_VOTER) {
 
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasJoined = true;

Review Comment:
   This doesn't seem correct. You want to have the has joined boolean because 
true when the voter is successfully added to the voter set.



##
raft/src/main/java/org/apache/kafka/raft/internals/KRaftControlRecordStateMachine.java:
##
@@ -302,6 +310,7 @@ private void handleBatch(Batch batch, OptionalLong 
overrideOffset) {
 case KRAFT_VOTERS:
 VoterSet voters = VoterSet.fromVotersRecord((VotersRecord) 
record.message());
 kafkaRaftMetrics.updateNumVoters(voters.size());
+seenReplicaKey.addAll(voters.voterKeys());

Review Comment:
   Kafka should not have data structures that only grow in size. E.g. there is 
no remove operation against `seenReplicaKey`. This means that the size of this 
data structure is unbounded and will continue to grow even if the voter set is 
as small as 3 to 5 voters.



##
raft/src/main/java/org/apache/kafka/raft/internals/AddVoterHandler.java:
##
@@ -156,6 +156,23 @@ public CompletableFuture 
handleAddVoterRequest(
 );
 }
 
+
+if (!ackWhenCommitted && partitionState.hasSeenReplicaKey(voterKey)) {
+// ackWhenCommitted is used to distinguish the request from admin
+// or broker. If it comes from broker and had been in cluster, we 
should
+// reject it.

Review Comment:
   This is peculiar association. Why should "ack when committed" requests fail 
if the voter key is in the log?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


TaiJuWu commented on PR #21025:
URL: https://github.com/apache/kafka/pull/21025#issuecomment-3600652601

   @showuon Thanks for your sharing. 
   Do we consider to apply the other solution which is adding a configuration 
between two auto-join? 
   We can make it a private at 4.2 and public at 4.3.  WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


showuon commented on PR #21025:
URL: https://github.com/apache/kafka/pull/21025#issuecomment-3600448983

   @jsancio @kevin-wu24 , I found the current solution has one drawback, which 
is that the snapshot only contains `the latest voters at the time the snapshot 
taken`. That means we lost the voters change history before the snapshot. This 
could cause the issue:
   1. controller 1, 2, 3 are the voters
   2. controller 3 is removed, so the current voters are [1, 2]
   3. controller 3 cannot be auto-joined because the leader controller found 3 
has ever been in the voter.
   4. The leader controller takes a snapshot for metadata log
   5. The leader controller restarts, and load the snapshot + records after 
snapshot, which gets the voter history contains only [1, 2]
   6. controller 3 restarts, addVoterRequest is sent because auto-join is 
enabled, but this time it will succeed because the leader controller doesn't 
know controller 3 has ever been in the voters.
   
   My suggestion: 
   1. Keep the same semantic: `new controllers (node id + directory UUID 
tuples) will automatically join the KRaft voter set once if they have not been 
a voter before.`, but add a note in doc to notify users to shutdown the removed 
controller soon to avoid the node being auto-joined again in some edge cases.
   2. Change the semantic to: `Only do the auto-join when a node is start up 
first time`. The start up first time is defined by: LEO of metadata_log is 0. 
The use case of auto-join is original designed for new formatted controller 
node, so it should not be a problem. The drawback of this issue is that the 
auto-join can only have one shot if the first auto-join failed and restarted. 
For example:
 a. controller A is formatted, with auto-join enabled
 b. controller A startup, addVoterRequest is sent (We can initialize the 
updateVoter timer as expired state, and reset after response received)
 c. The leader controller responded with error
 d. controller A fetches metadata log from the leader controller (metadata 
log LEO > 0)
 e. Before controller A retrying the addVoter request, it crashes
 f. After restart, controller A will not auto send addVoter request anymore 
due to (metadata log LEO > 0)
   
   
   I think the suggestion (2) is better since it has no edge cases at all. We 
just need to make it clear that the auto-join can only happen when a node has 
empty metadata log. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


TaiJuWu commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2579479629


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2339,6 +2341,9 @@ private boolean handleAddVoterResponse(
 error == Errors.DUPLICATE_VOTER) {
 
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
 return true;
+} else if (error == Errors.INVALID_VOTER_KEY) {
+hasJoined = true;
+return true;

Review Comment:
   It was fixed, I made a mistake at that moment :(



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


TaiJuWu commented on PR #21025:
URL: https://github.com/apache/kafka/pull/21025#issuecomment-3600031084

   > Thanks for the PR @TaiJuWu .
   > 
   > This solution has the unfortunate side effect that nodes cannot leave and 
join the quorum. The user cannot do the following operations:
   > 
   > 1. First add voter A.
   > 2. Remove voter A, on purpose or by accident.
   > 3. Add voter A again. This should succeed but it will fail with this PR.
   
   Our integration test also cover such case, it test follow:
   1. observers can join cluster with auto-join
   2. it be removed by manual and does not send addVoterRequest twice again
   3. admin can add this node to the cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


TaiJuWu commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2579429259


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2338,6 +2340,7 @@ private boolean handleAddVoterResponse(
 if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
 error == Errors.DUPLICATE_VOTER) {
 
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasJoined = true;

Review Comment:
   Error



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


kevin-wu24 commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2579434322


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2339,6 +2341,9 @@ private boolean handleAddVoterResponse(
 error == Errors.DUPLICATE_VOTER) {
 
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
 return true;
+} else if (error == Errors.INVALID_VOTER_KEY) {
+hasJoined = true;
+return true;

Review Comment:
   The `DUPLICATE_VOTER` case should be its own `else if` clause.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


TaiJuWu commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2579429259


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2338,6 +2340,7 @@ private boolean handleAddVoterResponse(
 if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
 error == Errors.DUPLICATE_VOTER) {
 
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasJoined = true;

Review Comment:
   Error



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


showuon commented on PR #21025:
URL: https://github.com/apache/kafka/pull/21025#issuecomment-3599930001

   > Thanks for the PR @TaiJuWu .
   > 
   > This solution has the unfortunate side effect that nodes cannot leave and 
join the quorum. The user cannot do the following operations:
   > 
   > 1. First add voter A.
   > 
   > 2. Remove voter A, on purpose or by accident.
   > 
   > 3. Add voter A again. This should succeed but it will fail with this 
PR.
   
   @jsancio , basically, the current solution only block the `auto-join` add 
voter request if it's not the 1st time joined the voters. So in your operation 
example, the step (3) will succeed if it's `manual join` (with 
`AckWhenCommitted=true` in KIP-1186). And if in step (3), it's `auto-join` 
(with `AckWhenCommitted=false`), it will be rejected.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


kevin-wu24 commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2578058942


##
raft/src/main/java/org/apache/kafka/raft/internals/AddVoterHandler.java:
##
@@ -156,6 +156,23 @@ public CompletableFuture 
handleAddVoterRequest(
 );
 }
 
+
+if (!ackWhenCommitted && partitionState.hasSeenReplicaKey(voterKey)) {
+// ackWhenCommitted is used to distinguish the request from admin
+// or broker. If it comes from broker and had been in cluster, we 
should
+// reject it.
+return CompletableFuture.completedFuture(
+RaftUtil.addVoterResponse(
+Errors.INVALID_VOTER_KEY,

Review Comment:
   Return `Errors.DUPLICATE_VOTER`.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2339,6 +2341,9 @@ private boolean handleAddVoterResponse(
 error == Errors.DUPLICATE_VOTER) {
 
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
 return true;
+} else if (error == Errors.INVALID_VOTER_KEY) {
+hasJoined = true;
+return true;

Review Comment:
   We want to use `Errors.DUPLICATE_VOTER` not `Errors.INVALID_VOTER_KEY`, 
since the former error code means the requested node is already a part of the 
voter set. 
   
   Whenever the requesting node receives a `DUPLICATE_VOTER` response, it knows 
for sure it can turn off auto-join code.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2667,8 +2672,6 @@ private Optional maybeHandleCommonResponse(
 } else if (error == Errors.BROKER_NOT_AVAILABLE) {
 return Optional.of(false);
 } else if (error == Errors.INVALID_VOTER_KEY) {
-// The voter key in the request for VOTE and BEGIN_QUORUM_EPOCH 
doesn't match the

Review Comment:
   Please revert this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


ppatierno commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2576492118


##
raft/src/main/java/org/apache/kafka/raft/internals/AddVoterHandler.java:
##
@@ -156,6 +156,23 @@ public CompletableFuture 
handleAddVoterRequest(
 );
 }
 
+
+if (!ackWhenCommitted && partitionState.hasSeenReplicaKey(voterKey)) {
+// ackWhenCommitted is used to distinguish the request from admin
+// or broker. If it comes from broker and had been in cluster, we 
should
+// reject it.
+return CompletableFuture.completedFuture(
+RaftUtil.addVoterResponse(
+Errors.INVALID_VOTER_KEY,
+String.format(
+"The voter key for %s has joined the set of voters.",

Review Comment:
   ```suggestion
   "The voter key for %s has already joined the set of 
voters.",
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


showuon commented on code in PR #21025:
URL: https://github.com/apache/kafka/pull/21025#discussion_r2576407012


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2339,6 +2341,9 @@ private boolean handleAddVoterResponse(
 error == Errors.DUPLICATE_VOTER) {
 
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
 return true;
+} else if (error == Errors.INVALID_VOTER_KEY) {
+  hasJoined = true;
+  return true;

Review Comment:
   Indention is wrong.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


showuon commented on PR #21025:
URL: https://github.com/apache/kafka/pull/21025#issuecomment-3595624376

   @ppatierno @kevin-wu24 , FYI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-12-01 Thread via GitHub


TaiJuWu commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3595464034

   new PR https://github.com/apache/kafka/pull/21025


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-30 Thread via GitHub


TaiJuWu commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594746804

   I will open a new PR for this, thank!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-30 Thread via GitHub


TaiJuWu closed pull request #20859: KAFKA-19850: KRaft voter auto join will add 
a removed voter immediately
URL: https://github.com/apache/kafka/pull/20859


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-30 Thread via GitHub


TaiJuWu commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594464927

   > Kevin and I had an offline discussion about this, we agree:
   > 
   > 1. Go with server-side `ReplicaKey` idempotency approach. Even if the 
`UpdateVoterSetTimer` is not required in the client side, and the `LEO==0` 
check should work, but the client-side gating is still as not reliable as in 
the server-side.
   > 2. We can get the implementation ready first (without test), and waiting 
for other reviewers' comments, especially the KIP-853 author @jsancio 's 
comments.
   > 3. After everyone is onboard with this solution, we add tests and review 
and merge it in v4.2.0.
   > 
   > @TaiJuWu , let us know if you have any other thoughts. Thank you.
   
   I also agree your approach, that is really better than current design, will 
close this PR, thanks for your participation! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-30 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594454885

   Kevin and I had an offline discussion about this, we agree:
   
   1. Go with server-side `ReplicaKey` idempotency approach. Even if the 
`UpdateVoterSetTimer` is not required in the client side, and the `LEO==0` 
check should work, but the client-side gating is still as not reliable as in 
the server-side.
   2. We can get the implementation ready first (without test), and waiting for 
other reviewers' comments, especially the KIP-853 author @jsancio 's comments.
   3. After everyone is onboard with this solution, we add tests and review and 
merge it in v4.2.0.
   
   @TaiJuWu , let us know if you have any other thoughts. Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-30 Thread via GitHub


kevin-wu24 commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594416314

   > Then, we can easily check the log end offset == 0 in metadata log in 
kakfaRaftClient#initialize() in the client side. If LEO is 0, we know this node 
is a brand new node, we will auto-join it if auto-join is enabled. Otherwise, 
we won't do that. Is this much clear to resolve this issue?
   
   Here is a case where this client side implementation would run into poor UX. 
New ReplicaKey X formats and starts up. It is in the "auto-joining" mode, 
performs a fetch first, writes the records to disk, and then restarts. If the 
above occurs before the `UpdateVoterSetTimer` expires and the client polls, the 
client has no way of turning auto-join on during subsequent restarts. In this 
case, the user expects a node to auto-join but it doesn't because of the 
process restarting.
   
   I believe the server-side `ReplicaKey` idempotency approach is best.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-30 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594404590

   > @showuon, the active controller can distinguish between a auto-join 
AddVoter request and a manual AddVoter request because of 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1186%3A+Update+AddRaftVoterRequest+RPC+to+support+auto-join.
 This KIP was needed as part of the initial implementation to make auto-join 
fully available. When a ReplicaKey auto-joins and is then manually removed, it 
will no longer auto-join again, but you can still add it manually. Hopefully 
that clears things up.
   
   Thanks for the update, @kevin-wu24 ! That looks good. 
   
   I have another thought. How about the `auto-join` is only allowed when the 
node _firstly_ start up. Basically, the purpose of the auto-join is to allow a 
node to easily join the voters after storage formatting, right? Then, we can 
easily check the `log end offset == 0 in metadata log` in 
`kakfaRaftClient#initialize()` in the client side. If LEO is 0, we know this 
node is a brand new node, we will auto-join it if `auto-join is enabled`. 
Otherwise, we won't do that. Is this much clear to resolve this solution?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-30 Thread via GitHub


kevin-wu24 commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594356326

   > I think the the semantic will change to Once a node(ReplicaKey) is removed 
it will never be able to join the voter again. However, we already allow one 
node join/remove multiple times before v4.2, so I don't think we can do that 
now without a KIP.
   
   @showuon, the active controller can distinguish between a auto-join 
`AddVoter` request and a manual `AddVoter` request because of 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1186%3A+Update+AddRaftVoterRequest+RPC+to+support+auto-join.
 This KIP was needed as part of the initial implementation to make auto-join 
fully available. When a ReplicaKey auto-joins and is then manually removed, it 
will no longer auto-join again, but you can still add it manually. Hopefully 
that clears things up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-30 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3594348819

   > Maybe a better documentation is that new controllers (node id + directory 
UUID tuples) will automatically join the KRaft voter set once if they have not 
been a voter before.
   
   @kevin-wu24 , I think the biggest problem is that the server side (active 
controller) doesn't know if an observer is doing `auto-join` or `manual join`. 
I think the the semantic will change to `Once a node(ReplicaKey) is removed it 
will never be able to join the voter again`. However, we already allow one node 
join/remove multiple times before v4.2, so I don't think we can do that now.
   
   I think the suggestion sounds good as long as we can provide a way to 
distinguish `auto-join` or `manual join` in the addVoter request, but that 
needs a KIP... :(
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-28 Thread via GitHub


kevin-wu24 commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3589283023

   > Does it count for the initial controllers or not? Does it count for 
manually join?
   Yes for both. When handling an auto-join `AddVoterRPC`, the leader will 
check its voter set history if the requesting `ReplicaKey` was part of it. 
   
   Maybe a better documentation is that new controllers (node id + directory 
UUID tuples) will automatically join the KRaft voter set once if they are not 
already a voter.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587892645

   > Why make the proposed semantic idempotent per (ReplicaKey + process 
incarnation) tuple? I would argue that having the node try to auto-join on 
restart is just as confusing UX. In the original use case, if the node just 
gets restarted, we end up in the same place of the controller rejoining the 
voter set when we don't "want" it to (i.e. after an explicit removal).
   
   Yes, I know what you mean. I don't have a good answer on this TBH. I just 
know it's better than "auto-joining a just removed voter".
   
   > Another approach is we can make auto-join idempotent per ReplicaKey and 
handle this server-side. I think this approach best captures the intent of the 
feature and maintains a good UX. Basically, when I startup a new observer 
controller with auto-join, it will join the quorum automatically at most once 
for the lifetime of its ReplicaKey. After an explicit removal, the leader can 
return a special error for a subsequent auto-join request from the same 
ReplicaKey to tell the node to stop sending AddVoter. If the controller's disk 
fails and it restarts, it will have a new ReplicaKey, and will auto-join again 
(this is the intended behavior because it is essentially a completely new 
controller). This does not require a KIP, and would be simpler to implement 
than either of the discussed approaches.
   
   
   1. `make auto-join idempotent per ReplicaKey ` -> +1
   2. `it will join the quorum automatically at most once for the lifetime of 
its ReplicaKey` -> I don't know how we define the `automatically join`. Does it 
count for the initial controllers or not? Does it count for `manually join`? 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


kevin-wu24 commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587856896

   > What I'm afraid is that we shipped in v4.2 with "the aggressive auto-join 
semantic even if one node is just removed from the voters set". Then in v4.3, 
the semantic suddenly changes to "auto-join only when startup". It will confuse 
users and even surprise users. I think we should decide which semantic we want 
to deliver first, instead of deliver it then change it.
   
   Yeah, that makes sense to me. 
   
   One thing that confuses me with the proposed semantic from the JIRA is: 
   
   Why make the proposed semantic idempotent per (`ReplicaKey` + process 
incarnation) tuple? I would argue that having the node try to auto-join on 
restart is just as confusing UX. In the original use case, if the node just 
gets restarted, we end up in the same place of the controller rejoining the 
voter set when we don't "want" it to (i.e. after an explicit removal).
   
   Another approach is we can make auto-join idempotent per `ReplicaKey` and 
handle this server-side. I think this approach best captures the intent of the 
feature and maintains a good UX. Basically, when I startup a new observer 
controller with auto-join, it will join the quorum automatically at most once 
for the lifetime of its `ReplicaKey`. If the controller's disk fails and it 
restarts, it will have a new `ReplicaKey`, and will auto-join again. This does 
not require a KIP, and would be simpler to implement than either of the 
discussed approaches.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587821188

   Thanks for the comment @kevin-wu24 !
   
   > My opinion is we should go with the KIP-design of server-side idempotency 
for auto-join additions if we are going to implement this semantic change at 
all. We should not try to hastily implement this change just to land it in 4.2, 
especially if we intend to just revert this change and do the server-side 
idempotency in 4.3. Remember that bugs with this implementation will also lead 
to confusing/bad UX. If we want to change the semantic, we should do it just 
once, with what we believe to be the best design + implementation.
   
   What I'm afraid is that we shipped in v4.2 with "the aggressive auto-join 
semantic even if one node is just removed from the voters set". Then in v4.3, 
the semantic suddenly changes to "auto-join only when startup". It will confuse 
users and even surprise users. I think we should decide which semantic we want 
to deliver first, instead of deliver it then change it. 
   
   > I believe we have gotten in KIPs after KIP-freeze before right? This KIP 
would be self-contained within KRaft module and only affect the auto-join 
feature, so maybe we can make this an exception and get this in for 4.2.
   
   I'm not sure if it will be accepted or not. Need the release manager's 
decision.
   
   > I don't see the harm in this waiting until 4.3, so we can give this change 
in semantic the proper time for review. Let's also wait and see what @jsancio 
thinks about all this too.
   
   I'd also like to hear @jsancio 's thought on this.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


TaiJuWu commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587811957

   > I believe we have gotten in KIPs after KIP-freeze before right? This KIP 
would be self-contained within KRaft module and only affect the auto-join 
feature, so maybe we can make this an exception and get this in for 4.2.
   
   I am happy to see there is a solution with KIP, actually it is difficult to 
resolve in client side (as you mentioned edge cases)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2570363560


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
 kafkaRaftMetrics,
 externalKRaftMetrics
 );
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already 
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added 
to voter set at offset {}",
+   localReplicaKey, offset);
+hasJoined = Optional.of(true);
+}
+}
+});
+
 // Read the entire log
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
 
+if (nodeId.isPresent() && canBecomeVoter && quorumConfig.autoJoin()
+&& isVoter(ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId))) 
{
+hasJoined = Optional.of(true);
+}
+

Review Comment:
   You need to wait for a fetch to complete, so that the starting voters can 
complete a fetch loop and transition from `UNKNOWN -> HAS_JOINED`.
   
   In `testBootstrapVoterSetDoesNotSendAddVoterAfterRemove`, I am pretty sure 
that `advanceTimeAndCompleteFetch` just does an empty fetch. You need to 
actually perform a fetch that contains the starting voters in the response. The 
code you currently have sets up a state where the initial local voter set state 
defined by `withStartingVoters` is "out-of-date" with the first fetch loop 
performed, so the node thinks it is in `HAS_NOT_JOINED` and tries to auto-join, 
which is the intended behavior of the design.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


kevin-wu24 commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587775523

   Thanks for the discussion @TaiJuWu and @showuon.
   
   > But for v4.2, I think we still need to have a temporary solution for it, 
which should be this PR. My imagination is after the improved KIP merged, we 
can revert the solutions in this PR. And in users point of view, there is no 
any change when using v4.2 and v4.3.
   
   But this implementation will still exist in the released 4.2 branch + 
binary. Again, although I've mentioned some cases, it is difficult to reason 
about the correctness of this client-side design. 
   
   My opinion is we should go with the KIP-design of server-side idempotency 
for auto-join additions if we are going to implement this semantic change at 
all. We should not try to hastily implement this change just to land it in 4.2, 
especially if we intend to just revert this change and do the server-side 
idempotency in 4.3. Remember that bugs with this implementation will also lead 
to confusing/bad UX. If we want to change the semantic, we should do it just 
once, with what we believe to be the best design + implementation. 
   
   I believe we have gotten in KIPs after KIP-freeze before right? This KIP 
would be self-contained within KRaft module and only affect the auto-join 
feature, so maybe we can make this an exception and get this in for 4.2.
   
   I hope my previous comments have shown that trying to implement this 
functionality client-side is not straightforward at all. Just because I have 
identified edge cases that the current design handles does not mean the current 
design handles all possible cases. I also will not have as much time to review 
this change like I had this week as we come up on 4.2 code freeze. I don't see 
the harm in this waiting until 4.3, so we can give this change in semantic the 
proper time for review. Let's also wait and see what @jsancio thinks about all 
this too.
   
   About your question @showuon:
   > We might need to add another clause in shouldSendAddOrRemoveVoter: && 
local node LEO >= leader HWM.
   > 
   > I can understand and agree with this check, the network partition issue 
will be resolved. But alternatively, I think we can also fix the issue by only 
improving KAFKA-19933 without this shouldSendAddOrRemoveVoter: && local node 
LEO >= leader HWM change. WDYT?
   
   Maybe? I can offer the following but I still need to think about this more. 
I am pretty sure that we need to implement KAFKA-19933 for the current design, 
but I do not know what that new "is caught up" check should be at the moment. I 
think the check on the leader is `auto-joing node's fetch offset >= leader's 
previous HWM`, but what is "previous HWM" here? This is yet another reason to 
do just the server-side approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2570324985


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
 kafkaRaftMetrics,
 externalKRaftMetrics
 );
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already 
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added 
to voter set at offset {}",
+   localReplicaKey, offset);
+hasJoined = Optional.of(true);
+}
+}
+});
+
 // Read the entire log
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
 
+if (nodeId.isPresent() && canBecomeVoter && quorumConfig.autoJoin()
+&& isVoter(ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId))) 
{
+hasJoined = Optional.of(true);
+}
+

Review Comment:
   @TaiJuWu , about your comment:
   1. Even if this is the temporary solution for v4.2, we still need to make it 
right, otherwise, this is not a `good  solution` at all.
   2. If the change will fail the tests or make them flaky, we should:
 a. figure it out if it is expected or not
 b. update the tests to make it work as what we expected
   
   I appreciate you stay with us for this long time. Please let us know if you 
need any help. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2570289277


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
 kafkaRaftMetrics,
 externalKRaftMetrics
 );
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already 
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added 
to voter set at offset {}",
+   localReplicaKey, offset);
+hasJoined = Optional.of(true);
+}
+}
+});
+
 // Read the entire log
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
 
+if (nodeId.isPresent() && canBecomeVoter && quorumConfig.autoJoin()
+&& isVoter(ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId))) 
{
+hasJoined = Optional.of(true);
+}
+

Review Comment:
   You are right here but if this is just quick fix for 4.2, I think this is ok 
for temporary solution, otherwise the new add test from me 
(`testRemovedControllerWontJoinAgain` and 
`testBootstrapVoterSetDoesNotSendAddVoterAfterRemove`) will flaky / fail which 
is not I expected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2570278282


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3372,8 +3393,8 @@ private boolean 
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
  * and are configured to auto join should attempt to automatically 
join the voter
  * set for the configured topic partition.
  */
-return !hasJoined && 
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter
-&& quorumConfig.autoJoin() && 
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
+return hasJoined.isPresent() && !hasJoined.get() && 
partitionState.lastKraftVersion().isReconfigSupported()

Review Comment:
   `hasJoined.isPresent` is also mean the local node catch hight watermark, 
right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2570278282


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3372,8 +3393,8 @@ private boolean 
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
  * and are configured to auto join should attempt to automatically 
join the voter
  * set for the configured topic partition.
  */
-return !hasJoined && 
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter
-&& quorumConfig.autoJoin() && 
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
+return hasJoined.isPresent() && !hasJoined.get() && 
partitionState.lastKraftVersion().isReconfigSupported()

Review Comment:
   `hasJoined.isPresent` is also mean, the local node catch hight watermark, 
right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3587572361

   > Thanks for the changes @TaiJuWu. Leaving a review of the current approach 
in `src/main`, but please consider the below as an alternative approach (I 
think the below is simpler, but I would like @jsancio and @showuon 's input on 
it). In the interest of not duplicating effort, I think we should all agree on 
a design first.
   > 
   > Sorry for the back and forth, but I think another, potentially easier way 
to go about this is actually to make the `ADD_RAFT_VOTER` sent by auto-join 
idempotent for the lifetime of the controller process. Basically, the leader 
will look through its voter set history and return a successful response if the 
requested node + incarnation ID was ever part of its voter set history.
   > 
   > This approach requires a KIP, because we need to send the incarnation ID 
as part of the request, and the VotersRecord schema needs to be updated 
(meaning we need a kraft.version=2 as well I think), but it is much easier to 
prove this approach's correctness.
   > 
   > You can think of our current approach as trying to handle every single 
edge case on the client-side, which is difficult to reason about because of 
partitioning + async nature of replication, whereas this approach makes our 
state change operation idempotent server-side, so the leader does not care if 
the client (a given controller process incarnation + `ReplicaKey` tuple) sends 
additional add voter requests.
   
   @kevin-wu24 , I'm +1 on having a KIP to have the gate in the leader 
controller side. After all, the controllers addition/removal are all handling 
in the leader controller side. I agree that will be much reliable and clean way 
to handle it. 
   
   But for v4.2, I think we still need to have a temporary solution for it, 
which should be this PR. My imagination is after the improved KIP merged, we 
can revert the solutions in this PR. And in users point of view, there is no 
any change when using v4.2 and v4.3. 
   
   So my suggestion is:
   1. in v4.2, we adopt the temporary solution in this PR
   2. in v4.3 or later, we adopt the new way which gating in the leader 
controller side, which requires a KIP, and then revert this PR after KIP 
implemented.
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


TaiJuWu commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3585878159

   > Thanks for the changes @TaiJuWu. Leaving a review of the current approach 
in `src/main`, but please consider the below as an alternative approach (I 
think the below is simpler, but I would like @jsancio and @showuon 's input on 
it). In the interest of not duplicating effort, I think we should all agree on 
a design first.
   
   We agree that we need to reach a design consensus first.
   
   > 
   > Sorry for the back and forth, but I think another, potentially easier way 
to go about this is actually to make the `ADD_RAFT_VOTER` sent by auto-join 
idempotent for the lifetime of the controller process. Basically, the leader 
will look through its voter set history and return a successful response if the 
requested node + incarnation ID was ever part of its voter set history.
   > 
   
   > This approach requires a KIP, because we need to send the incarnation ID 
as part of the request, and the VotersRecord schema needs to be updated 
(meaning we need a kraft.version=2 as well I think), but it is much easier to 
prove this approach's correctness.
   
   If we want to do this change, I think using epoch is more reasonable.
   
   When the voter set change , we bump epoch but not to elect new leader, is it 
a possible solution?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2568505975


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -497,11 +501,31 @@ public void initialize(
 kafkaRaftMetrics,
 externalKRaftMetrics
 );
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already 
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added 
to voter set at offset {}",

Review Comment:
   need to change level.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2568445112


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
 kafkaRaftMetrics,
 externalKRaftMetrics
 );
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {
+// We dont need to check high watermark here since it already 
check by
+// hasJoined is not empty.
+if (nodeId.isPresent() && hasJoined.isPresent()) {
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId);
+if (voterSet.isVoter(localReplicaKey) && !hasJoined.get()) {
+logger.error("Detected that local node {} has been added 
to voter set at offset {}",
+   localReplicaKey, offset);
+hasJoined = Optional.of(true);
+}
+}
+});
+
 // Read the entire log
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
 
+if (nodeId.isPresent() && canBecomeVoter && quorumConfig.autoJoin()
+&& isVoter(ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId))) 
{
+hasJoined = Optional.of(true);
+}
+

Review Comment:
   It is not super obvious, but I explained here why this code is incorrect: 
https://github.com/apache/kafka/pull/20859#pullrequestreview-3506989795



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3372,8 +3393,8 @@ private boolean 
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
  * and are configured to auto join should attempt to automatically 
join the voter
  * set for the configured topic partition.
  */
-return !hasJoined && 
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter
-&& quorumConfig.autoJoin() && 
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
+return hasJoined.isPresent() && !hasJoined.get() && 
partitionState.lastKraftVersion().isReconfigSupported()

Review Comment:
   We need to add `!hasFetchTimeoutExpired() && local node LEO >= leader HWM` 
to this check.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -1810,18 +1836,16 @@ private boolean handleFetchResponse(
 OptionalLong.empty() : 
OptionalLong.of(partitionResponse.highWatermark());
 updateFollowerHighWatermark(state, highWatermark);
 
-if (initHighWatermark < 0 && partitionResponse.highWatermark() 
>= 0) {
-initHighWatermark = partitionResponse.highWatermark();
-
-state.highWatermark().ifPresent(hw -> {
-if (hw.offset() >= initHighWatermark && 
nodeId.isPresent()) {
-hasJoined = 
partitionState.lastVoterSet().isVoter(
-ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId));
-}
-}
-);
+if (nodeId.isPresent() && logEndOffset() >= 
partitionResponse.highWatermark()) {

Review Comment:
   We should only make these state changes when auto join is enabled.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -500,11 +501,31 @@ public void initialize(
 kafkaRaftMetrics,
 externalKRaftMetrics
 );
+
+// Set up listener to track voter set changes
+partitionState.setVoterSetChangeListener((offset, voterSet) -> {

Review Comment:
   We should only set this listener when autojoin is enabled and we are on a 
controller.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-27 Thread via GitHub


TaiJuWu commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3585479651

   Rewrite code from design by @kevin-wu24 , if this change is acceptable, I 
will add test tomorrow.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3584352866

   > The follower can only move to HAS_JOINED via processing a FETCH_RESPONSE, 
which is too late (i.e. the local node has already sent an "incorrect" 
auto-join).
   > IMO this is the harder case to handle, and handling this on the leader 
side via KAFKA-19933 is kind of "wrong" too.
   
   Yes, so I think as long as KAFKA-19933 is implemented, we can make sure the 
observer cannot join the voters successfully due to 
`leaderState.isReplicaCaughtUp()` is false. And with the addVoter request sent, 
the UpdateVoterSet timer will be reset and fetch request will be sent next.
   
   > We might need to add another clause in `shouldSendAddOrRemoveVoter: && 
local node LEO >= leader HWM`.
   
   I can understand and agree with this check, the network partition issue will 
be resolved. But alternatively, I think we can also fix the issue by only 
improving `KAFKA-19933` without this `shouldSendAddOrRemoveVoter: && local node 
LEO >= leader HWM` change. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2567200843


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the 
node id of this node,
+// mark it as already joined because it is already in the voter 
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853 
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}

Review Comment:
   Alternatively, you can set `hasJoined = Optional.of(true)` when the leader 
is elected and `hasJoined.isEmpty()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2567198996


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the 
node id of this node,
+// mark it as already joined because it is already in the voter 
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853 
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}

Review Comment:
   ```
   if (canBecomeVoter && quorumConfig.autoJoin() && 
partitionState.lastVoterSet().size() == 1
   && 
partitionState.lastVoterSet().isVoter(ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId)) {
   hasJoined = Optional.of(true);
   ```
   is the correct check. The code you have always evaluates to true because the 
left hand side of the || is always true for a controller in `intialize()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2567198996


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the 
node id of this node,
+// mark it as already joined because it is already in the voter 
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853 
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}

Review Comment:
   ```
   if (quorumConfig.autoJoin() && partitionState.lastVoterSet().size() == 1
   && 
partitionState.lastVoterSet().isVoter(ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId)) {
   hasJoined = Optional.of(true);
   ```
   is the correct check. The code you have always evaluates to true because the 
left hand side of the || is always true for a controller in `intialize()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2567177319


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the 
node id of this node,
+// mark it as already joined because it is already in the voter 
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853 
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}

Review Comment:
   Sorry, from my understanding, this code should like below but after this 
change it always true for controller.
   
   ```
   // Q: After this condition, hasJoin always true
   if (nodeId.isPresent() && (hasJoined.isEmpty()) || 
partitionState.lastVoterSet().size() == 1
   && 
partitionState.lastVoterSet().isVoter(ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId)))
   {
   // if voter set is a singleton set contain only local ReplicaKey
   // it is the standalone bootstrap node
   hasJoined = Optional.of(true);
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3584172984

   > we can resolve this issue because the state will move to HAS_JOINED when 
finding the voter contains itself. But we can handle the issue separately since 
it's not directly related to this issue. Opened 
[KAFKA-19933](https://issues.apache.org/jira/browse/KAFKA-19933).
   
   The follower can only move to HAS_JOINED via processing a FETCH_RESPONSE, 
which is too late (i.e. the local node has already sent an "incorrect" 
auto-join). In the first scenario, based on whether the `UpdateVoterSet` timer 
is expired or not, the follower will either send an "incorrect" 
AddVoterRequest, or send a FetchRequest (this will lead to the correct state 
assuming we get a response). 
   
   IMO this is the harder case to handle, and handling this on the leader side 
via `KAFKA-19933` is kind of "wrong" too. I think the observer needs to look at 
its fetch timer, and if that is expired, sending a FETCH takes priority over 
running the auto-join algorithm. That way, long-partitioned followers will 
fetch first, and avoid the above case. The proposed fix here is to add a `&& 
!hasFetchTimeoutExpired()` to `shouldSendAddOrRemoveVoter`.
   
   I think my previous assertion about the conditions of transitioning from 
`HAS_NOT_JOINED -> HAS_JOINED` are incorrect. If the local node C, which 
currently believes the voter set as (A,B) and is in `HAS_NOT_JOINED` state, is 
so far behind it requires two fetches, and the first fetch does not contain any 
new VotersRecords, but the second fetch would contain the records (A,B,C) and 
(A,B), local node C would still try to auto-join "incorrectly"...
   
   We might need to add another clause in `shouldSendAddOrRemoveVoter`: `&& 
local node LEO >= leader HWM`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3584117480

   > Voter set is (A,B) with leader A. Node C is a caught up observer and tries 
to add itself. The new voter set (A,B,C) is committed by A and B, but C is 
partitioned and never learns its "auto-join" succeeded. The user removes C, and 
the voter set is back to (A,B). When the partition goes away, if updateVoterSet 
timer is expired, node C's shouldSendAddOrRemoveVoter evaluates to true and it 
will try to auto-join again.
   > I think in AddVoterHandler we check that the node being added is "caught 
up," but that check currently returns true if the node has fetched within the 
last hour (we can make this more strict obviously).
   
   Nice find! And I agree we can make the "caught up" criteria more strict to 
make sure the new added voter already knows the voters change, and then adopt 
the voter sets track in partitionState.updateState() solution below, we can 
resolve this issue because the state will move to `HAS_JOINED` when finding the 
voter contains itself. But we can handle the issue separately since it's not 
directly related to this issue. Opened 
[KAFKA-19933](https://issues.apache.org/jira/browse/KAFKA-19933).
   
   > Voter set is (A,B) with leader A. Node C is a caught up observer and tries 
to add itself. The new voter set (A,B,C) is committed by A and B, but C is 
partitioned and never learns its "auto-join" succeeded. The user removes C, and 
the voter set is back to (A,B). This is similar to the case above, but when the 
partition goes away, suppose node C's shouldSendAddOrRemoveVoter evaluates to 
false, and node C fetches instead of trying to auto-join. If this fetch 
contains both voter set changes in the same response (the latest voter set is 
(A,B)), the current code will still believe node C is in the HAS_NOT_JOINED 
state, and will attempt to auto-join again because we are only checking 
partitionState.lastVoterSet().
   > I still think we can handle this properly in the code, and it means we 
need transition HAS_NOT_JOINED -> HAS_JOINED if any VotersRecord from the 
FETCH_RESPONSE contains the local ReplicaKey. We go through the new voter sets 
in partitionState.updateState(), and remember we do not need the HWM here 
because to get to the HAS_NOT_JOINED state we had to be "caught up" at that 
point in time.
   
   Another good find! (How could you come up these edge cases :) ) Yes, I think 
tracking the voters set change in `partitionState.updateState()` should work. 
   
   
   @TaiJuWu , So in short, we have to do:
   1. When a node is the leader and the state is `UNKNOWN`, we make it to 
`HAS_JOINED`
   2. When a node is the follower (both voter and observer) and state is 
`UNKNOWN`, wait until the HWM updated, then check if the state is `HAS_JOINED` 
or `HAS_NOT_JOINED`.
   3. Implement the tracking the voters set change in 
`partitionState.updateState()` solution
   4. If in (3), we found the state is `HAS_NOT_JOINED` and local ReplicaKey 
appear in the voters set, we should change the state to `HAS_JOINED`.
   
   What do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3584110211

   @showuon regarding:
   
   > This will not work because if the node lag a lot. The 1st fetch might not 
be able to catch up with the leader.
   
   Where in the code can this situation happen? I was under the impression that 
when a lagging follower node sends a fetch request to the leader, the leader 
does one of two things if there is no diverging log:
   
   1. If the leader's log start offset is greater than the followers fetch 
offset, the leader sends a response telling the follower to fetch a snapshot.
   2. Otherwise, send all the records until the end of the log in the fetch 
response: 
https://github.com/apache/kafka/blob/trunk/raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java#L1625.
   
   This means when a node receives a fetch response, they get the uncommitted 
state (if it exists) on the leader. Am I missing something?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3583469356

   Hmmm, the more I think about this change the more complicated I find this 
new proposed semantic to be.
   
   Here is another case I thought about:
   
   Voter set is (A,B) with leader A. Node C is a caught up observer and tries 
to add itself. The new voter set (A,B,C) is committed by A and B, but C is 
partitioned and never learns its "auto-join" succeeded. The user removes C, and 
the voter set is back to (A,B). When the partition goes away, if 
`updateVoterSet` timer is expired, node C's `shouldSendAddOrRemoveVoter` 
evaluates to `true` and it will try to auto-join again.
   I think in `AddVoterHandler` we check that the node being added is "caught 
up," but that check currently returns true if the node has fetched within the 
last hour (we can make this more strict obviously). 
   
   Another thing we need to consider is that multiple voter set changes may 
occur in the same `FetchRequestResponse`. For example, we could have the 
following:
   
   Voter set is (A,B) with leader A. Node C is a caught up observer and tries 
to add itself. The new voter set (A,B,C) is committed by A and B, but C is 
partitioned and never learns its "auto-join" succeeded. The user removes C, and 
the voter set is back to (A,B). This is similar to the case above, but when the 
partition goes away, suppose node C's `shouldSendAddOrRemoveVoter` evaluates to 
`false`, and node C fetches instead of trying to auto-join. If this fetch 
contains both voter set changes in the same response (the latest voter set is 
(A,B)), the current code will still believe node C is in the `HAS_NOT_JOINED` 
state, and will attempt to auto-join again because we are only checking 
`partitionState.lastVoterSet()`. 
   I still think we can handle this properly in the code, and it means we need 
transition `HAS_NOT_JOINED -> HAS_JOINED` if any `VotersRecord` from the 
`FETCH_RESPONSE` contains the local `ReplicaKey`. We go through the new voter 
sets in `partitionState.updateState()`, and remember we do not need the HWM 
here because to get to the `HAS_NOT_JOINED` state we had to be "caught up" at 
that point in time.
   
   I want to highlight the above two cases to say that despite the desired 
semantic seeming very simple (auto-join at most once per process lifetime), 
implementing this properly is not straightforward at all, since there are a ton 
of edge cases because our local node's voter set can be out-of-date by an 
arbitrary number of VotersRecords. Additionally, at least for me, trying to 
verify that this implementation does indeed match our desired semantic is just 
as difficult (I do think I am out of edge cases now... no promises though 😆 ).
   
   Let me know your thoughts @TaiJuWu @showuon @jsancio.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565548226


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the 
node id of this node,
+// mark it as already joined because it is already in the voter 
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853 
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}

Review Comment:
   H. Actually, we need to set `hasJoined = Optional.of(true)` IFF 
`hasJoined.isEmpty()` for the first leader elected, even if it is not a 
standalone leader. Otherwise, if we remove the leader, it can auto-join back 
because the state can go from UNKNOWN -> HAS_NOT_JOINED if the user manually 
removes the leader.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean hasJoined = false;

Review Comment:
   I have an idea how we can model this state properly. We can use an 
Optional. Now we can represent 3 distinct "hasJoined" states:
   
   1. `UNKNOWN`: We'll use Optional.empty(), which applies to all nodes on 
startup, including brokers. Brokers, static controller quorums, or dynamic 
controllers without auto-join enabled will always remain in this state.
   2. `HAS_NOT_JOINED`: We'll use Optional.of(false), which only controllers 
with auto-join can transition to.
   3. `HAS_JOINED`: We'll use Optional.of(true), which only controllers with 
auto-join can transition to.
   
   The possible state transitions for controllers who have auto-join enabled 
are:
   
   1. `UNKNOWN -> HAS_JOINED`: This occurs when the controller handles a fetch 
response, is "up-to-date" with the HWM, and discovers it IS in the voter set. 
This transition can also occur when the controller is the only member of the 
voter set when starting up (standalone controller), or if the controller wins 
the initial election.
   2. `UNKNOWN -> HAS_NOT_JOINED`: This occurs when the controller handles a 
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the 
voter set. 
   3. `HAS_NOT_JOINED -> HAS_JOINED`: This also occurs when the controller 
handles a fetch response, and discovers it is in the voter set. However, we do 
not need to do the HWM check here because the transition from `UNKNOWN -> 
HAS_NOT_JOINED` has already made the check (i.e. we know we are up to date).
   
   Importantly, state transition 2 can only occur once in the lifetime of the 
controller, because we cannot transition to UNKNOWN from any state during the 
lifetime of the process. Additionally, we cannot transition from `HAS_JOINED -> 
HAS_NOT_JOINED` ever.
   
   `shouldSendAddOrRemoveVoter` will be conditioned on the state being 
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its 
process lifetime, a controller will not try to auto-join again after being 
removed from the voter set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean hasJoined = false;

Review Comment:
   I have an idea how we can model this state properly. We can use an 
Optional. Now we can represent 3 distinct "hasJoined" states:
   
   1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on 
startup, including brokers. Brokers, static controller quorums, or dynamic 
controllers without auto-join enabled will always remain in this state.
   2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers with 
auto-join can transition to.
   3. HAS_JOINED: We'll use Optional.of(true), which only controllers with 
auto-join can transition to.
   
   The possible state transitions for controllers who have auto-join enabled 
are:
   
   1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch 
response, is "up-to-date" with the HWM, and discovers it IS in the voter set. 
This transition can also occur when the controller is the only member of the 
voter set when starting up (standalone controller), or if the controller wins 
the initial election.
   2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a 
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the 
voter set. 
   3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller 
handles a fetch response, and discovers it is in the voter set. However, we do 
not need to do the HWM check here because the transition from UNKNOWN -> 
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
   
   Importantly, state transition 2 can only occur once in the lifetime of the 
controller, because we cannot transition to UNKNOWN during the lifetime of the 
process. Additionally, the controller process cannot go from HAS_JOINED -> 
HAS_NOT_JOINED ever.
   
   `shouldSendAddOrRemoveVoter` will be conditioned on the state being 
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its 
process lifetime, a controller will not try to auto-join again after being 
removed from the voter set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean hasJoined = false;

Review Comment:
   I have an idea how we can model this state properly. We can use an 
Optional. Now we can represent 3 distinct "hasJoined" states:
   
   1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on 
startup, including brokers
   2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers with 
auto-join can transition to.
   3. HAS_JOINED: We'll use Optional.of(true), which only controllers with 
auto-join can transition to.
   
   The possible state transitions for controllers who have auto-join enabled 
are:
   
   1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch 
response, is "up-to-date" with the HWM, and discovers it IS in the voter set. 
This transition can also occur when the controller is the only member of the 
voter set when starting up (standalone controller), or if the controller wins 
the initial election.
   2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a 
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the 
voter set. 
   3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller 
handles a fetch response, and discovers it is in the voter set. However, we do 
not need to do the HWM check here because the transition from UNKNOWN -> 
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
   
   Importantly, state transition 2 can only occur once in the lifetime of the 
controller, because we cannot transition to UNKNOWN during the lifetime of the 
process. Additionally, the controller process cannot go from HAS_JOINED -> 
HAS_NOT_JOINED ever.
   
   `shouldSendAddOrRemoveVoter` will be conditioned on the state being 
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its 
process lifetime, a controller will not try to auto-join again after being 
removed from the voter set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean hasJoined = false;

Review Comment:
   I have an idea how we can model this state properly. We can use an 
Optional. Now we can represent 3 distinct "hasJoined" states:
   
   1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on 
startup, including brokers
   2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers with 
auto-join can transition to.
   3. HAS_JOINED: We'll use Optional.of(true), which only controllers with 
auto-join can transition to.
   
   The possible state transitions for controllers who have auto-join enabled 
are:
   
   1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch 
response, is "up-to-date" with the HWM, and discovers it IS in the voter set, 
or if the controller is the only member of the voter set when starting up.
   2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a 
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the 
voter set. 
   3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller 
handles a fetch response, and discovers it is in the voter set. However, we do 
not need to do the HWM check here because the transition from UNKNOWN -> 
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
   
   Importantly, state transition 2 can only occur once in the lifetime of the 
controller, because we cannot transition to UNKNOWN during the lifetime of the 
process. Additionally, the controller process cannot go from HAS_JOINED -> 
HAS_NOT_JOINED ever.
   
   `shouldSendAddOrRemoveVoter` will be conditioned on the state being 
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its 
process lifetime, a controller will not try to auto-join again after being 
removed from the voter set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565548226


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the 
node id of this node,
+// mark it as already joined because it is already in the voter 
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853 
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}

Review Comment:
   H. Actually, we need to set `hasJoined = Optional.of(true)` IFF 
`hasJoined.isEmpty()` for the first leader elected, even if it is not a 
standalone leader. Otherwise, if we remove the leader, it can auto-join back 
because the state can go from UNKNOWN -> HAS_NOT_JOINED.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565548226


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the 
node id of this node,
+// mark it as already joined because it is already in the voter 
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853 
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}

Review Comment:
   H. Actually, we need to set `hasJoined = Optional.of(true)` for the 
first leader elected, even if it is not a standalone leader. Otherwise, if we 
remove the leader, it can auto-join back because the state can go from UNKNOWN 
-> HAS_NOT_JOINED.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean hasJoined = false;

Review Comment:
   I have an idea how we can model this state properly. We can use an 
Optional. Now we can represent 3 distinct "hasJoined" states:
   
   1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on 
startup, including brokers
   2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers can 
transition to.
   3. HAS_JOINED: We'll use Optional.of(true), which only controllers can 
transition to.
   
   The possible state transitions for controllers who have auto-join enabled 
are:
   
   1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch 
response, is "up-to-date" with the HWM, and discovers it IS in the voter set, 
or if the controller is the only member of the voter set when starting up.
   2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a 
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the 
voter set. 
   3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller 
handles a fetch response, and discovers it is in the voter set. However, we do 
not need to do the HWM check here because the transition from UNKNOWN -> 
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
   
   Importantly, state transition 2 can only occur once in the lifetime of the 
controller, because we cannot transition to UNKNOWN during the lifetime of the 
process. Additionally, the controller process cannot go from HAS_JOINED -> 
HAS_NOT_JOINED ever.
   
   `shouldSendAddOrRemoveVoter` will be conditioned on the state being 
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its 
process lifetime, a controller will not try to auto-join again after being 
removed from the voter set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean hasJoined = false;

Review Comment:
   I have an idea how we can model this state properly. We can use an 
Optional. Now we can represent 3 distinct "hasJoined" states:
   
   1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on 
startup, including brokers
   2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers can 
transition to.
   3. HAS_JOINED: We'll use Optional.of(true), which only controllers can 
transition to.
   
   The possible state transitions for controllers are:
   
   1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch 
response, is "up-to-date" with the HWM, and discovers it IS in the voter set, 
or if the controller is the only member of the voter set when starting up.
   2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a 
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the 
voter set. 
   3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller 
handles a fetch response, and discovers it is in the voter set. However, we do 
not need to do the HWM check here because the transition from UNKNOWN -> 
HAS_NOT_JOINED has already done it (i.e. we know we are up to date).
   
   Importantly, state transition 2 can only occur once in the lifetime of the 
controller, because we cannot transition to UNKNOWN during the lifetime of the 
process. Additionally, the controller process cannot go from HAS_JOINED -> 
HAS_NOT_JOINED ever.
   
   `shouldSendAddOrRemoveVoter` will be conditioned on the state being 
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its 
process lifetime, a controller will not try to auto-join again after being 
removed from the voter set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565460133


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -1805,6 +1809,19 @@ private boolean handleFetchResponse(
 OptionalLong highWatermark = partitionResponse.highWatermark() 
< 0 ?
 OptionalLong.empty() : 
OptionalLong.of(partitionResponse.highWatermark());
 updateFollowerHighWatermark(state, highWatermark);
+
+if (initHighWatermark < 0 && partitionResponse.highWatermark() 
>= 0) {
+initHighWatermark = partitionResponse.highWatermark();
+
+state.highWatermark().ifPresent(hw -> {
+if (hw.offset() >= initHighWatermark && 
nodeId.isPresent()) {
+hasJoined = 
partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId));
+}
+}
+);

Review Comment:
   hmmm, this if check seems incorrect. Doesn't this mean we won't ever enter 
this method again if the leader sends us a partitionResponse.highWatermark() >= 
0?
   
   We should be checking the `highWatermark` against our local LEO, and if our 
LEO >= HWM, the local node considers itself "caught up" and can set the 
`hasJoined` state to the proper value. I believe this is correct because we can 
only have at most one uncommitted voters record at a time. 
   
   Importantly, we need to enforce the state transition requirements in my 
comment above here.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -1805,6 +1809,19 @@ private boolean handleFetchResponse(
 OptionalLong highWatermark = partitionResponse.highWatermark() 
< 0 ?
 OptionalLong.empty() : 
OptionalLong.of(partitionResponse.highWatermark());
 updateFollowerHighWatermark(state, highWatermark);
+
+if (initHighWatermark < 0 && partitionResponse.highWatermark() 
>= 0) {
+initHighWatermark = partitionResponse.highWatermark();
+
+state.highWatermark().ifPresent(hw -> {
+if (hw.offset() >= initHighWatermark && 
nodeId.isPresent()) {
+hasJoined = 
partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId));
+}
+}
+);

Review Comment:
   hmmm, this if check seems incorrect. Doesn't this mean we won't ever enter 
this method again if the leader sends us a `partitionResponse.highWatermark() 
>= 0`?
   
   We should be checking the `highWatermark` against our local LEO, and if our 
LEO >= HWM, the local node considers itself "caught up" and can set the 
`hasJoined` state to the proper value. I believe this is correct because we can 
only have at most one uncommitted voters record at a time. 
   
   Importantly, we need to enforce the state transition requirements in my 
comment above here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565350148


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean hasJoined = false;

Review Comment:
   I have an idea how we can model this state properly. We can use an 
Optional. Now we can represent 3 distinct "hasJoined" states:
   
   1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on 
startup, including brokers
   2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers can 
transition to.
   3. HAS_JOINED: We'll use Optional.of(true), which only controllers can 
transition to.
   
   The possible state transitions for controllers are:
   
   1. UNKNOWN -> HAS_JOINED: This occurs when the controller handles a fetch 
response, is "up-to-date" with the HWM, and discovers it IS in the voter set, 
or if the controller is the only member of the voter set when starting up.
   2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller handles a 
fetch response, is "up-to-date" with the HWM, and discovers it is NOT in the 
voter set. 
   3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller 
handles a fetch response is "up-to-date" with the HWM, and discovers it is in 
the voter set.
   
   Importantly, state transition 2 can only occur once in the lifetime of the 
controller, because we cannot transition to UNKNOWN during the lifetime of the 
process. Additionally, the controller process cannot go from HAS_JOINED -> 
HAS_NOT_JOINED ever.
   
   `shouldSendAddOrRemoveVoter` will be conditioned on the state being 
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its 
process lifetime, a controller will not try to auto-join again after being 
removed from the voter set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2565395726


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3285,6 +3301,9 @@ private void 
prospectiveTransitionAfterElectionLoss(ProspectiveState prospective
 private long pollFollower(long currentTimeMs) {
 FollowerState state = quorum.followerStateOrThrow();
 if (quorum.isVoter()) {
+if (quorumConfig.autoJoin()) {
+hasJoined = true;
+}

Review Comment:
   We can remove this code.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,14 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the the VotersRecord with the highest offset contain the 
node id of this node,
+// mark it as already joined because it is already in the voter 
set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853 
properly
+hasJoined = partitionState.lastVoterSet().isVoter(
+ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
+);
+}

Review Comment:
   Here we should only set `hasJoined = Optional.of(true)` IFF the 
partitionState.lastVoterSet() is a singleton set containing only the local 
`ReplicaKey`.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3352,8 +3371,8 @@ private boolean 
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
  * and are configured to auto join should attempt to automatically 
join the voter
  * set for the configured topic partition.
  */
-return partitionState.lastKraftVersion().isReconfigSupported() && 
canBecomeVoter &&
-quorumConfig.autoJoin() && 
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
+return !hasJoined && 
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter

Review Comment:
   ```suggestion
   return hasJoined.isPresent() && !hasJoined.get() && 
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter
   ```



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean hasJoined = false;

Review Comment:
   I have an idea how we can model this state properly. We can use an 
Optional. Now we can represent 3 distinct "hasJoined" states:
   
   1. UNKNOWN: We'll use Optional.empty(), which applies to all nodes on 
startup, including brokers
   2. HAS_NOT_JOINED: We'll use Optional.of(false), which only controllers can 
transition to.
   3. HAS_JOINED: We'll use Optional.of(true), which only controllers can 
transition to.
   
   The possible state transitions for controllers are:
   
   1. UNKNOWN -> HAS_JOINED: This occurs when the controller fetches, is 
"up-to-date" with the HWM, and discovers it IS in the voter set, or if the 
controller is the only member of the voter set when starting up.
   2. UNKNOWN -> HAS_NOT_JOINED: This occurs when the controller fetches, is 
"up-to-date" with the HWM, and discovers it is NOT in the voter set. 
   3. HAS_NOT_JOINED -> HAS_JOINED: This also occurs when the controller 
fetches, is "up-to-date" with the HWM, and discovers it is in the voter set.
   
   Importantly, state transition 2 can only occur once in the lifetime of the 
controller, because we cannot transition to UNKNOWN during the lifetime of the 
process. Additionally, the controller process cannot go from HAS_JOINED -> 
HAS_NOT_JOINED ever.
   
   `shouldSendAddOrRemoveVoter` will be conditioned on the state being 
`HAS_NOT_JOINED`. Therefore, the two invariants above mean that, during its 
process lifetime, a controller will not try to auto-join again after being 
removed from the voter set.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -1805,6 +1809,19 @@ private boolean handleFetchResponse(
 OptionalLong highWatermark = partitionResponse.highWatermark() 
< 0 ?
 OptionalLong.empty() : 
OptionalLong.of(partitionResponse.highWatermark());
 updateFollowerHighWatermark(state, highWatermark);
+
+if (initHighWatermark < 0 && partitionResponse.highWatermark() 
>= 0) {
+initHighWatermark = partitionResponse.highWatermark();
+
+state.highWatermark().ifPresent(hw -> {
+if (hw.offset() >= initHighWatermark && 
nodeId.isPresent()) {
+

Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


TaiJuWu commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3581768804

   > > Consider the following case, which IMO results in equally confusing UX:
   > > We have a voter set of (A,B,C) with leader A, KIP-853, and auto-join 
enabled. We remove node C via RemoveRaftVoter, and the new (A,B) VotersRecord 
is committed (i.e. A and B both persist it). However, C could be partitioned 
and not replicate the (A,B) VotersRecord, meaning its 
partitionState.lastVoterSet() == (A,B,C). Node C restarts and enters 
initialize(), and it will set hasAutoJoined = true. In this case, the user 
"expects" C to auto-join because when they look at the quorum status, the voter 
set is (A,B) and C just restarted with auto-join enabled, but C will not 
attempt to auto-join. The point here is that A, B, and the user believe the 
voter set is (A,B), whereas C believes the voter set is (A,B,C) and changes its 
state based on it.
   > 
   > This is a really good point @kevin-wu24 ! Yes, the state in the 
`KafkaRaftClient#initialize` is just the `the VotersRecord with the highest 
offset in local node`. So in the case of network partition above, the state 
will not be update-to-date. Actually, not only the network partition case, 
suppose the node C is down, and during the period, C is removed from the voters 
set. After node C startup, the initial voters set is also [A,B,C]. So I think 
we should fix this issue.
   > 
   > > I'm not sure how to fix this case at the moment, and need to think about 
how to set hasAutoJoined properly. Maybe this value starts as true and the 
local node can set it to false after successfully fetching and finding out it 
is not a voter?
   > 
   > This will not work because if the node lag a lot. The 1st fetch might not 
be able to catch up with the leader.
   > 
   > I think what we need to do is to make sure the node is caught up with the 
leader controller. By making sure it is caught up, the network partition case 
above can be resolved because once we catch up with the leader, we know the 
`latest voter state on startup`. And we can identify the node caught up or not 
by checking the highwatermark in fetch response from leader. The flow is like 
this (using the situation above):
   > 
   > 1. node C starts up with network partition (voters = [ABC])
   > 2. network partition resolved
   > 3. node C sends the 1st fetch request to the leader
   > 4. leader responds with highwatermark=100
   > 5. After appending with the fetch response data, the LEO in the node C is 
80 (not caught up yet)
   > 6. Node C sends 2nd fetch request
   > 7. The LEO moves to 102 (suppose the leader has new data appended), which 
is >= 100
   > 8. Node C updates the `hasJoin` value with the latest voters set at this 
moment (i.e. voters = [A, B]), and `hasJoined=false`)
   > 9. Node C auto joined the voters set due to `hasJoined=false`
   > 
   > @TaiJuWu @kevin-wu24 , does this make sense to you?
   
   It make sense to me, address it by 
https://github.com/apache/kafka/pull/20859/commits/61984cf3d8119f0953d04abfd9b582d21734d015


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


ppatierno commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3580795668

   @kevin-wu24 thank you very much for looking into this! So to answer your 
questions ...
   
   > Question for @ppatierno regarding their use case: Just to confirm my 
understanding on the issue: when you remove the controller role from these 
nodes, is the operator "watching" the config file, and then sends the 
RemoveRaftVoter based on that? And then you end up in this state where the 
combined "controllers" are continuously being removed + added back because of 
auto-join?
   
   I am not sure what you mean by "is the operator "watching" the config file" 
but let me explain how it works in practice.
   What the operator detects (because the user has update the corresponding 
`KafkaNodePool` custom resource, which hosts info for the broker/controller 
nodes) is that it has to remove the controllers from the combined-nodes. So it 
knows what should be the new configuration for these nodes (i.e. removing 
"controller" from the `process.roles` property) and then it rolls them to allow 
restarting as broker only with the new configuration. Before rolling, it's 
going to remove them from voters list by using the RemoveRaftVoter Admin API. 
You can see here that with auto-join enabled, these controllers re-join 
immediately right after the RemoveRaftVoter Admin API and before rolling. So 
they would restart as broker-only but seeing them still in the voters list and 
try to send the UpdateVoteRequest (see the 
https://issues.apache.org/jira/browse/KAFKA-19867).
   
   > I think what you want is actually to first disable auto-join on the 
combined controller, restart it, and then remove it from the voter set. Then 
you are free to make it broker-only.
   
   This would make the controllers scaling more complex and longer, even 
because the operator should:
   
   * disable auto-join, restart the nodes still as "controller" (as combined 
nodes)
   * remove them from the voters set
   * re-configuring them by removing the "controller" (making broker only), 
restarting them.
   
   Two rollings while just one would be sufficient, and for a big cluster it's 
really a lot imho other than the complexity being added to the operator which 
would be not necessary thanks to this fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-26 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3580486460

   > Consider the following case, which IMO results in equally confusing UX:
   > We have a voter set of (A,B,C) with leader A, KIP-853, and auto-join 
enabled. We remove node C via RemoveRaftVoter, and the new (A,B) VotersRecord 
is committed (i.e. A and B both persist it). However, C could be partitioned 
and not replicate the (A,B) VotersRecord, meaning its 
partitionState.lastVoterSet() == (A,B,C). Node C restarts and enters 
initialize(), and it will set hasAutoJoined = true. In this case, the user 
"expects" C to auto-join because when they look at the quorum status, the voter 
set is (A,B) and C just restarted with auto-join enabled, but C will not 
attempt to auto-join. The point here is that A, B, and the user believe the 
voter set is (A,B), whereas C believes the voter set is (A,B,C) and changes its 
state based on it.
   
   This is a really good point @kevin-wu24 ! Yes, the state in the 
`KafkaRaftClient#initialize` is just the `the VotersRecord with the highest 
offset in local node`. So in the case of network partition above, the state 
will not be update-to-date. Actually, not only the network partition case, 
suppose the node C is down, and during the period, C is removed from the voters 
set. After node C startup, the initial voters set is also [A,B,C]. So I think 
we should fix this issue.
   
   > I'm not sure how to fix this case at the moment, and need to think about 
how to set hasAutoJoined properly. Maybe this value starts as true and the 
local node can set it to false after successfully fetching and finding out it 
is not a voter?
   
   This will not work because if the node lag a lot. The 1st fetch might not be 
able to catch up with the leader.
   
   I think what we need to do is to make sure the node is caught up with the 
leader controller. By making sure it is caught up, the network partition case 
above can be resolved because once we catch up with the leader, we know the 
`latest voter state  on startup`. And we can identify the node caught up or not 
by checking the highwatermark in fetch response from leader. The flow is like 
this (using the situation above):
   
   1. node C starts up with network partition (voters = [ABC])
   5. network partition resolved
   6. node C sends the 1st fetch request to the leader
   7. leader responds with highwatermark=100
   8. After appending with the fetch response data, the LEO in the node C is 80 
(not caught up yet)
   9. Node C sends 2nd fetch request
   10. The LEO moves to 102 (suppose the leader has new data appended), which 
is >= 100
   11. Node C updates the `hasJoin` value with the latest voters set at this 
moment (i.e. voters = [A, B]), and `hasJoined=false`)
   12. Node C auto joined the voters set due to `hasJoined=false`
   
   @TaiJuWu @kevin-wu24 , does this make sense to you?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562686075


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark 
it as already joined

Review Comment:
   fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562640152


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2335,8 +2344,11 @@ private boolean handleAddVoterResponse(
 /* These error codes indicate the replica was successfully added or 
the leader is unable to
  * process the request. In either case, reset the update voter set 
timer to back off.
  */
-if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
-error == Errors.DUPLICATE_VOTER) {
+if (error == Errors.NONE) {
+
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasAutoJoined = true;

Review Comment:
   Thanks your detailed explanation, I missed this part indeed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562519856


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark 
it as already joined

Review Comment:
   If we have a static voters set, controllers cannot auto-join because they 
are on `kraft.verison=0`, not `kraft.version=1`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562507354


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark 
it as already joined

Review Comment:
   If we use static voters for initialization, I believe this should not be 
treated as a VotersRecord but rather as an internal variable within 
partitionState. From that perspective, I think the comment is acceptable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2562507354


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark 
it as already joined

Review Comment:
   If we use static voter for initialization , I think it is not voter record 
just a variable in partitionState and I think this comment is ok.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2561593244


##
raft/src/main/java/org/apache/kafka/raft/QuorumConfig.java:
##
@@ -105,7 +105,7 @@ public class QuorumConfig {
 
 public static final String QUORUM_AUTO_JOIN_ENABLE_CONFIG = QUORUM_PREFIX 
+ "auto.join.enable";
 public static final String QUORUM_AUTO_JOIN_ENABLE_DOC = "Controls whether 
a KRaft controller should automatically " +
-"join the cluster metadata partition for its cluster id.";
+"join the cluster metadata partition for its cluster id when the node 
startup.";

Review Comment:
   "Controls whether a KRaft controller ( tuple) should 
automatically join the cluster metadata partition as a voter during node 
startup if the controller is not already a voter"
   
   There is probably a better way to word the above.
   
   This most accurate describes what we are changing the semantic too. I think 
I am okay with this definition of the semantic from a UX POV. I guess we do not 
really care if the `ReplicaKey` was added by auto-join or not, just that it was 
added as a voter.
   
   Our proposed semantic is essentially to make the auto-join code "at most 
once" for the lifetime of a given controller process execution, conditioned on 
the local ReplicaKey not being part of our voter set. However, this semantic 
can run into UX issues too.



##
raft/src/test/java/org/apache/kafka/raft/KafkaRaftClientAutoJoinTest.java:
##
@@ -317,6 +321,131 @@ private void pollAndDeliverFetchToUpdateVoterSet(
 context.client.poll();
 }
 
+@Test
+public void testBootstrapVoterSetDoesNotSendAddVoterAfterRemove() throws 
Exception {
+// Test that a bootstrap voter node, after being removed by 
RemoveVoter RPC,
+// will NOT automatically rejoin (canAutoJoin remains false)
+final var leader = replicaKey(randomReplicaId(), true);
+final var follower = replicaKey(leader.id() + 1, true);
+final var thisVoter = replicaKey(follower.id() + 1, true);
+
+final int epoch = 1;
+final var context = new RaftClientTestContext.Builder(
+thisVoter.id(),
+thisVoter.directoryId().get())  // Same directory ID -> node is 
voter
+.withRaftProtocol(KIP_853_PROTOCOL)
+.withStartingVoters(
+VoterSetTest.voterSet(Stream.of(leader, follower, thisVoter)), 
KRaftVersion.KRAFT_VERSION_1
+)
+.withElectedLeader(epoch, leader.id())
+.withAutoJoin(true)
+.withCanBecomeVoter(true)
+.build();
+
+// Node should be a follower (voter) initially
+assertTrue(context.client.quorum().isFollower());
+
+// Complete initial fetch
+context.advanceTimeAndCompleteFetch(epoch, leader.id(), true);
+
+// Simulate this node being removed by RemoveVoter RPC (initiated by 
another node)
+// Update voter set via fetch - this node is now removed from voter set
+pollAndDeliverFetchToUpdateVoterSet(
+context,
+epoch,
+VoterSetTest.voterSet(Stream.of(leader, follower))  // This node 
is no longer in voter set
+);
+
+// Node should now be an observer after being removed
+assertTrue(context.client.quorum().isObserver());
+
+// Advance time to expire update voter set timer
+context.time.sleep(context.fetchTimeoutMs);
+
+// Poll and verify that NO AddVoter request is sent
+// With canAutoJoin=false (set during init because ID was in voter 
set),
+// node should NOT automatically rejoin
+context.time.sleep(1);
+context.pollUntilRequest();
+
+// Should not include AddVoter
+assertEquals(0, 
context.channel.drainSentRequests(Optional.of(ApiKeys.ADD_RAFT_VOTER)).size());
+
+}
+
+@Test
+public void testSuccessfulAddVoterSetsCanAutoJoinAgain() throws Exception {
+// Test scenario:
+// 1. Node initializes as observer
+// 2. Node joins as follower via AddVoter
+// 3. Node is removed via RemoveVoter and becomes observer again
+// 4. Node should NOT send AddVoter request again
+final var leader = replicaKey(randomReplicaId(), true);
+final var follower = replicaKey(leader.id() + 1, true);
+final var newVoter = replicaKey(follower.id() + 1, true);
+final int epoch = 1;
+final var context = new RaftClientTestContext.Builder(
+newVoter.id(),
+newVoter.directoryId().get())
+.withRaftProtocol(KIP_853_PROTOCOL)
+.withStartingVoters(
+VoterSetTest.voterSet(Stream.of(leader, follower)), 
KRaftVersion.KRAFT_VERSION_1
+)
+.withElectedLeader(epoch, leader.id())
+.withAutoJoin(true)
+.withCanBecomeVoter(true)
+.build();
+
+// 1. 

Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2561465070


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2335,8 +2344,11 @@ private boolean handleAddVoterResponse(
 /* These error codes indicate the replica was successfully added or 
the leader is unable to
  * process the request. In either case, reset the update voter set 
timer to back off.
  */
-if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
-error == Errors.DUPLICATE_VOTER) {
+if (error == Errors.NONE) {
+
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasAutoJoined = true;

Review Comment:
   Take a look at this code from `AddVoterHandler#handleApiVersionsResponse`:
   ```
   current.setLastOffset(leaderState.appendVotersRecord(newVoters, 
currentTimeMs));
   if (!current.ackWhenCommitted()) { ***we execute the below when 
auto-join is enabled***
   // complete the future to send response, but do not reset the 
state,
   // since the new voter set is not yet committed
   current.future().complete(RaftUtil.addVoterResponse(Errors.NONE, 
null));
   }
   ```
   When auto-join is enabled, we send the RPC response BEFORE the new voters 
record is committed. The motivation behind this is detailed in: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1186%3A+Update+AddRaftVoterRequest+RPC+to+support+auto-join,
 and is necessary for a fully available auto-join feature.
   
   What this means here is that our local node may not actually have 
auto-joined, but we set `hasAutoJoined = true`. For example, what if the remote 
leader fails over before the VotersRecord adding the local node is committed? 
The local node has not added itself via auto-join, and will not until it 
restarts, despite the user expecting it too.
   
   The "more correct" place to set `hasAutoJoined = true` is in `pollFollower` 
if we are in the `quorum.isVoter()` case AND `quorumConfig.autoJoin() == true`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2561429495


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark 
it as already joined
+// because it is already in the voter set.
+// Check using ReplicaKey (id + directoryId) to handle KIP-853 
properly
+ReplicaKey localReplicaKey = ReplicaKey.of(nodeId.getAsInt(), 
nodeDirectoryId);
+hasAutoJoined = 
partitionState.lastVoterSet().isVoter(localReplicaKey);

Review Comment:
   ```suggestion
   hasAutoJoined = partitionState.lastVoterSet().isVoter(
   ReplicaKey.of(nodeId.getAsInt(), nodeDirectoryId)
   );
   ```



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -2335,8 +2344,11 @@ private boolean handleAddVoterResponse(
 /* These error codes indicate the replica was successfully added or 
the leader is unable to
  * process the request. In either case, reset the update voter set 
timer to back off.
  */
-if (error == Errors.NONE || error == Errors.REQUEST_TIMED_OUT ||
-error == Errors.DUPLICATE_VOTER) {
+if (error == Errors.NONE) {
+
quorum.followerStateOrThrow().resetUpdateVoterSetPeriod(currentTimeMs);
+hasAutoJoined = true;

Review Comment:
   Take a look at this code from `AddVoterHandler#handleApiVersionsResponse`:
   ```
   current.setLastOffset(leaderState.appendVotersRecord(newVoters, 
currentTimeMs));
   if (!current.ackWhenCommitted()) { ***we execute the below when 
auto-join is enabled***
   // complete the future to send response, but do not reset the 
state,
   // since the new voter set is not yet committed
   current.future().complete(RaftUtil.addVoterResponse(Errors.NONE, 
null));
   }
   ```
   When auto-join is enabled, we send the RPC response BEFORE the new voters 
record is committed. The motivation behind this is detailed in: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1186%3A+Update+AddRaftVoterRequest+RPC+to+support+auto-join,
 and is necessary for a fully available auto-join feature.
   
   What this means here is that our local node may not actually have 
auto-joined, but we set `hasAutoJoined = true`. For example, what if the remote 
leader fails over before the VotersRecord adding the local node is committed? 
   
   The "more correct" place to set `hasAutoJoined = true` is in `pollFollower` 
if we are in the `quorum.isVoter()` case AND `quorumConfig.autoJoin() == true`.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3352,8 +3364,13 @@ private boolean 
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
  * and are configured to auto join should attempt to automatically 
join the voter
  * set for the configured topic partition.
  */
-return partitionState.lastKraftVersion().isReconfigSupported() && 
canBecomeVoter &&
-quorumConfig.autoJoin() && 
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
+if (!partitionState.lastKraftVersion().isReconfigSupported() || 
!canBecomeVoter ||
+!quorumConfig.autoJoin() || 
!state.hasUpdateVoterSetPeriodExpired(currentTimeMs)) {
+return false;
+}
+
+// Only attempt auto-join if we haven't already auto-joined
+return !hasAutoJoined;

Review Comment:
   ```suggestion
   return !hasAutoJoined && 
partitionState.lastKraftVersion().isReconfigSupported() && canBecomeVoter && 
quorumConfig.autoJoin() && state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
   ```
   IMO this reads cleaner. It will also short circuit after a node successfully 
auto-joins and sets `hasAutoJoined = true`, whereas this code evaluates all 
boolean expressions always.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,13 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, mark 
it as already joined

Review Comment:
   > if the starting voters contain the node id of this node, mark it as 
already joined
   
   This comment is inaccurate. We are checking `lastVoterSet()`, which is the 
VotersRecord with the highest 

Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


TaiJuWu commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3577632887

   Hi @kevin-wu24 , thanks for your detailed review and suggestions, all 
comments are addressed, PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2561448757


##
docs/upgrade.html:
##
@@ -171,8 +171,8 @@ Notable changes in 4
 
 
 
-The controller.quorum.auto.join.enable has been added to 
QuorumConfig, enabling KRaft controllers to automatically join the 
cluster's voter set,
-and defaults to false.
+The controller.quorum.auto.join.enable has been added to 
QuorumConfig, enabling KRaft controllers to automatically join the 
cluster's voter set
+when the node startup, and defaults to false.

Review Comment:
   Update it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2560048766


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean canAutoJoin = true;

Review Comment:
   It seems to me that the intention of the change is the following semantic: 
   
   For each lifetime of the controller process on a KRaft node, the node should 
be in the "auto-joining" state at most once (i.e. Observer state && 
`shouldSendAddOrRemoveVoterRequest(state, currentTimeMs)`). This means when we 
remove a node and it goes back to Observer state, the removed node will not 
automatically join back via `AddVoterRPC`. However, if the user removes the 
node as a voter and then the node restarts, the node will still try to 
auto-join the voter set.
   
   I think a better name for this boolean is `hasAutoJoined` with a value on 
startup of `false`. In `initialize()`, each controller node checks their 
`partitionState.lastVoterSet()` and sets `hasAutoJoined = true` if they are a 
part of the voter set already. Whenever an observer that has auto-join enabled 
successfully joins the voter set, we can set `hasAutoJoined = true`.
   
   One issue with the proposed semantic is that we don't know if previous 
additions to the voter set were a result of auto-join or manually adding the 
voter. Remember that the `auto.join.enable` config can also change when a node 
is restarted. I'm not sure how big of an issue that is, but there are a lot of 
edge cases around this feature that lead to confusing UX (i.e. this node is 
auto-joining even though I don't "expect" it to) if we try to implement a 
semantic like the above. That is one reason why we wanted to keep the 
pre-requisite for executing the auto-join code simple.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2560048766


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean canAutoJoin = true;

Review Comment:
   It seems to me that the intention of the change is the following semantic: 
   
   For each lifetime of the controller process on a KRaft node, the node should 
be in the "auto-joining" state at most once (i.e. Observer state && 
`shouldSendAddOrRemoveVoterRequest(state, currentTimeMs)`). This means when we 
remove a node and it goes back to Observer state, the removed node will not 
automatically join back via `AddVoterRPC`. However, if the user removes the 
node as a voter and then the node restarts, the node will still try to 
auto-join the voter set.
   
   I think a better name for this boolean is `hasAutoJoined` with a value on 
startup of `false`. In `initialize()`, each controller node checks their 
`partitionState.lastVoterSet()` and sets `hasAutoJoined = true` if they are a 
part of the voter set already. Whenever an observer that has auto-join enabled 
successfully joins the voter set, we can set `hasAutoJoined = true`.
   
   The main problem with this functionality is that we don't know if previous 
additions to the voter set were a result of auto-join or manually adding the 
voter. Remember that the `auto.join.enable` config can also change when a node 
is restarted. I'm not sure how big of an issue that is, but there are a lot of 
edge cases around this feature that lead to confusing UX (i.e. this node is 
auto-joining even though I don't "expect" it to) if we try to implement a 
semantic like the above. That is one reason why we wanted to keep the 
pre-requisite for executing the auto-join code simple.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2560048766


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean canAutoJoin = true;

Review Comment:
   It seems to me that the intention of the change is the following semantic: 
   
   For each lifetime of the controller process on a KRaft node, the node should 
be in the "auto-joining" state at most once (i.e. Observer state && 
`shouldSendAddOrRemoveVoterRequest(state, currentTimeMs)`). This means when we 
remove a node and it goes back to Observer state, the removed node will not 
automatically join back via `AddVoterRPC`. However, if the user removes the 
node as a voter and then the node restarts, the node will still try to 
auto-join the voter set.
   
   I think a better name for this boolean is `hasAutoJoined` with a value on 
startup of `false`. In `initialize()`, each controller node checks their 
`partitionState.lastVoterSet()` and sets `hasAutoJoined = true` if they are a 
part of the voter set already. Whenever an observer that has auto-join enabled 
successfully joins the voter set, we can set `hasAutoJoined = true`.
   
   The main problem with this functionality is that we don't know if previous 
additions to the voter set were a result of auto-join or manually adding the 
voter. I'm not sure how big of an issue that is, but there are a lot of edge 
cases around this feature that lead to confusing UX (i.e. this node is 
auto-joining even though I don't "expect" it to) if we have a semantic like the 
above. That is one reason why we wanted to keep the pre-requisite for executing 
the auto-join code simple.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-25 Thread via GitHub


kevin-wu24 commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2560073825


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3356,30 +3366,56 @@ private boolean 
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
 quorumConfig.autoJoin() && 
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
 }
 
+private boolean shouldSendAddVoterRequest(FollowerState state, long 
currentTimeMs) {
+return canAutoJoin && shouldAutoJoin(state, currentTimeMs);
+}
+
+private boolean shouldSendRemoveVoterRequest(FollowerState state, long 
currentTimeMs) {
+final var localReplicaKey = quorum.localReplicaKeyOrThrow();
+final var voters = partitionState.lastVoterSet();
+

Review Comment:
   In my opinion, these are too many boolean states and they are very difficult 
to reason about. If we add an additional `&& !hasAutoJoined` to the 
`shouldSendAddOrRemoveVoterRequest` predicate and ensure `hasAutoJoined` is set 
properly, we can keep all of the existing auto-join code the same.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3347,7 +3357,7 @@ private long pollFollowerAsVoter(FollowerState state, 
long currentTimeMs) {
 );
 }
 
-private boolean shouldSendAddOrRemoveVoterRequest(FollowerState state, 
long currentTimeMs) {
+private boolean shouldAutoJoin(FollowerState state, long currentTimeMs) {

Review Comment:
   Let's keep this method name as is.



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,11 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if the starting voters contain the node id of this node, it 
can't auto join to the cluster
+// because it is already in.
+canAutoJoin = 
!partitionState.lastVoterSet().voterIds().contains(nodeId.getAsInt());
+}

Review Comment:
   I think I understand what this is trying to do. Basically, if the local node 
is already part of the voter set we should avoid sending `AddVoterRPC`.  
However, there is an issue with this code:
   
   With KIP-853 enabled, a voter is no longer just a single Integer ID, but 
rather a tuple of `` represented by the `ReplicaKey` class, so 
checking the ID is not sufficient. Look at `QuorumState#isVoter()` for the 
right way to do this check.



##
docs/upgrade.html:
##
@@ -171,8 +171,8 @@ Notable changes in 4
 
 
 
-The controller.quorum.auto.join.enable has been added to 
QuorumConfig, enabling KRaft controllers to automatically join the 
cluster's voter set,
-and defaults to false.
+The controller.quorum.auto.join.enable has been added to 
QuorumConfig, enabling KRaft controllers to automatically join the 
cluster's voter set
+when the node startup, and defaults to false.

Review Comment:
   We should also add the documentation from the KIP here to make this more 
clear: 
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217391519#KIP853:KRaftControllerMembershipChanges-controller.quorum.auto.join.enable



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -222,6 +222,8 @@ public final class KafkaRaftClient implements 
RaftClient {
 private volatile RemoveVoterHandler removeVoterHandler;
 private volatile UpdateVoterHandler updateVoterHandler;
 
+private volatile boolean canAutoJoin = true;

Review Comment:
   It seems to me that the intention of the change is the following semantic: 
   
   For each lifetime of the controller process on a KRaft node, the node should 
be in the "auto-joining" state at most once (i.e. Observer state && 
`shouldSendAddOrRemoveVoterRequest(state, currentTimeMs)`). This means when we 
remove a node and it goes back to Observer state, the removed node will not 
automatically join back via `AddVoterRPC`. However, if the user removes the 
node as a voter and then the node restarts, the node will still try to 
auto-join the voter set.
   
   I think a better name for this boolean is `hasAutoJoined` with a value on 
startup of `false`. In `initialize()`, each controller node checks their 
`partitionState.lastVoterSet()` and sets `hasAutoJoined = true` if they are a 
part of the voter set already. Whenever an observer that has auto-join enabled 
successfully joins the voter set, we can set `hasAutoJoined = true`.
   
   The main problem with this functionality is that we don't know if previous 
additions to the voter set were a result of auto-join or manually adding the 
voter. I'm not sure how big of an issue that is, but there are a lot of edge 
cases around this feature if we have 

Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-24 Thread via GitHub


kevin-wu24 commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3573689124

   @TaiJuWu I can take a look tomorrow. Thanks for the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-24 Thread via GitHub


showuon commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3573683953

   @kevin-wu24 @jsancio , do you want to have another look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-24 Thread via GitHub


TaiJuWu commented on PR #20859:
URL: https://github.com/apache/kafka/pull/20859#issuecomment-3570844088

   > LGTM! @TaiJuWu , please update the PR description. Thanks.
   
   Updated description  and merge trunk, thanks for review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-21 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2549453089


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId = 
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send 
AddVoter request automatically.
+Thread.sleep(2000);
+
+// Verify 3002 is already fetch and does not send add voter 
request
+long removedAtHighWatermark = 
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong();
+TestUtils.waitForCondition(() ->
+
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
 > removedAtHighWatermark,
+30_000, 100, () -> "High watermark is not advanced in 
3ms"
+);
+
+// 3002 does not join the voter set after high watermark 
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+}

Review Comment:
   Nice suggestion, add it to the end of this test but the Set\ 
comes from line 253 to avoiding NPE.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-21 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2549453089


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId = 
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send 
AddVoter request automatically.
+Thread.sleep(2000);
+
+// Verify 3002 is already fetch and does not send add voter 
request
+long removedAtHighWatermark = 
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong();
+TestUtils.waitForCondition(() ->
+
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
 > removedAtHighWatermark,
+30_000, 100, () -> "High watermark is not advanced in 
3ms"
+);
+
+// 3002 does not join the voter set after high watermark 
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+}

Review Comment:
   Nice suggestion, add it to the end of this test but the Set\ 
comes from line 3002 to avoiding NPE.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-21 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2549453089


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId = 
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send 
AddVoter request automatically.
+Thread.sleep(2000);
+
+// Verify 3002 is already fetch and does not send add voter 
request
+long removedAtHighWatermark = 
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong();
+TestUtils.waitForCondition(() ->
+
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
 > removedAtHighWatermark,
+30_000, 100, () -> "High watermark is not advanced in 
3ms"
+);
+
+// 3002 does not join the voter set after high watermark 
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+}

Review Comment:
   Nice suggestion, add it to the end of this test but the Set 
comes from line 3002 to avoiding NPE.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-21 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2549453089


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId = 
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send 
AddVoter request automatically.
+Thread.sleep(2000);
+
+// Verify 3002 is already fetch and does not send add voter 
request
+long removedAtHighWatermark = 
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong();
+TestUtils.waitForCondition(() ->
+
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
 > removedAtHighWatermark,
+30_000, 100, () -> "High watermark is not advanced in 
3ms"
+);
+
+// 3002 does not join the voter set after high watermark 
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+}

Review Comment:
   Nice suggestion, add it to the end of this test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548678971


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId = 
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+// We need to wait update timer to expire and then send 
AddVoter request automatically.
+Thread.sleep(2000);

Review Comment:
   1. We can add a variable for the 2 sec to explain it. Ex: 
`defaultUpdateVoterSetPeriodTimeout = 2000`. 
   2. It's "update voter set timer".
   3. `We need to wait for the update voter set timer expiring to allow the 
addVoter request to be sent.`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548669325


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId = 
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}

Review Comment:
   This can be removed now because we already do the assertion below.



##
raft/src/test/java/org/apache/kafka/raft/KafkaRaftClientAutoJoinTest.java:
##
@@ -317,6 +321,131 @@ private void pollAndDeliverFetchToUpdateVoterSet(
 context.client.poll();
 }
 
+@Test
+public void testBootstrapVoterSetDoesNotSendAddVoterAfterRemove() throws 
Exception {

Review Comment:
   Nice tests added in KafkaRaftClientAutoJoinTest. Thanks.



##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,75 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+Uuid dirId = 
cluster.nodes().controllerNodes().get(3002).metadataDirectoryId();
+admin.removeRaftVoter(3002, dirId).all().get();
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+assertEquals(Set.of(3000, 

Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548605894


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,74 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});

Review Comment:
   Good catch, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548531651


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,74 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});

Review Comment:
   This work, though, it's not clear because if the voters update takes some 
time, you could retry many times. And on each retry, you'll try to remove voter 
3002 again and again. We can do something similar like 
[this](https://github.com/apache/kafka/blob/trunk/server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java#L179-L186).
 It will remove voter once, and retry the validation until the voter updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548463582


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() -> 
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in 
3ms"
+);
+
+// 3002 does not join the voter set after high watermark 
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}

Review Comment:
   Sorry, I was wrong. In L277, we already verify that the voters are only 
[3000, 3001], so it is fine now. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2548357630


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() -> 
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in 
3ms"
+);
+
+// 3002 does not join the voter set after high watermark 
advance
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}

Review Comment:
   This is not right. Here, we want to verify `3002 does not join the voter set 
after high watermark advance`, but you're testing if [3000, 3001] is in the 
voters set. So, if 3002 is joined the voters, the test can still pass, right?



##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+

Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546323712


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different 
directory ID.
+// This simulates the case where the controller failed and is brought 
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+// Remove 3002 from voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be removed
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// do not join the voter set in the next twenty seconds
+for (int i = 0; i < 20; ++i) {
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () 
-> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be 
removed
+return;
+}
+
+admin.removeRaftVoter(3002, 
voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+});
+Thread.sleep(1000);
+}

Review Comment:
   > This test doesn't make sense to me.
   > 
   > 1. `do not join the voter set in the next twenty seconds`: The 20 seconds 
is not true because the `retryOnExceptionWithTimeout` could take up to 30 
seconds one time.
   
   It make sense to me and time-base test is not a good thing as you say bellow.
   
   > 2. Why should we use 2 `retryOnExceptionWithTimeout` here?
   
   My mistake.
   
   > 3. The test is not testing what we expected.
   >3.1. What are you asserting here? I can only see you assert the node 
3000, 3001 are in the voters, but we want to test 3002, right? If 3002 is in 
the voters, the test can still be passed!?
   
   From my understanding, only the voter keys (3000 and 3001) are in the 
cluster is the same meaning as 3002 is removed.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: 

Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546716203


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different 
directory ID.
+// This simulates the case where the controller failed and is brought 
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}

Review Comment:
   removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546691075


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() -> 
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in 
3ms"
+);

Review Comment:
   @showuon Your suggestion make sense to me but there is an issue which is the 
high watermark is tuned base on different compute.
   If I use 
   ```
   
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
 > removedAtHighWatermark.get() 
   ```
   
   The original code is still passed, you can use 
b0136c1f8c8beb3ce853a83f9b43f9d1ed623e9b to check.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546691075


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() -> 
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in 
3ms"
+);

Review Comment:
   @showuon Your suggestion make sense to me but there is an issue which is the 
high watermark is tuned base on different compute.
   If I use 
   ```
   
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
 > removedAtHighWatermark.get() 
   ```
   
   The original code is still passed, you can use 
b0136c1f8c8beb3ce853a83f9b43f9d1ed623e9b to check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-20 Thread via GitHub


TaiJuWu commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2546222726


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3356,30 +3366,56 @@ private boolean 
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
 quorumConfig.autoJoin() && 
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
 }
 
+private boolean shouldSendAddVoterRequest(FollowerState state, long 
currentTimeMs) {
+return canAutoJoin && maybeAutoJoin(state, currentTimeMs);
+}
+
+private boolean shouldSendRemoveVoterRequest(FollowerState state, long 
currentTimeMs) {
+final var localReplicaKey = quorum.localReplicaKeyOrThrow();
+final var voters = partitionState.lastVoterSet();
+
+if (voters.voterIds().contains(localReplicaKey.id())) {
+if (maybeAutoJoin(state, currentTimeMs)) {
+// When the bootstrap controller needs to update directory id,
+// it should be removed and then rejoining to cluster
+// In such a case, we should set canAutoJoin to true, and it 
will
+// be removed from the cluster, update directory id and rejoin
+// to the cluster.

Review Comment:
   The disk failure is not only the case, one case is at line 3403.
   
   ```
/* The replica's id is in the voter set but the replica is not a voter 
because
 * the directory id of the voter set entry is different. Remove the old 
voter.
 * Local replica is not in the voter set because the replica is an observer.
 */
   ```



##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +219,70 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+final Map initialVoters = new HashMap<>();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(), controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+AtomicLong removedAtHighWatermark = new AtomicLong();
+// Remove 3002 from the voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be return
+
removedAtHighWatermark.set(cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong());
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+TestUtils.waitForCondition(() -> 
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
+> removedAtHighWatermark.get() + 10,
+30_000, 100, () -> "High watermark is not advanced in 
3ms"
+);

Review Comment:
   @showuon Your suggestion make sense to me but there is an issue which is the 
high watermark is tuned base on different compute.
   If I use 
   ```
   
cluster.controllers().get(3002).raftManager().client().highWatermark().getAsLong()
 > removedAtHighWatermark.get() 
   ```
   
   The original code is still passed.



##

Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-19 Thread via GitHub


showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2544638061


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different 
directory ID.
+// This simulates the case where the controller failed and is brought 
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+// Remove 3002 from voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be removed
+return;
+}
+
+admin.removeRaftVoter(3002, voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+
+// do not join the voter set in the next twenty seconds
+for (int i = 0; i < 20; ++i) {
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () 
-> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be 
removed
+return;
+}
+
+admin.removeRaftVoter(3002, 
voters.get(3002)).all().get();
+assertEquals(Set.of(3000, 3001), voters.keySet());
+for (int replicaId : new int[] {3000, 3001}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+});
+Thread.sleep(1000);
+}

Review Comment:
   We can test won't auto join the removed node by:
   
   1. Remove a controller node, ex: 3002.
   2. Verify the highwatermark in controller metadata log is advancing by:
   `cluster.raftManagers().get(3002)..client().highWatermark()`
   The high watermark increasing means the follower is fetching the data from 
the leader, and because we auto join the controller node before fetching from 
the leader, if the node can be auto joined, it'll be already added.
   3. Verify the voters are still 3000, 3001, without 3002.
   
   So, we don't need the mysterious 20 times loop to do the voter removal.
   
   What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-19 Thread via GitHub


showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2544580605


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different 
directory ID.
+// This simulates the case where the controller failed and is brought 
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}

Review Comment:
   What is this for?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-19 Thread via GitHub


showuon commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2544517164


##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different 
directory ID.
+// This simulates the case where the controller failed and is brought 
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+for (int replicaId : new int[] {3000, 3001, 3002}) {
+
assertEquals(nodes.controllerNodes().get(replicaId).metadataDirectoryId(), 
voters.get(replicaId));
+}
+});
+
+// Remove 3002 from voter set
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+if (!voters.containsKey(3002)) {
+// if there are no node 3002, it should be removed
+return;
+}

Review Comment:
   We should fail the test in this unexpected case, right?



##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -501,6 +503,11 @@ public void initialize(
 logger.info("Reading KRaft snapshot and log as part of the 
initialization");
 partitionState.updateState();
 logger.info("Starting voters are {}", partitionState.lastVoterSet());
+if (nodeId.isPresent()) {
+// if starting voters contain node id, it can't join to the cluster
+// because it is already in.

Review Comment:
   nit: if [the] starting voters contain [the node id of this node], it can't 
[auto] join to the cluster ...



##
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##
@@ -218,6 +218,82 @@ public void testControllersAutoJoinStandaloneVoter() 
throws Exception {
 }
 }
 
+@Test
+public void testRemovedControllerWontJoinAgain() throws Exception {
+final var nodes = new TestKitNodes.Builder().
+setNumBrokerNodes(1).
+setNumControllerNodes(3).
+build();
+
+// Configure the initial voters with one voter having a different 
directory ID.
+// This simulates the case where the controller failed and is brought 
back up with a different directory ID.
+final Map initialVoters = new HashMap<>();
+final var oldDirectoryId = Uuid.randomUuid();
+for (final var controllerNode : nodes.controllerNodes().values()) {
+initialVoters.put(
+controllerNode.id(),
+controllerNode.id() == TestKitDefaults.CONTROLLER_ID_OFFSET ?
+oldDirectoryId : controllerNode.metadataDirectoryId()
+);
+}
+
+try (KafkaClusterTestKit cluster = new 
KafkaClusterTestKit.Builder(nodes).
+setConfigProp(QuorumConfig.QUORUM_AUTO_JOIN_ENABLE_CONFIG, 
true).
+setInitialVoterSet(initialVoters).
+build()
+) {
+cluster.format();
+cluster.startup();
+try (var admin = Admin.create(cluster.clientProperties())) {
+// Static voter set is initialized
+TestUtils.retryOnExceptionWithTimeout(30_000, 100, () -> {
+Map voters = findVoterDirs(admin);
+assertEquals(Set.of(3000, 3001, 3002), voters.keySet());
+  

Re: [PR] KAFKA-19850: KRaft voter auto join will add a removed voter immediately [kafka]

2025-11-19 Thread via GitHub


ppatierno commented on code in PR #20859:
URL: https://github.com/apache/kafka/pull/20859#discussion_r2542831283


##
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##
@@ -3356,30 +3366,56 @@ private boolean 
shouldSendAddOrRemoveVoterRequest(FollowerState state, long curr
 quorumConfig.autoJoin() && 
state.hasUpdateVoterSetPeriodExpired(currentTimeMs);
 }
 
+private boolean shouldSendAddVoterRequest(FollowerState state, long 
currentTimeMs) {
+return canAutoJoin && maybeAutoJoin(state, currentTimeMs);
+}
+
+private boolean shouldSendRemoveVoterRequest(FollowerState state, long 
currentTimeMs) {
+final var localReplicaKey = quorum.localReplicaKeyOrThrow();
+final var voters = partitionState.lastVoterSet();
+
+if (voters.voterIds().contains(localReplicaKey.id())) {
+if (maybeAutoJoin(state, currentTimeMs)) {
+// When the bootstrap controller needs to update directory id,
+// it should be removed and then rejoining to cluster
+// In such a case, we should set canAutoJoin to true, and it 
will
+// be removed from the cluster, update directory id and rejoin
+// to the cluster.

Review Comment:
   if I got it correctly, it can happen when, for example, for any reasons the 
controller has a new directory ID (i.e. disk failure, ...) but it's registered 
within the voters with the old directory ID. For this reason, it needs to 
remove itself with the old directory ID and re-join with the new directory ID. 
Can we improve the comment by specify the "disk failure" as an example of a new 
directory id and why an update is needed (because of voters still include the 
old directory ID)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



  1   2   >