jsancio commented on code in PR #19668:
URL: https://github.com/apache/kafka/pull/19668#discussion_r2114266052
##########
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##########
@@ -2744,7 +2746,11 @@ private void handleInboundMessage(RaftMessage message,
long currentTimeMs) {
if (message instanceof RaftRequest.Inbound request) {
handleRequest(request, currentTimeMs);
} else if (message instanceof RaftResponse.Inbound response) {
- if (requestManager.isResponseExpected(response.source(),
response.correlationId())) {
+ if (requestManager.isResponseExpected(
+ response.source(),
+ response.correlationId(),
+ ApiKeys.forId(message.data().apiKey()))
+ ) {
Review Comment:
Minor but it think this formatting is a bit more consistent formatting:
```java
if (requestManager.isResponseExpected(
response.source(),
response.correlationId(),
ApiKeys.forId(message.data().apiKey())
)
) {
```
##########
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##########
@@ -3301,20 +3313,23 @@ private long maybeSendFetchToBestNode(FollowerState
state, long currentTimeMs) {
private long maybeSendFetchOrFetchSnapshot(FollowerState state, long
currentTimeMs) {
final Supplier<ApiMessage> requestSupplier;
+ final ApiKeys requestType;
if (state.fetchingSnapshot().isPresent()) {
RawSnapshotWriter snapshot = state.fetchingSnapshot().get();
long snapshotSize = snapshot.sizeInBytes();
requestSupplier = () ->
buildFetchSnapshotRequest(snapshot.snapshotId(), snapshotSize);
+ requestType = ApiKeys.FETCH_SNAPSHOT;
Review Comment:
Should we fix all of the `build.*Request` methods to instead return a
`RequestSupplier`? Quickly look at the it looks like we always these kind of
methods inside a lambda.
##########
raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java:
##########
@@ -2849,14 +2858,15 @@ private long maybeSendRequest(
long currentTimeMs,
Set<ReplicaKey> remoteVoters,
Function<Integer, Node> destinationSupplier,
- Function<ReplicaKey, ApiMessage> requestSupplier
+ Function<ReplicaKey, ApiMessage> requestSupplier,
+ ApiKeys requestType
Review Comment:
Don't you want this type?
```java
Function<ReplicaKey, RequestSupplier> requestSupplier,
```
This may fix some of the other code where you can't directly call
`build.*Request`.
##########
raft/src/test/java/org/apache/kafka/raft/KafkaRaftClientReconfigTest.java:
##########
@@ -2250,13 +2250,16 @@ void
testFollowerSendsUpdateVoterWithKraftVersion0(Errors updateVoterError) thro
new LeaderAndEpoch(OptionalInt.of(voter1.id()), epoch)
)
);
+ // polling sends a fetch because no fetches are in flight, only the
update voter
context.client.poll();
+ RaftRequest.Outbound fetchRequest = context.assertSentFetchRequest();
+ context.assertFetchRequestData(fetchRequest, epoch, 0L, 0);
Review Comment:
Hmm. Is it better to just remove this block of code and rely on the `for`
loop with `pollUntilRequest` below?
##########
raft/src/main/java/org/apache/kafka/raft/RequestManager.java:
##########
@@ -17,23 +17,26 @@
package org.apache.kafka.raft;
import org.apache.kafka.common.Node;
+import org.apache.kafka.common.protocol.ApiKeys;
+import org.apache.kafka.raft.internals.RequestType;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
-import java.util.Iterator;
import java.util.Map;
import java.util.Optional;
import java.util.OptionalLong;
import java.util.Random;
/**
- * The request manager keeps tracks of the connection with remote replicas.
+ * The request manager keeps track of the pending requests with remote
replicas. The manager supports
+ * one pending request per type per node, except for FETCH and FETCH_SNAPSHOT
requests. For those,
+ * the manager allows at most one pending request across all nodes to prevent
writing the same offset twice.
*
- * When sending a request update this type by calling {@code
onRequestSent(Node, long, long)}. When
- * the RPC returns a response, update this manager with {@code
onResponseResult(Node, long, boolean, long)}.
+ * When sending a request update this type by calling {@code
onRequestSent(Node, long, long, ApiKeys)}.
+ * When the RPC returns a response, update this manager with {@code
onResponseResult(Node, long, boolean, long, ApiKeys)}.
*
- * Connections start in the ready state ({@code isReady(Node, long)} returns
true).
+ * Requests start in the ready state ({@code isReady(Node, long, ApiKeys)}
returns true).
*
* When a request times out or completes successfully the collection will
transition back to the
* ready state.
Review Comment:
I think this paragraph meant to meant to say
```java
* When a request times out or completes successfully the connection will
transition back to the
* ready state.
```
We can change this to
```java
* When a request times out or completes successfully the request state will
transition back to the
* ready state.
```
##########
raft/src/main/java/org/apache/kafka/raft/RequestManager.java:
##########
@@ -62,30 +70,35 @@ public RequestManager(
}
/**
- * Returns true if there are any connections with pending requests.
+ * Returns true if there are any in-flight requests for a request type.
*
- * This is useful for satisfying the invariant that there is only one
pending Fetch request.
+ * This is useful for satisfying the invariant that there is only one
pending Fetch
+ * and FetchSnapshot request.
* If there are more than one pending fetch request, it is possible for
the follower to write
* the same offset twice.
*
* @param currentTimeMs the current time
- * @return true if the request manager is tracking at least one request
+ * @param wantRequestKey the request type to check for in-flight requests
+ * @return true if the request manager is tracking at least one request of
the given type
*/
- public boolean hasAnyInflightRequest(long currentTimeMs) {
+ public boolean hasAnyInflightRequest(long currentTimeMs, ApiKeys
wantRequestKey) {
boolean result = false;
- Iterator<ConnectionState> iterator = connections.values().iterator();
+ final var iterator = inflightRequests.entrySet().iterator();
while (iterator.hasNext()) {
- ConnectionState connection = iterator.next();
- if (connection.hasRequestTimedOut(currentTimeMs)) {
+ final var entry = iterator.next();
+ final var requestKey = entry.getKey().requestType().apiKey();
+ final var requestState = entry.getValue();
+ if (requestState.hasRequestTimedOut(currentTimeMs)) {
// Mark the node as ready after request timeout
iterator.remove();
- } else if (connection.isBackoffComplete(currentTimeMs)) {
+ } else if (requestState.isBackoffComplete(currentTimeMs)) {
// Mark the node as ready after completed backoff
iterator.remove();
- } else if (connection.hasInflightRequest(currentTimeMs)) {
+ } else if (requestKey == wantRequestKey &&
Review Comment:
We should avoid comparing `ApiKeys`. We want to compare `RequestType`, no? I
think we are missing tests if all of the test pass.
##########
raft/src/main/java/org/apache/kafka/raft/internals/RequestType.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.raft.internals;
+
+import org.apache.kafka.common.protocol.ApiKeys;
+
+/**
+ * This class is used to wrap the ApiKeys enum for KRaft RPCs so the KRaft
request
+ * manager can treat the FETCH and FETCH_SNAPSHOT requests as the same type
when
+ * managing in-flight requests. This is useful for satisfying the invariant
+ * that at most one FETCH or FETCH_SNAPSHOT request is pending at any time.
+ */
+public class RequestType {
+ private final ApiKeys apiKey;
Review Comment:
I would add a static for FETCH_AND_FEATCH_SNAPSHOT:
```java
public class RequestType {
private final static RequestType FETCH_AND_FEATCH_SNAPSHOT = new
RequestType(ApiKeys.FETCH);
private final ApiKeys apiKey;
```
And use that in the static method `of`.
##########
raft/src/test/java/org/apache/kafka/raft/RequestManagerTest.java:
##########
@@ -234,15 +330,61 @@ public void testFindReadyWithRequestTimedOut() {
);
// Send request to a node that is not in the bootstrap list
- cache.onRequestSent(otherNode, 1, time.milliseconds());
- assertTrue(cache.isResponseExpected(otherNode, 1));
+ cache.onRequestSent(otherNode, 1, time.milliseconds(), fetch);
+ assertTrue(cache.isResponseExpected(otherNode, 1, fetch));
assertEquals(Optional.empty(),
cache.findReadyBootstrapServer(time.milliseconds()));
// Timeout the request
time.sleep(requestTimeoutMs);
Node bootstrapNode =
cache.findReadyBootstrapServer(time.milliseconds()).get();
assertTrue(bootstrapList.contains(bootstrapNode));
- assertFalse(cache.isResponseExpected(otherNode, 1));
+ assertFalse(cache.isResponseExpected(otherNode, 1, fetch));
+ }
+
+ @Test
+ public void testAnyInflightRequestWithMultipleRequestTypes() {
+ Node otherNode = new Node(1, "other-node", 1234);
+ List<Node> bootstrapList = makeBootstrapList(3);
+ RequestManager cache = new RequestManager(
+ bootstrapList,
+ retryBackoffMs,
+ requestTimeoutMs,
+ random
+ );
+
+ assertFalse(cache.hasAnyInflightRequest(time.milliseconds(), fetch));
+ assertFalse(cache.hasAnyInflightRequest(time.milliseconds(),
updateVoter));
+
+ // Send a request and check state
+ cache.onRequestSent(otherNode, 11, time.milliseconds(), fetch);
+ assertTrue(cache.hasAnyInflightRequest(time.milliseconds(), fetch));
+ assertFalse(cache.hasAnyInflightRequest(time.milliseconds(),
updateVoter));
+
+ // Send the other request and check state
+ cache.onRequestSent(otherNode, 11, time.milliseconds(), updateVoter);
+ assertTrue(cache.hasAnyInflightRequest(time.milliseconds(), fetch));
+ assertTrue(cache.hasAnyInflightRequest(time.milliseconds(),
updateVoter));
+
+ // Wait until the request times out
+ time.sleep(requestTimeoutMs);
+ assertFalse(cache.hasAnyInflightRequest(time.milliseconds(), fetch));
+ assertFalse(cache.hasAnyInflightRequest(time.milliseconds(),
updateVoter));
+
+ // Results should not affect the connection state of other request
types
+ cache.onRequestSent(otherNode, 12, time.milliseconds(), updateVoter);
+
+ // Send another request and fail it
+ cache.onRequestSent(otherNode, 12, time.milliseconds(), fetch);
+ cache.onResponseResult(otherNode, 12, false, time.milliseconds(),
fetch);
+ assertFalse(cache.hasAnyInflightRequest(time.milliseconds(), fetch));
+ assertTrue(cache.hasAnyInflightRequest(time.milliseconds(),
updateVoter));
+
+ // Send fetch snapshot request, it should be treated the same as fetch
+ cache.onRequestSent(otherNode, 12, time.milliseconds(), fetchSnapshot);
+ assertTrue(cache.hasAnyInflightRequest(time.milliseconds(), fetch));
Review Comment:
This should also pass, no?
```java
assertTrue(cache.hasAnyInflightRequest(time.milliseconds(),
fetchSnapshot));
```
##########
raft/src/test/java/org/apache/kafka/raft/KafkaRaftClientReconfigTest.java:
##########
@@ -2817,9 +2830,16 @@ void testUpdateVoterResponseCausesEpochChange() throws
Exception {
)
);
- // check that there is a fetch to the new leader
+ // the first poll can still send a fetch request to the old leader,
because there is not one in flight
+ // but will handle the update voter response afterwards to update state
context.pollUntilRequest();
RaftRequest.Outbound fetchRequest = context.assertSentFetchRequest();
+ context.assertFetchRequestData(fetchRequest, epoch, 0L, 0);
+ assertEquals(voter1.id(), fetchRequest.destination().id());
Review Comment:
Interesting behavior. We should add a test that this FETCH response is
ignored because the UpdateVoterResponse above has a high epoch which forces the
local replica to transition. We can make this check here since this test is
already trying to test what happens during an epoch change.
##########
raft/src/test/java/org/apache/kafka/raft/RequestManagerTest.java:
##########
@@ -256,25 +398,25 @@ public void testAnyInflightRequestWithAnyRequest() {
random
);
- assertFalse(cache.hasAnyInflightRequest(time.milliseconds()));
Review Comment:
Do you want to parametrize this test and do it over all of the ApiKeys?
##########
raft/src/test/java/org/apache/kafka/raft/RequestManagerTest.java:
##########
@@ -234,15 +330,61 @@ public void testFindReadyWithRequestTimedOut() {
);
// Send request to a node that is not in the bootstrap list
- cache.onRequestSent(otherNode, 1, time.milliseconds());
- assertTrue(cache.isResponseExpected(otherNode, 1));
+ cache.onRequestSent(otherNode, 1, time.milliseconds(), fetch);
+ assertTrue(cache.isResponseExpected(otherNode, 1, fetch));
assertEquals(Optional.empty(),
cache.findReadyBootstrapServer(time.milliseconds()));
// Timeout the request
time.sleep(requestTimeoutMs);
Node bootstrapNode =
cache.findReadyBootstrapServer(time.milliseconds()).get();
assertTrue(bootstrapList.contains(bootstrapNode));
- assertFalse(cache.isResponseExpected(otherNode, 1));
+ assertFalse(cache.isResponseExpected(otherNode, 1, fetch));
+ }
+
+ @Test
+ public void testAnyInflightRequestWithMultipleRequestTypes() {
+ Node otherNode = new Node(1, "other-node", 1234);
+ List<Node> bootstrapList = makeBootstrapList(3);
+ RequestManager cache = new RequestManager(
+ bootstrapList,
+ retryBackoffMs,
+ requestTimeoutMs,
+ random
+ );
+
+ assertFalse(cache.hasAnyInflightRequest(time.milliseconds(), fetch));
+ assertFalse(cache.hasAnyInflightRequest(time.milliseconds(),
updateVoter));
+
+ // Send a request and check state
+ cache.onRequestSent(otherNode, 11, time.milliseconds(), fetch);
+ assertTrue(cache.hasAnyInflightRequest(time.milliseconds(), fetch));
Review Comment:
This should also pass, no?
```java
assertTrue(cache.hasAnyInflightRequest(time.milliseconds(),
fetchSnapshot));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
