[GitHub] [kafka] ableegoldman commented on a change in pull request #8662: HOTFIX: skip listOffsets request for newly created changelog topics
ableegoldman commented on a change in pull request #8662:
URL: https://github.com/apache/kafka/pull/8662#discussion_r425314470
##
File path:
streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java
##
@@ -169,6 +173,9 @@ public void makeReady(final Map topics) {
log.error(timeoutAndRetryError);
throw new StreamsException(timeoutAndRetryError);
}
+log.debug("Completed validating internal topics and created {}",
newlyCreatedTopics);
Review comment:
I do agree it would be useful though. Feel free to create a ticket :P
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #8662: HOTFIX: skip listOffsets request for newly created changelog topics
ableegoldman commented on a change in pull request #8662:
URL: https://github.com/apache/kafka/pull/8662#discussion_r425314263
##
File path:
streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java
##
@@ -169,6 +173,9 @@ public void makeReady(final Map topics) {
log.error(timeoutAndRetryError);
throw new StreamsException(timeoutAndRetryError);
}
+log.debug("Completed validating internal topics and created {}",
newlyCreatedTopics);
Review comment:
I think this race condition was particularly severe since we do the
listOffsets request pretty much immediately after creating the topics, whereas
whatever we're doing with that topic next will not be until the rebalance was
completed.
AFAIK we've never had any users report subsequent operations failing after
the first rebalance due to not-yet-fully-created topics, but it could have just
slipped past us
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #8662: HOTFIX: skip listOffsets request for newly created changelog topics
ableegoldman commented on a change in pull request #8662:
URL: https://github.com/apache/kafka/pull/8662#discussion_r424751772
##
File path:
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamsPartitionAssignorTest.java
##
@@ -1862,6 +1866,35 @@ public void
shouldThrowIllegalStateExceptionIfAnyTopicsMissingFromChangelogEndOf
assertThrows(IllegalStateException.class, () ->
partitionAssignor.assign(metadata, new GroupSubscription(subscriptions)));
}
+@Test
+public void shouldSkipListOffsetsRequestForNewlyCreatedChangelogTopics() {
+adminClient = EasyMock.createMock(AdminClient.class);
+final ListOffsetsResult result =
EasyMock.createNiceMock(ListOffsetsResult.class);
+final KafkaFutureImpl>
allFuture = new KafkaFutureImpl<>();
+allFuture.complete(emptyMap());
+
+expect(adminClient.listOffsets(emptyMap())).andStubReturn(result);
+expect(result.all()).andReturn(allFuture);
+
+builder.addSource(null, "source1", null, null, null, "topic1");
+builder.addProcessor("processor1", new MockProcessorSupplier(),
"source1");
+builder.addStateStore(new MockKeyValueStoreBuilder("store1", false),
"processor1");
+
+subscriptions.put("consumer10",
+ new Subscription(
+ singletonList("topic1"),
+ defaultSubscriptionInfo.encode()
+ ));
+
+EasyMock.replay(result);
Review comment:
It gets replayed during configuration (at the end of `configureDefault`
below)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
