mjsax commented on code in PR #20326:
URL: https://github.com/apache/kafka/pull/20326#discussion_r2281010781
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java:
##########
@@ -461,120 +466,119 @@ public Set<String> makeReady(final Map<String,
InternalTopicConfig> topics) {
// have existed with the expected number of partitions, or some create
topic returns fatal errors.
log.debug("Starting to validate internal topics {} in partition
assignor.", topics);
- long currentWallClockMs = time.milliseconds();
+ final long currentWallClockMs = time.milliseconds();
final long deadlineMs = currentWallClockMs + retryTimeoutMs;
- Set<String> topicsNotReady = new HashSet<>(topics.keySet());
- final Set<String> newlyCreatedTopics = new HashSet<>();
+ final Set<String> topicsNotReady = new HashSet<>(topics.keySet());
+ final Set<String> newTopics = new HashSet<>();
while (!topicsNotReady.isEmpty()) {
- final Set<String> tempUnknownTopics = new HashSet<>();
- topicsNotReady = validateTopics(topicsNotReady, topics,
tempUnknownTopics);
- newlyCreatedTopics.addAll(topicsNotReady);
-
+ final Set<NewTopic> topicsToCreate = computeTopicsToCreate(topics,
topicsNotReady, newTopics);
+ if (!topicsToCreate.isEmpty()) {
+ readyTopics(topicsToCreate, topicsNotReady);
+ }
if (!topicsNotReady.isEmpty()) {
- final Set<NewTopic> newTopics = new HashSet<>();
+ maybeThrowTimeout(new TimeoutContext(
+ Collections.singleton("makeReadyCheck"), // dummy
collection just to trigger if `topicsNotReady` is non-empty
+ deadlineMs,
+ "MakeReady timeout",
+ String.format("Could not create topics within %d
milliseconds. This can happen if the Kafka cluster is temporarily not
available.", retryTimeoutMs),
+ null
+ ));
- for (final String topicName : topicsNotReady) {
- if (tempUnknownTopics.contains(topicName)) {
- // for the tempUnknownTopics, don't create topic for
them
- // we'll check again later if remaining retries > 0
- continue;
- }
- final InternalTopicConfig internalTopicConfig =
Objects.requireNonNull(topics.get(topicName));
- final Map<String, String> topicConfig =
internalTopicConfig.properties(defaultTopicConfigs,
windowChangeLogAdditionalRetention);
+ }
+ }
+ log.debug("Completed validating internal topics and created {}",
newTopics);
- log.debug("Going to create topic {} with {} partitions and
config {}.",
- internalTopicConfig.name(),
- internalTopicConfig.numberOfPartitions(),
- topicConfig);
+ return newTopics;
+ }
+
+ private Set<NewTopic> computeTopicsToCreate(final Map<String,
InternalTopicConfig> topics,
+ final Set<String>
topicsNotReady,
+ final Set<String>
newTopics) {
+ final Set<String> tempUnknownTopics = new HashSet<>();
+ final Set<String> validatedTopics = validateTopics(topicsNotReady,
topics, tempUnknownTopics);
Review Comment:
> The return value represents topics that are configured correctly, and that
also need to get created
Well, the method does validate topics **_if_** they exist. That's why the
method is called `validateTopics` -- but topics that do not exist, can by
definition not be validated, but need to be created. So the returned set of
topics, as you also confirmed (`also need to get created`), are non-existing
topics, and thus, they did not get validated as there is nothing to be done
("validation" mean, checks if an existing topic meets the expected partition
count).
Does this make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
