sanghyeok An created KAFKA-21513:
------------------------------------
Summary: Streams internal topic creation retries can omit topics
and prevent group startup
Key: KAFKA-21513
URL: https://issues.apache.org/jira/browse/KAFKA-21513
Project: Kafka
Issue Type: Bug
Reporter: sanghyeok An
Assignee: sanghyeok An
After Streams internal topic creation fails, subsequent retries can omit the
affected topics even after the backoff expires. As a result, internal topics
may remain missing after the original failure condition is resolved, leaving
the group in {{{}MISSING_INTERNAL_TOPICS{}}}.
*Scenario*
# A Streams group starts with its source topics already present, but a
required internal repartition topic does not exist.
# The Group Coordinator caches the internal topic creation information in
{{{}ConfiguredTopology{}}}, and the broker uses it to send a {{CreateTopics}}
request to the controller.
# A creation request using the topic objects cached in the existing group
fails due to a temporary condition. For example, creation can fail when the
replication factor is 3 but only two brokers are available.
# Another broker becomes available, allowing topic creation to succeed, and
the error backoff expires. The topology and relevant topic metadata remain
unchanged, so the coordinator continues using the existing cache.
# During a subsequent heartbeat, internal topic creation is retried, but the
previously attempted topic is omitted from the actual {{CreateTopics}} request.
# The internal topic remains missing, preventing the group from becoming
ready. Subsequent retries can encounter the same problem while the same cache
is retained.
The retry call itself executes, but the controller does not receive a creation
entry for the affected topic. If it is the only topic being retried, the
request contains an empty {{Topics}} array. Heartbeats can continue succeeding
while the client waits for internal topics to become available.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)