A. Sophie Blee-Goldman created KAFKA-13332:
----------------------------------------------
Summary: New pattern-matched topic with more partitions than
existing matched topics can crash Kafka Streams
Key: KAFKA-13332
URL: https://issues.apache.org/jira/browse/KAFKA-13332
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: A. Sophie Blee-Goldman
The partition count resolution logic in Streams is used to determine the number
of partitions for any repartition topics that don't already exist. This is done
by parsing the topology to find the number of partitions of all upstream
topics, and taking the max. For Pattern-subscribed subtopologies, this means
you need to ensure that at least one topic matching this pattern is created
prior to starting up the app. That topic, or topics, will determine the number
of partitions for any downstream repartitions.
The problem is that repartition topics are created once, the first time the app
is started up. After that, during each rebalance Streams will validate all
repartition topics including checking for their existence, and verifying they
have the correct number of partitions. This check will fail if a new topic is
created after the first initialization, which matches the pattern but has more
partitions than any of the existing topics.
This means that unfortunately, you can't create a new input topic that matches
the pattern your app is subscribed to unless it has equal or fewer partitions
than the existing matching topics. If you do, you would need to stop all
instances and delete the existing repartition topics before creating this new
topic
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
