[
https://issues.apache.org/jira/browse/KAFKA-13332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17426279#comment-17426279
]
Guozhang Wang commented on KAFKA-13332:
---------------------------------------
Yeah I agree with that.
Atm Streams do not yet have a way to tell the difference during assignment
between newly created topics and existing topics since they are all just coming
from the consumer's metadata, but what we can do is to at least log how we come
up with the max(num.partitions) of all source topics.
> New pattern-matched topic with more partitions than existing matched topics
> can crash Kafka Streams
> ---------------------------------------------------------------------------------------------------
>
> Key: KAFKA-13332
> URL: https://issues.apache.org/jira/browse/KAFKA-13332
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Priority: Major
>
> The partition count resolution logic in Streams is used to determine the
> number of partitions for any repartition topics that don't already exist.
> This is done by parsing the topology to find the number of partitions of all
> upstream topics, and taking the max. For Pattern-subscribed subtopologies,
> this means you need to ensure that at least one topic matching this pattern
> is created prior to starting up the app. That topic, or topics, will
> determine the number of partitions for any downstream repartitions.
> The problem is that repartition topics are created once, the first time the
> app is started up. After that, during each rebalance Streams will validate
> all repartition topics including checking for their existence, and verifying
> they have the correct number of partitions. This check will fail if a new
> topic is created after the first initialization, which matches the pattern
> but has more partitions than any of the existing topics.
> This means that unfortunately, you can't create a new input topic that
> matches the pattern your app is subscribed to unless it has equal or fewer
> partitions than the existing matching topics. If you do, you would need to
> stop all instances and delete the existing repartition topics before creating
> this new topic
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
