[
https://issues.apache.org/jira/browse/KAFKA-13332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421883#comment-17421883
]
Matthias J. Sax commented on KAFKA-13332:
-----------------------------------------
The original idea to fail if the number of partitions for an existing
repartition topic changes was meant as a "safe guard". Do we believe that this
assumption was incorrect and that we don't need such a save guard (and just
remove the check if the repartition topic exists?
Or could this be a case of "won't fix" and users would need to work around by
expliclity calling `repartition()` and set the number of partitions manually?
This issue is kinda known since we introduce auto-repartitioning (I guess
0.10.1 release?), but it never came up in practice as a problem.
> New pattern-matched topic with more partitions than existing matched topics
> can crash Kafka Streams
> ---------------------------------------------------------------------------------------------------
>
> Key: KAFKA-13332
> URL: https://issues.apache.org/jira/browse/KAFKA-13332
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: A. Sophie Blee-Goldman
> Priority: Major
>
> The partition count resolution logic in Streams is used to determine the
> number of partitions for any repartition topics that don't already exist.
> This is done by parsing the topology to find the number of partitions of all
> upstream topics, and taking the max. For Pattern-subscribed subtopologies,
> this means you need to ensure that at least one topic matching this pattern
> is created prior to starting up the app. That topic, or topics, will
> determine the number of partitions for any downstream repartitions.
> The problem is that repartition topics are created once, the first time the
> app is started up. After that, during each rebalance Streams will validate
> all repartition topics including checking for their existence, and verifying
> they have the correct number of partitions. This check will fail if a new
> topic is created after the first initialization, which matches the pattern
> but has more partitions than any of the existing topics.
> This means that unfortunately, you can't create a new input topic that
> matches the pattern your app is subscribed to unless it has equal or fewer
> partitions than the existing matching topics. If you do, you would need to
> stop all instances and delete the existing repartition topics before creating
> this new topic
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
