[
https://issues.apache.org/jira/browse/KAFKA-13543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17459556#comment-17459556
]
Guozhang Wang commented on KAFKA-13543:
---------------------------------------
As per our off-line sync, I think on the consumer we could fix by extending
beyond {{if (subscriptions.hasPatternSubscription()) }} to also include the
collection subscription, since for the latter it's also better to wait for at
least one metadata refresh to make sure that we have it updated before
re-joining the group.
> Consumer may pass stale cluster metadata to the assignor following a
> subscription update
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-13543
> URL: https://issues.apache.org/jira/browse/KAFKA-13543
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Reporter: A. Sophie Blee-Goldman
> Priority: Major
>
> A consumer only ever tracks metadata corresponding to its subscribed topics,
> which can cause a race condition during a rebalance immediately after a
> change to the consumer's subscription. Particularly, when new topics are
> added to the subscription but a rebalance in kicked off before the consumer's
> metadata is updated with the new topics, it will pass a stale copy of the
> cluster metadata in to the ConsumerPartitionAssignor#assign method, which may
> not include the newly subscribed topics regardless of whether they do or do
> not exist.
> Most apps are likely unaffected by this, including any consumer client apps
> using OOTB assignors, since a new rebalance will be kicked off when the
> metadata is updated and any partitions from the new topics will be assigned
> at that time. But in Kafka Streams, we do a check during each rebalance to
> ensure that any user input topics are created ahead of time. This race
> condition can result in Streams incorrectly identifying user topics as
> missing and throwing a MissingSourceTopicException when a new topology
> subscribed to new topics is added to the application
> We can work around this for now, but it's unfortunate that we can't
> distinguish between true missing source topics and a transient lack of these
> topics in the metadata. There might also be some plain consumer client apps
> with custom assignors that run into this as well, for more advanced users.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
