A. Sophie Blee-Goldman created KAFKA-13543:
----------------------------------------------
Summary: Consumer may pass stale cluster metadata to the assignor
following a subscription update
Key: KAFKA-13543
URL: https://issues.apache.org/jira/browse/KAFKA-13543
Project: Kafka
Issue Type: Bug
Components: consumer
Reporter: A. Sophie Blee-Goldman
A consumer only ever tracks metadata corresponding to its subscribed topics,
which can cause a race condition during a rebalance immediately after a change
to the consumer's subscription. Particularly, when new topics are added to the
subscription but a rebalance in kicked off before the consumer's metadata is
updated with the new topics, it will pass a stale copy of the cluster metadata
in to the ConsumerPartitionAssignor#assign method, which may not include the
newly subscribed topics regardless of whether they do or do not exist.
Most apps are likely unaffected by this, including any consumer client apps
using OOTB assignors, since a new rebalance will be kicked off when the
metadata is updated and any partitions from the new topics will be assigned at
that time. But in Kafka Streams, we do a check during each rebalance to ensure
that any user input topics are created ahead of time. This race condition can
result in Streams incorrectly identifying user topics as missing and throwing a
MissingSourceTopicException when a new topology subscribed to new topics is
added to the application
We can work around this for now, but it's unfortunate that we can't distinguish
between true missing source topics and a transient lack of these topics in the
metadata. There might also be some plain consumer client apps with custom
assignors that run into this as well, for more advanced users.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
