[
https://issues.apache.org/jira/browse/KAFKA-13405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446375#comment-17446375
]
Rajini Sivaram commented on KAFKA-13405:
----------------------------------------
I am not familiar with the specific restore consumer referred to in the ticket,
but Kafka producers and consumers resolve bootstrap servers when they start up.
All the matching addresses are cached, so some changes are ok. Clients which
are connected to brokers receive metadata from brokers and will continue to
work with a rolling upgrade that changes all IPs since those get re-resolved.
But if a client started up with bootstrap servers that resolved to old IPs and
attempts to get metadata using bootstrap servers after all ips changed, then
that wouldn't work because we don't resolve bootstrap servers again. Looks like
that is the scenario with the restore consumer.
> Kafka Streams restore-consumer fails to refresh broker IPs after upgrading
> Kafka cluster
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-13405
> URL: https://issues.apache.org/jira/browse/KAFKA-13405
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.8.0, 2.7.1
> Reporter: Daniel O'Halloran
> Priority: Critical
> Attachments: KafkaConfig.txt, KafkaLogs.txt
>
>
> *+Description+*
> After upgrading our Kafka clusters from 2.7 to 2.8 the Streams
> restore-consumers never update their broker IPs.
> The applications continue to process events as normal, until there is a
> rebalance.
> Once a rebalance occurs the restore consumers attempts to connect to the old
> brokers IPs indefinitely and the streams tasks never go back into a RUNNING
> state.
> We were able to replicate this behaviour with kafka-streams client libraries
> 2.5.1, 2.7.1 and 2.8.0
>
> *+Steps to reproduce+*
> # Upgrade brokers from Kafka 2.7 to Kafka 2.8
> # Ensure old brokers are completely shut down
> # Trigger a rebalance of a streams application
>
> *+Expected result+*
> * Application rebalances as normal
>
> *+Actual Result+*
> * Application cannot restore its data
> * restore consumer tries to connect to old brokers indefinitely
>
> *+Observations+*
> * The cluster metadata was updated on all stream consumer threads during the
> broker upgrade (multiple times) as the new brokers were brought online
> (corresponding to leader election occurring on the subscribed partitions),
> however no cluster metadata was logged on the restore-consumer thread.
> * None of the original broker IPs are valid/accessible after the upgrade (as
> expected)
> * No partition rebalance occurs during the kafka upgrade process.
> * When the first re-balance was triggered after upgrade, the
> restore-consumer loops failing to connect on each of the 3 original IPs, but
> none of the new broker IPs.
>
> *+Workaround+*
> * Restart our applications after upgrading our Kafka cluster
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
