[
https://issues.apache.org/jira/browse/KAFKA-13405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax resolved KAFKA-13405.
-------------------------------------
Resolution: Not A Bug
> Kafka Streams restore-consumer fails to refresh broker IPs after upgrading
> Kafka cluster
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-13405
> URL: https://issues.apache.org/jira/browse/KAFKA-13405
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 2.8.0, 2.7.1
> Reporter: Daniel O'Halloran
> Priority: Critical
> Attachments: KafkaConfig.txt, KafkaLogs.txt
>
>
> *+Description+*
> After upgrading our Kafka clusters from 2.7 to 2.8 the Streams
> restore-consumers never update their broker IPs.
> The applications continue to process events as normal, until there is a
> rebalance.
> Once a rebalance occurs the restore consumers attempts to connect to the old
> brokers IPs indefinitely and the streams tasks never go back into a RUNNING
> state.
> We were able to replicate this behaviour with kafka-streams client libraries
> 2.5.1, 2.7.1 and 2.8.0
>
> *+Steps to reproduce+*
> # Upgrade brokers from Kafka 2.7 to Kafka 2.8
> # Ensure old brokers are completely shut down
> # Trigger a rebalance of a streams application
>
> *+Expected result+*
> * Application rebalances as normal
>
> *+Actual Result+*
> * Application cannot restore its data
> * restore consumer tries to connect to old brokers indefinitely
>
> *+Observations+*
> * The cluster metadata was updated on all stream consumer threads during the
> broker upgrade (multiple times) as the new brokers were brought online
> (corresponding to leader election occurring on the subscribed partitions),
> however no cluster metadata was logged on the restore-consumer thread.
> * None of the original broker IPs are valid/accessible after the upgrade (as
> expected)
> * No partition rebalance occurs during the kafka upgrade process.
> * When the first re-balance was triggered after upgrade, the
> restore-consumer loops failing to connect on each of the 3 original IPs, but
> none of the new broker IPs.
>
> *+Workaround+*
> * Restart our applications after upgrading our Kafka cluster
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
