A. Sophie Blee-Goldman created KAFKA-10716:
----------------------------------------------
Summary: Streams processId is unstable across restarts resulting
in task mass migration
Key: KAFKA-10716
URL: https://issues.apache.org/jira/browse/KAFKA-10716
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.6.0
Reporter: A. Sophie Blee-Goldman
The new high availability feature of KIP-441 relies on deterministic assignment
to produce an eventually-stable assignment. The HighAvailabilityTaskAssignor
assigns tasks based on the unique processId assigned to each client, so if the
same set of Kafka Streams applications participate in a rebalance it should
generate the same task assignment every time.
Unfortunately the processIds aren't stable across restarts. We generate a
random UUID in the KafkaStreams constructor, so each time the process starts up
it would be assigned a completely different processId. Unless this new
processId happens to be in exactly the same order as the previous one, a single
bounce or crash/restart can result in a large scale shuffling of tasks based on
a completely different eventual assignment.
Ultimately we should fix this via KAFKA-10121, but that's a nontrivial
undertaking and this bug merits some immediate relief if we don't intend to
tackle the larger problem in the upcoming releases
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
