[jira] [Updated] (KAFKA-10716) Streams processId is unstable across restarts resulting in task mass migration
[ https://issues.apache.org/jira/browse/KAFKA-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman updated KAFKA-10716: --- Priority: Critical (was: Major) > Streams processId is unstable across restarts resulting in task mass migration > -- > > Key: KAFKA-10716 > URL: https://issues.apache.org/jira/browse/KAFKA-10716 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: A. Sophie Blee-Goldman >Assignee: A. Sophie Blee-Goldman >Priority: Critical > Fix For: 2.8.0, 2.7.1, 2.6.2 > > > The new high availability feature of KIP-441 relies on deterministic > assignment to produce an eventually-stable assignment. The > HighAvailabilityTaskAssignor assigns tasks based on the unique processId > assigned to each client, so if the same set of Kafka Streams applications > participate in a rebalance it should generate the same task assignment every > time. > Unfortunately the processIds aren't stable across restarts. We generate a > random UUID in the KafkaStreams constructor, so each time the process starts > up it would be assigned a completely different processId. Unless this new > processId happens to be in exactly the same order as the previous one, a > single bounce or crash/restart can result in a large scale shuffling of tasks > based on a completely different eventual assignment. > Ultimately we should fix this via KAFKA-10121, but that's a nontrivial > undertaking and this bug merits some immediate relief if we don't intend to > tackle the larger problem in the upcoming releases -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10716) Streams processId is unstable across restarts resulting in task mass migration
[ https://issues.apache.org/jira/browse/KAFKA-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman updated KAFKA-10716: --- Fix Version/s: 2.6.2 > Streams processId is unstable across restarts resulting in task mass migration > -- > > Key: KAFKA-10716 > URL: https://issues.apache.org/jira/browse/KAFKA-10716 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: A. Sophie Blee-Goldman >Assignee: A. Sophie Blee-Goldman >Priority: Major > Fix For: 2.8.0, 2.7.1, 2.6.2 > > > The new high availability feature of KIP-441 relies on deterministic > assignment to produce an eventually-stable assignment. The > HighAvailabilityTaskAssignor assigns tasks based on the unique processId > assigned to each client, so if the same set of Kafka Streams applications > participate in a rebalance it should generate the same task assignment every > time. > Unfortunately the processIds aren't stable across restarts. We generate a > random UUID in the KafkaStreams constructor, so each time the process starts > up it would be assigned a completely different processId. Unless this new > processId happens to be in exactly the same order as the previous one, a > single bounce or crash/restart can result in a large scale shuffling of tasks > based on a completely different eventual assignment. > Ultimately we should fix this via KAFKA-10121, but that's a nontrivial > undertaking and this bug merits some immediate relief if we don't intend to > tackle the larger problem in the upcoming releases -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-10716) Streams processId is unstable across restarts resulting in task mass migration
[ https://issues.apache.org/jira/browse/KAFKA-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] A. Sophie Blee-Goldman updated KAFKA-10716: --- Fix Version/s: 2.7.1 2.8.0 > Streams processId is unstable across restarts resulting in task mass migration > -- > > Key: KAFKA-10716 > URL: https://issues.apache.org/jira/browse/KAFKA-10716 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.6.0 >Reporter: A. Sophie Blee-Goldman >Assignee: A. Sophie Blee-Goldman >Priority: Major > Fix For: 2.8.0, 2.7.1 > > > The new high availability feature of KIP-441 relies on deterministic > assignment to produce an eventually-stable assignment. The > HighAvailabilityTaskAssignor assigns tasks based on the unique processId > assigned to each client, so if the same set of Kafka Streams applications > participate in a rebalance it should generate the same task assignment every > time. > Unfortunately the processIds aren't stable across restarts. We generate a > random UUID in the KafkaStreams constructor, so each time the process starts > up it would be assigned a completely different processId. Unless this new > processId happens to be in exactly the same order as the previous one, a > single bounce or crash/restart can result in a large scale shuffling of tasks > based on a completely different eventual assignment. > Ultimately we should fix this via KAFKA-10121, but that's a nontrivial > undertaking and this bug merits some immediate relief if we don't intend to > tackle the larger problem in the upcoming releases -- This message was sent by Atlassian Jira (v8.3.4#803005)
