[
https://issues.apache.org/jira/browse/KAFKA-21142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Westerby updated KAFKA-21142:
-------------------------------------
Description:
Kafka quorum controller leader may fail to correctly update state in ZK when it
crashes in the following scenario
*Setup*
Consider a 3 controller quorum in dual-write mode, i.e. the quorum leader is
acting as the coordinator and owning writes to ZK via KRaftMigrationDriver.
Also consider the brokers to not be bounced in KRaft mode.
*Failure Scenario*
1. Let's say a client sends a request to increase the topic-partition count of
topic {{T1}} from {{2}} to {{3}}
2. After the request is processed by the quorum leader, the delta is propagated
to KRaftMigrationDriver via the callback {{{}onMetadataUpdate{}}}, which
in-turn appends a {{MetadataChangeEvent}} to the event queue. Ultimately, the
control flow ends up at {{{}KRaftMigrationZkWriter#handleDelta{}}}, followed by
{{{}KRaftMigrationZkWriter#handleTopicsDelta{}}}. _Let's assume the quorum
leader crashes at this point before any updates to ZK._
3. Eventually, a new quorum leader is elected and it attempts to synchronise
the state with ZK while in {{SYNC_KRAFT_TO_ZK}} state. This results in
{{SyncKRaftMetadataEvent}} and eventually the invocation of
{{KRaftMigrationZkWriter#handleSnapshot}} followed by
{{KRaftMigrationZkWriter#handleTopicsSnapshot}}
4. Eventually,
[newPartitions|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L217]
and
[changedTopics|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L224]
are populated.
5. {{changedTopics}} results in the {{TopicZnode}} being updated with
information about the new replica
6. {{newPartitions}} should result in {{TopicPartitionStateZnode}} being
updated but the handler incorrectly calls
{{migrationClient.topicClient().updateTopicPartitions()}} instead of
{{migrationClient.topicClient().createTopicPartitions}}
[here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L259]
7. This results in a SetData on a ZNode which doesn't exist and ultimately a
{{MigrationClientException}} which results in the {{SyncKRaftMetadataEvent}} to
be retried.
8. This time around, {{newPartitions}} is not populated because
{{ZkTopicMigrationClient#iterateTopics}} invokes {{visitPartition}} with a
placeholder record
[here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/core/src/main/scala/kafka/zk/migration/ZkTopicMigrationClient.scala#L91]
9. Ultimately, the event is processed with ZK state not having been updated
was:
Kafka quorum controller leader may fail to correctly update state in ZK when it
crashes in the following scenario
*Setup*
Consider a 3 controller quorum in dual-write mode, i.e. the quorum leader is
acting as the coordinator and owning writes to ZK via KRaftMigrationDriver.
Also consider the brokers to not be bounced in KRaft mode.
*Failure Scenario*
1. Let's say a client sends a request to increase the topic-partition count of
topic {{T1}} from {{2}} to {{3}}
2. After the request is processed by the quorum leader, the delta is propagated
to KRaftMigrationDriver via the callback {{{}onMetadataUpdate{}}}, which
in-turn appends a {{MetadataChangeEvent}} to the event queue. Ultimately, the
control flow ends up at {{{}KRaftMigrationZkWriter#handleDelta{}}}, followed by
{{{}KRaftMigrationZkWriter#handleTopicDelta{}}}. _Let's assume the quorum
leader crashes at this point before any updates to ZK._
3. Eventually, a new quorum leader is elected and it attempts to synchronise
the state with ZK while in {{SYNC_KRAFT_TO_ZK}} state. This results in
{{SyncKRaftMetadataEvent}} and eventually the invocation of
{{KRaftMigrationZkWriter#handleSnapshot}} followed by
{{KRaftMigrationZkWriter#handleTopicsSnapshot}}
4. Eventually,
[newPartitions|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L217]
and
[changedTopics|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L224]
are populated.
5. {{changedTopics}} results in the {{TopicZnode}} being updated with
information about the new replica
6. {{newPartitions}} should result in {{TopicPartitionStateZnode}} being
updated but the handler incorrectly calls
{{migrationClient.topicClient().updateTopicPartitions()}} instead of
{{migrationClient.topicClient().createTopicPartitions}}
[here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L259]
7. This results in a SetData on a ZNode which doesn't exist and ultimately a
{{MigrationClientException}} which results in the {{SyncKRaftMetadataEvent}} to
be retried.
8. This time around, {{newPartitions}} is not populated because
{{ZkTopicMigrationClient#iterateTopics}} invokes {{visitPartition}} with a
placeholder record
[here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/core/src/main/scala/kafka/zk/migration/ZkTopicMigrationClient.scala#L91]
9. Ultimately, the event is processed with ZK state not having been updated
> KRaftMigrationZkWriter may fail to create TopicPartitionStateZnode during a
> crash fault
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-21142
> URL: https://issues.apache.org/jira/browse/KAFKA-21142
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.9.2
> Reporter: Gaurav Narula
> Assignee: Gaurav Narula
> Priority: Major
>
> Kafka quorum controller leader may fail to correctly update state in ZK when
> it crashes in the following scenario
>
> *Setup*
> Consider a 3 controller quorum in dual-write mode, i.e. the quorum leader is
> acting as the coordinator and owning writes to ZK via KRaftMigrationDriver.
> Also consider the brokers to not be bounced in KRaft mode.
> *Failure Scenario*
>
> 1. Let's say a client sends a request to increase the topic-partition count
> of topic {{T1}} from {{2}} to {{3}}
> 2. After the request is processed by the quorum leader, the delta is
> propagated to KRaftMigrationDriver via the callback {{{}onMetadataUpdate{}}},
> which in-turn appends a {{MetadataChangeEvent}} to the event queue.
> Ultimately, the control flow ends up at
> {{{}KRaftMigrationZkWriter#handleDelta{}}}, followed by
> {{{}KRaftMigrationZkWriter#handleTopicsDelta{}}}. _Let's assume the quorum
> leader crashes at this point before any updates to ZK._
> 3. Eventually, a new quorum leader is elected and it attempts to synchronise
> the state with ZK while in {{SYNC_KRAFT_TO_ZK}} state. This results in
> {{SyncKRaftMetadataEvent}} and eventually the invocation of
> {{KRaftMigrationZkWriter#handleSnapshot}} followed by
> {{KRaftMigrationZkWriter#handleTopicsSnapshot}}
> 4. Eventually,
> [newPartitions|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L217]
> and
> [changedTopics|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L224]
> are populated.
> 5. {{changedTopics}} results in the {{TopicZnode}} being updated with
> information about the new replica
> 6. {{newPartitions}} should result in {{TopicPartitionStateZnode}} being
> updated but the handler incorrectly calls
> {{migrationClient.topicClient().updateTopicPartitions()}} instead of
> {{migrationClient.topicClient().createTopicPartitions}}
> [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L259]
> 7. This results in a SetData on a ZNode which doesn't exist and ultimately a
> {{MigrationClientException}} which results in the {{SyncKRaftMetadataEvent}}
> to be retried.
> 8. This time around, {{newPartitions}} is not populated because
> {{ZkTopicMigrationClient#iterateTopics}} invokes {{visitPartition}} with a
> placeholder record
> [here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/core/src/main/scala/kafka/zk/migration/ZkTopicMigrationClient.scala#L91]
> 9. Ultimately, the event is processed with ZK state not having been updated
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
