Gaurav Narula created KAFKA-21142:
-------------------------------------
Summary: KRaftMigrationZkWriter may fail to create
TopicPartitionStateZnode during a crash fault
Key: KAFKA-21142
URL: https://issues.apache.org/jira/browse/KAFKA-21142
Project: Kafka
Issue Type: Bug
Affects Versions: 3.9.2
Reporter: Gaurav Narula
Assignee: Gaurav Narula
Kafka quorum controller leader may fail to correctly update state in ZK when it
crashes in the following scenario
*Setup*
Consider a 3 controller quorum in dual-write mode, i.e. the quorum leader is
acting as the coordinator and owning writes to ZK via KRaftMigrationDriver.
Also consider the brokers to not be bounced in KRaft mode.
*Failure Scenario*
1. Let's say a client sends a request to increase the topic-partition count of
topic {{T1}} from {{2}} to {{3}}
2. After the request is processed by the quorum leader, the delta is propagated
to KRaftMigrationDriver via the callback {{{}onMetadataUpdate{}}}, which
in-turn appends a {{MetadataChangeEvent}} to the event queue. Ultimately, the
control flow ends up at {{{}KRaftMigrationZkWriter#handleDelta{}}}, followed by
{{{}KRaftMigrationZkWriter#handleTopicDelta{}}}. _Let's assume the quorum
leader crashes after this operation._
3. Eventually, a new quorum leader is elected and it attempts to synchronise
the state with ZK while in {{SYNC_KRAFT_TO_ZK}} state. This results in
{{SyncKRaftMetadataEvent}} and eventually the invocation of
{{KRaftMigrationZkWriter#handleSnapshot}} followed by
{{KRaftMigrationZkWriter#handleTopicsSnapshot}}
4. Eventually,
[newPartitions|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L217]
and
[changedTopics|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L224]
are populated.
5. {{changedTopics}} results in the {{TopicZnode}} being updated with
information about the new replica
6. {{newPartitions}} should result in {{TopicPartitionStateZnode}} being
updated but the handler incorrectly calls
{{migrationClient.topicClient().updateTopicPartitions()}} instead of
{{migrationClient.topicClient().createTopicPartitions}}
[here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/metadata/src/main/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriter.java#L259]
7. This results in a SetData on a ZNode which doesn't exist and ultimately a
{{MigrationClientException}} which results in the {{SyncKRaftMetadataEvent}} to
be retried.
8. This time around, {{newPartitions}} is not populated because
{{ZkTopicMigrationClient#iterateTopics}} invokes {{visitPartition}} with a
placeholder record
[here|https://github.com/apache/kafka/blob/0a58d469fc7ebea5962d67a9718043b3a2bd60d1/core/src/main/scala/kafka/zk/migration/ZkTopicMigrationClient.scala#L91]
9. Ultimately, the event is processed with ZK state not having been updated
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
