[jira] [Created] (KAFKA-16892) TopicsDelta#localChanges get empty deletes when load from snapshot
HanXu created KAFKA-16892: - Summary: TopicsDelta#localChanges get empty deletes when load from snapshot Key: KAFKA-16892 URL: https://issues.apache.org/jira/browse/KAFKA-16892 Project: Kafka Issue Type: Bug Components: kraft Affects Versions: 3.7.0, 3.6.0, 3.5.0, 3.4.0 Reporter: HanXu Assignee: HanXu MetadataLoader#loadSnapshot get empty TopicsDelta.localChanges(brokerId).deletes https://github.com/apache/kafka/blob/02c794dfd3bda12066031155bd62234ad863860b/metadata/src/main/java/org/apache/kafka/image/TopicsDelta.java#L71C1-L76C6 Bug: 1. 当从 Snapshot 中生成 MetadataDelta 时,TopicsDelta 会首先 #replay(TopicRecord); 2. 然而 TopicsDelta 直接创建了一个无 Partitions 的 TopicDelta,而不是使用之前的 TopicImage; 3. 因为 TopicDelta.image 没有任何 Partitions,导致 TopicDelta#localChanges 生成 deletes 列表时,返回了 empty deletes; https://github.com/apache/kafka/blob/02c794dfd3bda12066031155bd62234ad863860b/metadata/src/main/java/org/apache/kafka/image/TopicsDelta.java#L71C1-L76C6 https://github.com/apache/kafka/blob/02c794dfd3bda12066031155bd62234ad863860b/metadata/src/main/java/org/apache/kafka/image/TopicDelta.java#L142-L148 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-16583) Update from 3.4.0 to 3.7.0 image write failed in Kraft mode
HanXu created KAFKA-16583: - Summary: Update from 3.4.0 to 3.7.0 image write failed in Kraft mode Key: KAFKA-16583 URL: https://issues.apache.org/jira/browse/KAFKA-16583 Project: Kafka Issue Type: Bug Components: kraft Affects Versions: 3.7.0 Reporter: HanXu Assignee: HanXu How to reproduce: 1. Launch a 3.4.0 controller and a 3.4.0 broker(BrokerA) in Kraft mode; 2. Create a topic with 1 partition; 3. Launch a 3.4.0 broker(Broker B) in Kraft mode and reassign the step 2 partition to Broker B; 4. Upgrade Broker B to 3.7.0; The Broker B will keep log the following error: {code:java} [2024-04-18 14:46:54,144] ERROR Encountered metadata loading fault: Unhandled error initializing new publishers (org.apache.kafka.server.fault.LoggingFaultHandler) org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been lost because the following could not be represented in metadata version 3.4-IV0: the directory assignment state of one or more replicas at org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94) at org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391) at org.apache.kafka.image.TopicImage.write(TopicImage.java:71) at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84) at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155) at org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295) at org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) at java.base/java.lang.Thread.run(Thread.java:840) {code} Bug: - When reassigning partition, PartitionRegistration#merge will set the new replicas with UNASSIGNED directory; - But in metadata version 3.4.0 PartitionRegistration#toRecord only allows MIGRATING directory; {code:java} if (options.metadataVersion().isDirectoryAssignmentSupported()) { record.setDirectories(Uuid.toList(directories)); } else { for (Uuid directory : directories) { if (!DirectoryId.MIGRATING.equals(directory)) { options.handleLoss("the directory assignment state of one or more replicas"); break; } } } {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)