[ https://issues.apache.org/jira/browse/KAFKA-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848424#comment-17848424 ]
Jianbin Chen edited comment on KAFKA-16662 at 5/22/24 3:10 AM: --------------------------------------------------------------- I have encountered the same issue. Can anyone help me with this? I upgraded from 3.5.1 to 3.7.0, and I have already changed inter.broker.protocol.version to 3.7 and ran it for some time. But I have never executed {code:java} ./bin/kafka-features.sh upgrade --metadata 3.7 {code} The last time I restarted the cluster, I found that it could not be started anymore. The last line of the log is as follows: {code:java} [2024-05-22 11:01:41,087] INFO [MetadataLoader id=3] maybePublishMetadata(LOG_DELTA): The loader is still catching up because we have loaded up to offset 41872530, but the high water mark is 41872532 (org.apache.kafka.image.loader.MetadataLoader) [2024-05-22 11:01:41,088] INFO [MetadataLoader id=3] maybePublishMetadata(LOG_DELTA): The loader finished catching up to the current high water mark of 41872532 (org.apache.kafka.image.loader.MetadataLoader) [2024-05-22 11:01:41,092] INFO [BrokerLifecycleManager id=3] The broker has caught up. Transitioning from STARTING to RECOVERY. (kafka.server.BrokerLifecycleManager) [2024-05-22 11:01:41,092] INFO [BrokerServer id=3] Finished waiting for the controller to acknowledge that we are caught up (kafka.server.BrokerServer) [2024-05-22 11:01:41,092] INFO [BrokerServer id=3] Waiting for the initial broker metadata update to be published (kafka.server.BrokerServer) [2024-05-22 11:01:41,095] ERROR Encountered fatal fault: Unhandled error initializing new publishers (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler) org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been lost because the following could not be represented in metadata version 3.5-IV2: the directory assignment state of one or more replicas at org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94) at org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391) at org.apache.kafka.image.TopicImage.write(TopicImage.java:71) at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84) at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155) at org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295) at org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) at java.base/java.lang.Thread.run(Thread.java:1583) {code} was (Author: jianbin): I have encountered the same issue. Can anyone help me with this? I upgraded from 3.5.1 to 3.7.0, and I have already changed inter.broker.protocol.version to 3.7 and ran it for some time. But I have never executed `./bin/kafka-features.sh upgrade --metadata 3.7` The last time I restarted the cluster, I found that it could not be started anymore. The last line of the log is as follows: ``` [2024-05-22 11:01:41,087] INFO [MetadataLoader id=3] maybePublishMetadata(LOG_DELTA): The loader is still catching up because we have loaded up to offset 41872530, but the high water mark is 41872532 (org.apache.kafka.image.loader.MetadataLoader) [2024-05-22 11:01:41,088] INFO [MetadataLoader id=3] maybePublishMetadata(LOG_DELTA): The loader finished catching up to the current high water mark of 41872532 (org.apache.kafka.image.loader.MetadataLoader) [2024-05-22 11:01:41,092] INFO [BrokerLifecycleManager id=3] The broker has caught up. Transitioning from STARTING to RECOVERY. (kafka.server.BrokerLifecycleManager) [2024-05-22 11:01:41,092] INFO [BrokerServer id=3] Finished waiting for the controller to acknowledge that we are caught up (kafka.server.BrokerServer) [2024-05-22 11:01:41,092] INFO [BrokerServer id=3] Waiting for the initial broker metadata update to be published (kafka.server.BrokerServer) [2024-05-22 11:01:41,095] ERROR Encountered fatal fault: Unhandled error initializing new publishers (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler) org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has been lost because the following could not be represented in metadata version 3.5-IV2: the directory assignment state of one or more replicas at org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94) at org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391) at org.apache.kafka.image.TopicImage.write(TopicImage.java:71) at org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84) at org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155) at org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295) at org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) at java.base/java.lang.Thread.run(Thread.java:1583) ``` > UnwritableMetadataException: Metadata has been lost > --------------------------------------------------- > > Key: KAFKA-16662 > URL: https://issues.apache.org/jira/browse/KAFKA-16662 > Project: Kafka > Issue Type: Bug > Affects Versions: 3.7.0 > Environment: Docker Image (bitnami/kafka:3.7.0) > via Docker Compose > Reporter: Tobias Bohn > Priority: Major > Attachments: log.txt > > > Hello, > First of all: I am new to this Jira and apologize if anything is set or > specified incorrectly. Feel free to advise me. > We currently have an error in our test system, which unfortunately I can't > solve, because I couldn't find anything related to it. No solution could be > found via the mailing list either. > The error occurs when we want to start up a node. The node runs using Kraft > and is both a controller and a broker. The following error message appears at > startup: > {code:java} > kafka | [2024-04-16 06:18:13,707] ERROR Encountered fatal fault: Unhandled > error initializing new publishers > (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler) > kafka | org.apache.kafka.image.writer.UnwritableMetadataException: Metadata > has been lost because the following could not be represented in metadata > version 3.5-IV2: the directory assignment state of one or more replicas > kafka | at > org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94) > kafka | at > org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391) > kafka | at org.apache.kafka.image.TopicImage.write(TopicImage.java:71) > kafka | at > org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84) > kafka | at > org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155) > kafka | at > org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295) > kafka | at > org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266) > kafka | at > org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127) > kafka | at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210) > kafka | at > org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181) > kafka | at java.base/java.lang.Thread.run(Thread.java:840) > kafka exited with code 0 {code} > We use Docker to operate the cluster. The error occurred while we were trying > to restart a node. All other nodes in the cluster are still running correctly. > If you need further information, please let us know. The complete log is > attached to this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)