[
https://issues.apache.org/jira/browse/KAFKA-16969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17855230#comment-17855230
]
Luke Chen commented on KAFKA-16969:
-----------------------------------
Yes, you're right. The reproduce steps is like:
# Run storage_tool.sh in v3.6 release to format a combined node containing
multiple log.dirs value
# Run bin/kafka-server-start.sh in v3.7.1 to start the combined KRaft node
# The node failed to startup with:
{code:java}
Caused by: java.lang.IllegalArgumentException: requirement failed: Multiple log
directories (aka JBOD) are not supported in the current MetadataVersion
3.6-IV2. Need 3.7-IV2 or higher{code}
> That's the exact situation that KAFKA-16606 was meant to catch, it is
> incorrect to configure a KRaft broker with multiple log directories pre
> 3.7-IV2, and it seems the system tests may be doing that.
Yes, we should catch this error since this is the wrong configuration. But this
"error behavior" has existed for many releases. I don't know if our "fix" will
cause surprise to users or not. Besides, like in isolated node, even if the
user set the wrong config, we only log the error without exit the startup
process. So after metadata version upgraded, everything works well. I think
that should be what we expected. WDYT?
> KRaft unable to upgrade to v3.7.1 and later when multiple log dir is set
> ------------------------------------------------------------------------
>
> Key: KAFKA-16969
> URL: https://issues.apache.org/jira/browse/KAFKA-16969
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.8.0, 3.7.1
> Reporter: Luke Chen
> Priority: Blocker
>
> After KAFKA-16606, we added validation metadata version for JBOD support.
> This validation works well in isolated KRaft mode (i.e. separate
> controller/broker node). But when in combined mode, this validation will let
> the node fail to startup. The log will be like this:
> {code:java}
> [2024-06-15 16:00:45,621] INFO [BrokerServer id=1] Waiting for the broker
> metadata publishers to be installed (kafka.server.BrokerServer)
> [2024-06-15 16:00:45,621] INFO [BrokerServer id=1] Finished waiting for the
> broker metadata publishers to be installed (kafka.server.BrokerServer)
> [2024-06-15 16:00:45,621] INFO [BrokerServer id=1] Waiting for the controller
> to acknowledge that we are caught up (kafka.server.BrokerServer)
> [2024-06-15 16:00:45,621] INFO [MetadataLoader id=1] InitializeNewPublishers:
> initializing MetadataVersionPublisher(id=1) with a snapshot at offset 4
> (org.apache.kafka.image.loader.MetadataLoader)
> [2024-06-15 16:00:45,621] ERROR Encountered metadata publishing fault: Broker
> configuration does not support the cluster MetadataVersion
> (org.apache.kafka.server.fault.LoggingFaultHandler)
> java.lang.IllegalArgumentException: requirement failed: Multiple log
> directories (aka JBOD) are not supported in the current MetadataVersion
> 3.6-IV2. Need 3.7-IV2 or higher
> at scala.Predef$.require(Predef.scala:337)
> at
> kafka.server.KafkaConfig.validateWithMetadataVersion(KafkaConfig.scala:2545)
> at
> kafka.server.MetadataVersionConfigValidator.onMetadataVersionChanged(MetadataVersionConfigValidator.java:62)
> at
> kafka.server.MetadataVersionConfigValidator.onMetadataUpdate(MetadataVersionConfigValidator.java:55)
> at
> org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:309)
> at
> org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> at java.base/java.lang.Thread.run(Thread.java:1623)
> [2024-06-15 16:00:45,622] ERROR Encountered fatal fault: Unhandled error
> initializing MetadataVersionPublisher(id=1) with a snapshot at offset 4
> (org.apache.kafka.server.fault.ProcessTerminatingFaultHandler)
> org.apache.kafka.server.fault.FaultHandlerException: Broker configuration
> does not support the cluster MetadataVersion
> at scala.Predef$.require(Predef.scala:337)
> at
> kafka.server.KafkaConfig.validateWithMetadataVersion(KafkaConfig.scala:2545)
> at
> kafka.server.MetadataVersionConfigValidator.onMetadataVersionChanged(MetadataVersionConfigValidator.java:62)
> at
> kafka.server.MetadataVersionConfigValidator.onMetadataUpdate(MetadataVersionConfigValidator.java:55)
> at
> org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:309)
> at
> org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> at java.base/java.lang.Thread.run(Thread.java:1623)
> Caused by: java.lang.IllegalArgumentException: requirement failed: Multiple
> log directories (aka JBOD) are not supported in the current MetadataVersion
> 3.6-IV2. Need 3.7-IV2 or higher
> ... 10 more{code}
>
> This will block combined node setting multiple log dirs upgrade to v3.7.1 or
> later.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
