josefk31 opened a new pull request, #18424: URL: https://github.com/apache/kafka/pull/18424
Fixes broken system tests. This has likely been broken for at least a year but may have only been detected now when [new tests](https://github.com/confluentinc/ce-kafka/pull/17847) where added. The test does the following: 1. starts kafka in older version of kafka 2. upgrades kafka to newest (stopping, starting) 3. downgrades to old version again (stopping, starting) All while sending off verifiable producer/consumer to make sure all messages are read. The test suite only fails for kafka-3.3.2 at the starting phase of step 3. According to logs, the test suite fails because of: ``` kafka.common.InconsistentBrokerMetadataException: BrokerMetadata is not consistent across log.dirs. This could happen if multiple brokers shared a log directory (log.dirs) or partial data was manually copied from another broker. Found: - /mnt/kafka/kafka-metadata-logs -> {node.id=1, directory.id=ItAoMTrsidYVfoRnX3gsAA, version=1, cluster.id=I2eXt9rvSnyhct8BYmW6-w} - /mnt/kafka/kafka-data-logs-2 -> {node.id=1, directory.id=MiQDnIX6WuYL0NdMaLOsRQ, version=1, cluster.id=I2eXt9rvSnyhct8BYmW6-w} - /mnt/kafka/kafka-data-logs-1 -> {node.id=1, directory.id=F1m5lsdOIsGtTpTYT0Ao9g, version=1, cluster.id=I2eXt9rvSnyhct8BYmW6-w} at kafka.server.BrokerMetadataCheckpoint$.getBrokerMetadataAndOfflineDirs(BrokerMetadataCheckpoint.scala:194) at kafka.server.KafkaRaftServer$.initializeLogDirs(KafkaRaftServer.scala:184) at kafka.server.KafkaRaftServer.<init>(KafkaRaftServer.scala:61) at kafka.Kafka$.buildServer(Kafka.scala:79) at kafka.Kafka$.main(Kafka.scala:87) at kafka.Kafka.main(Kafka.scala) ``` This is only broken in 3.3.2 version of Kafka [BrokerMetatadataCheckpoint.scala](https://github.com/apache/kafka/blob/c9c03dd7ef9ff4edf2596e905cabececc72a9e9d/core/src/main/scala/kafka/server/BrokerMetadataCheckpoint.scala#L186). In 3.3.2 some metadata information about the log directories from `metadata.properties` files and expect that the properties are duplicated for all log directories. We crash with a fatal error if they are non-duplicate which at that time would mean that another instance of kafka was using the same log directories. However, in [#14628](https://github.com/apache/kafka/pull/14628) we dropped the requirement that the `metadata.properties` files is duplicate in each directory since they will now contain a non-unique `directory.id` field for each dir. This has no effect on versions of kafka running `kraft` mode, greater than 3.4.x (they care only about uniqueness of node.id and cluster.id) but does affect 3.3.2 since it expects every `metadata.properties` file to be the same or else. If we assume that this has been broken for a while and there is not a forward compat requirement, I propose removing this specific test version. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
