[jira] [Updated] (KAFKA-15648) QuorumControllerTest#testBootstrapZkMigrationRecord is flaky

[David Arthur (Jira)]

     [ 
https://issues.apache.org/jira/browse/KAFKA-15648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Arthur updated KAFKA-15648:
---------------------------------
    Description: 
Noticed that this test failed on Jenkins with 

{code}
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: Should not have ZK migrations 
enabled on a cluster running metadata.version 3.0-IV1
        at 
app//org.apache.kafka.controller.ActivationRecordsGenerator.recordsForNonEmptyLog(ActivationRecordsGenerator.java:154)
        at 
app//org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:229)
        at 
app//org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
app//org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
app//org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
app//org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
app//org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base@11.0.16.1/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Should not have ZK migrations enabled on 
a cluster running metadata.version 3.0-IV1
        ... 8 more
{code}

I think this exception is a red herring (it's expected from one of the negative 
test cases). When trying to reproduce this issue locally, I do occasionally see 
the following type of failure

{code}
[2023-10-19 13:42:10,091] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:10,091] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=1)), nextEndOffset=0) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:10,091] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:10,091] DEBUG [LocalLogManager 0] initialized local log 
manager for node 0 (org.apache.kafka.metalog.LocalLogManager:685)
[2023-10-19 13:42:10,091] DEBUG [QuorumController id=0] Creating in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:203)
[2023-10-19 13:42:10,091] INFO [QuorumController id=0] Creating new 
QuorumController with clusterId 6xRUXZ_kQ1GfuaHK42AS9Q. ZK migration mode is 
enabled. (org.apache.kafka.controller.QuorumController:1912)
[2023-10-19 13:42:10,092] INFO [LocalLogManager 0] Node 0: registered 
MetaLogListener 1082153924 (org.apache.kafka.metalog.LocalLogManager:703)
[2023-10-19 13:42:10,092] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:10,092] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:10,092] INFO [QuorumController id=0] In the new epoch 1, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 50 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:10,092] INFO [QuorumController id=0] Becoming the active 
controller at epoch 1, next write offset 1. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 77 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:10,092] WARN [QuorumController id=0] Performing controller 
activation. The metadata log appears to be empty. Appending 1 bootstrap 
record(s) at metadata.version 3.4-IV0 from bootstrap source 'test'. Putting the 
controller into pre-migration mode. No metadata updates will be allowed until 
the ZK metadata has been migrated. 
(org.apache.kafka.controller.QuorumController:108)
[2023-10-19 13:42:10,092] INFO [QuorumController id=0] Replayed a 
FeatureLevelRecord setting metadata version to 3.4-IV0 
(org.apache.kafka.controller.FeatureControlManager:400)
[2023-10-19 13:42:10,092] INFO [QuorumController id=0] Replayed a 
ZkMigrationStateRecord changing the migration state from NONE to PRE_MIGRATION. 
(org.apache.kafka.controller.FeatureControlManager:421)
[2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[1](342832465). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.raft.errors.NotLeaderException: Append failed because the 
given epoch 0 is stale. Current leader epoch = 1
        at 
org.apache.kafka.metalog.LocalLogManager$SharedLogData.tryAppend(LocalLogManager.java:253)
        at 
org.apache.kafka.metalog.LocalLogManager$SharedLogData.tryAppend(LocalLogManager.java:233)
        at 
org.apache.kafka.metalog.LocalLogManager.scheduleAtomicAppend(LocalLogManager.java:795)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent$1.apply(QuorumController.java:839)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent$1.apply(QuorumController.java:816)
        at 
org.apache.kafka.controller.QuorumController.appendRecords(QuorumController.java:918)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:815)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:10,092] INFO [QuorumController id=0] completeActivation[1]: 
event failed with NotLeaderException (treated as NotControllerException) at 
epoch 1 in 387 microseconds. Renouncing leadership and reverting to the last 
committed offset -1. (org.apache.kafka.controller.QuorumController:542)
[2023-10-19 13:42:10,092] ERROR Encountered fatalFaultHandler fault: exception 
while renouncing leadership (org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.IllegalArgumentException: Attempt to resign from epoch 1 which is 
larger than the current epoch 0
        at 
org.apache.kafka.metalog.LocalLogManager.resign(LocalLogManager.java:808)
        at 
org.apache.kafka.controller.QuorumController.renounce(QuorumController.java:1268)
        at 
org.apache.kafka.controller.QuorumController.handleEventException(QuorumController.java:545)
        at 
org.apache.kafka.controller.QuorumController.access$800(QuorumController.java:177)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:879)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:869)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:10,093] DEBUG [QuorumController id=0] Processed read 
migration state(176059074) in 4 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:10,093] INFO [QuorumController id=0] 
QuorumController#beginShutdown: shutting down event queue. 
(org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:10,093] DEBUG [QuorumController id=0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:10,093] INFO [QuorumController id=0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)
[2023-10-19 13:42:10,093] INFO [LocalLogManager 0] beginShutdown: shutting down 
event queue. (org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:10,093] DEBUG [LocalLogManager 0] Node 0: closing. 
(org.apache.kafka.metalog.LocalLogManager:653)
[2023-10-19 13:42:10,094] DEBUG [LocalLogManager 0] beginShutdown: Event queue 
is already shutting down. (org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:10,094] DEBUG [LocalLogManager 0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:10,094] DEBUG [LocalLogManager 0] Node 0: beginning shutdown. 
(org.apache.kafka.metalog.LocalLogManager:637)
[2023-10-19 13:42:10,094] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=2)), nextEndOffset=1) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:10,094] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=3). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:10,094] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=3)), nextEndOffset=2) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:10,094] DEBUG [QuorumController id=0] 
MetaLogManager.Listener: Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:10,094] INFO [LocalLogManager 0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)

org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while renouncing leadership: Attempt to resign from epoch 1 which is 
larger than the current epoch 0

        at 
org.apache.kafka.metalog.LocalLogManager.resign(LocalLogManager.java:808)
        at 
org.apache.kafka.controller.QuorumController.renounce(QuorumController.java:1268)
        at 
org.apache.kafka.controller.QuorumController.handleEventException(QuorumController.java:545)
        at 
org.apache.kafka.controller.QuorumController.access$800(QuorumController.java:177)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:879)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:869)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalArgumentException: Attempt to resign from epoch 1 
which is larger than the current epoch 0
        ... 11 more
{code}

  was:
Noticed that this test failed on Jenkins with 

{code}
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: Should not have ZK migrations 
enabled on a cluster running metadata.version 3.0-IV1
        at 
app//org.apache.kafka.controller.ActivationRecordsGenerator.recordsForNonEmptyLog(ActivationRecordsGenerator.java:154)
        at 
app//org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:229)
        at 
app//org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
app//org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
app//org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
app//org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
app//org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base@11.0.16.1/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Should not have ZK migrations enabled on 
a cluster running metadata.version 3.0-IV1
        ... 8 more
{code}

When trying to reproduce this failure locally, I ran into a separate flaky 
failure

{code}
[2023-10-19 13:42:09,442] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,442] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=1)), nextEndOffset=0) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,442] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,442] DEBUG [LocalLogManager 0] initialized local log 
manager for node 0 (org.apache.kafka.metalog.LocalLogManager:685)
[2023-10-19 13:42:09,442] DEBUG [QuorumController id=0] Creating in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:203)
[2023-10-19 13:42:09,442] INFO [QuorumController id=0] Creating new 
QuorumController with clusterId K8TDRiYZQuepVQHPgwP91A. ZK migration mode is 
enabled. (org.apache.kafka.controller.QuorumController:1912)
[2023-10-19 13:42:09,442] INFO [LocalLogManager 0] Node 0: registered 
MetaLogListener 1238203422 (org.apache.kafka.metalog.LocalLogManager:703)
[2023-10-19 13:42:09,443] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,443] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,443] INFO [QuorumController id=0] In the new epoch 1, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 25 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,443] INFO [QuorumController id=0] Becoming the active 
controller at epoch 1, next write offset 1. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 34 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,443] WARN [QuorumController id=0] Performing controller 
activation. The metadata log appears to be empty. Appending 1 bootstrap 
record(s) at metadata.version 3.4-IV0 from bootstrap source 'test'. Putting the 
controller into pre-migration mode. No metadata updates will be allowed until 
the ZK metadata has been migrated. 
(org.apache.kafka.controller.QuorumController:108)
[2023-10-19 13:42:09,443] INFO [QuorumController id=0] Replayed a 
FeatureLevelRecord setting metadata version to 3.4-IV0 
(org.apache.kafka.controller.FeatureControlManager:400)
[2023-10-19 13:42:09,443] INFO [QuorumController id=0] Replayed a 
ZkMigrationStateRecord changing the migration state from NONE to PRE_MIGRATION. 
(org.apache.kafka.controller.FeatureControlManager:421)
[2023-10-19 13:42:09,443] DEBUG append(batch=LocalRecordBatch(leaderEpoch=1, 
appendTimestamp=10, 
records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', 
featureLevel=8) at version 0), 
ApiMessageAndVersion(ZkMigrationStateRecord(zkMigrationState=2) at version 
0)]), nextEndOffset=2) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Creating in-memory 
snapshot 2 (org.apache.kafka.timeline.SnapshotRegistry:203)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Read-write operation 
completeActivation[1](1689476270) will be completed when the log reaches offset 
2. (org.apache.kafka.controller.QuorumController:849)
[2023-10-19 13:42:09,443] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Executing 
handleCommit[baseOffset=1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Completing purgatory 
items up to offset 2 and epoch 1. 
(org.apache.kafka.controller.QuorumController:1065)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Deleting in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:247)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] completeUpTo(2): 
successfully completing completeActivation[1](1689476270) 
(org.apache.kafka.deferred.DeferredEventQueue:61)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Processed 
completeActivation[1](1689476270) in 462 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,443] DEBUG [QuorumController id=0] Processed 
handleCommit[baseOffset=1] in 209 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,447] DEBUG [QuorumController id=0] Processed read 
migration state(565300828) in 9 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,447] INFO [QuorumController id=0] 
QuorumController#beginShutdown: shutting down event queue. 
(org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:09,447] DEBUG [QuorumController id=0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,448] INFO [QuorumController id=0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)
[2023-10-19 13:42:09,448] INFO [LocalLogManager 0] beginShutdown: shutting down 
event queue. (org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:09,448] DEBUG [LocalLogManager 0] Node 0: closing. 
(org.apache.kafka.metalog.LocalLogManager:653)
[2023-10-19 13:42:09,448] DEBUG [LocalLogManager 0] Node 0: beginning shutdown. 
(org.apache.kafka.metalog.LocalLogManager:637)
[2023-10-19 13:42:09,448] DEBUG [LocalLogManager 0] beginShutdown: Event queue 
is already shutting down. (org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,448] DEBUG [LocalLogManager 0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,448] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=2)), nextEndOffset=3) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,448] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=3). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,448] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=3)), nextEndOffset=4) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,448] DEBUG [QuorumController id=0] 
MetaLogManager.Listener: Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,448] INFO [LocalLogManager 0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)
[2023-10-19 13:42:09,449] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,450] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=1)), nextEndOffset=0) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,450] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,450] DEBUG [LocalLogManager 0] initialized local log 
manager for node 0 (org.apache.kafka.metalog.LocalLogManager:685)
[2023-10-19 13:42:09,450] DEBUG [QuorumController id=0] Creating in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:203)
[2023-10-19 13:42:09,450] INFO [QuorumController id=0] Creating new 
QuorumController with clusterId jhXUIK37QHCACVQD4CArkQ. 
(org.apache.kafka.controller.QuorumController:1912)
[2023-10-19 13:42:09,450] INFO [LocalLogManager 0] Node 0: registered 
MetaLogListener 521943667 (org.apache.kafka.metalog.LocalLogManager:703)
[2023-10-19 13:42:09,450] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,450] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,450] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,450] INFO [QuorumController id=0] In the new epoch 1, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,450] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 24 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,450] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,450] INFO [QuorumController id=0] Becoming the active 
controller at epoch 1, next write offset 1. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,450] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 34 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,451] WARN [QuorumController id=0] Performing controller 
activation. The metadata log appears to be empty. Appending 1 bootstrap 
record(s) at metadata.version 3.4-IV0 from bootstrap source 'test'. Setting the 
ZK migration state to NONE since this is a de-novo KRaft cluster. 
(org.apache.kafka.controller.QuorumController:108)
[2023-10-19 13:42:09,451] INFO [QuorumController id=0] Replayed a 
FeatureLevelRecord setting metadata version to 3.4-IV0 
(org.apache.kafka.controller.FeatureControlManager:400)
[2023-10-19 13:42:09,451] DEBUG [QuorumController id=0] Replayed a 
ZkMigrationStateRecord which did not alter the state from NONE. 
(org.apache.kafka.controller.FeatureControlManager:417)
[2023-10-19 13:42:09,451] DEBUG append(batch=LocalRecordBatch(leaderEpoch=1, 
appendTimestamp=10, 
records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', 
featureLevel=8) at version 0), 
ApiMessageAndVersion(ZkMigrationStateRecord(zkMigrationState=0) at version 
0)]), nextEndOffset=2) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,451] DEBUG [QuorumController id=0] Creating in-memory 
snapshot 2 (org.apache.kafka.timeline.SnapshotRegistry:203)
[2023-10-19 13:42:09,451] DEBUG [QuorumController id=0] Read-write operation 
completeActivation[1](211991183) will be completed when the log reaches offset 
2. (org.apache.kafka.controller.QuorumController:849)
[2023-10-19 13:42:09,451] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,451] DEBUG [QuorumController id=0] Executing 
handleCommit[baseOffset=1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,451] DEBUG [QuorumController id=0] Completing purgatory 
items up to offset 2 and epoch 1. 
(org.apache.kafka.controller.QuorumController:1065)
[2023-10-19 13:42:09,451] DEBUG [QuorumController id=0] Deleting in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:247)
[2023-10-19 13:42:09,451] DEBUG [QuorumController id=0] completeUpTo(2): 
successfully completing completeActivation[1](211991183) 
(org.apache.kafka.deferred.DeferredEventQueue:61)
[2023-10-19 13:42:09,451] DEBUG [QuorumController id=0] Processed 
completeActivation[1](211991183) in 233 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,451] DEBUG [QuorumController id=0] Processed 
handleCommit[baseOffset=1] in 64 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,454] DEBUG [QuorumController id=0] Processed read 
migration state(144519107) in 15 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,455] INFO [QuorumController id=0] 
QuorumController#beginShutdown: shutting down event queue. 
(org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:09,455] DEBUG [QuorumController id=0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,455] INFO [QuorumController id=0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)
[2023-10-19 13:42:09,455] INFO [LocalLogManager 0] beginShutdown: shutting down 
event queue. (org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:09,455] DEBUG [LocalLogManager 0] Node 0: closing. 
(org.apache.kafka.metalog.LocalLogManager:653)
[2023-10-19 13:42:09,455] DEBUG [LocalLogManager 0] beginShutdown: Event queue 
is already shutting down. (org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,455] DEBUG [LocalLogManager 0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,455] DEBUG [LocalLogManager 0] Node 0: beginning shutdown. 
(org.apache.kafka.metalog.LocalLogManager:637)
[2023-10-19 13:42:09,456] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=2)), nextEndOffset=3) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,456] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=3). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,456] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=3)), nextEndOffset=4) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,456] DEBUG [QuorumController id=0] 
MetaLogManager.Listener: Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,456] INFO [LocalLogManager 0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)
[2023-10-19 13:42:09,458] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,458] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=1)), nextEndOffset=0) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,458] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,458] DEBUG [LocalLogManager 0] initialized local log 
manager for node 0 (org.apache.kafka.metalog.LocalLogManager:685)
[2023-10-19 13:42:09,458] DEBUG [QuorumController id=0] Creating in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:203)
[2023-10-19 13:42:09,458] INFO [QuorumController id=0] Creating new 
QuorumController with clusterId ACZHODtYRUWy_02iltPCfQ. 
(org.apache.kafka.controller.QuorumController:1912)
[2023-10-19 13:42:09,458] INFO [LocalLogManager 0] Node 0: registered 
MetaLogListener 1558249098 (org.apache.kafka.metalog.LocalLogManager:703)
[2023-10-19 13:42:09,458] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,458] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,458] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,458] INFO [QuorumController id=0] In the new epoch 1, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,458] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 65 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,458] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,458] INFO [QuorumController id=0] Becoming the active 
controller at epoch 1, next write offset 1. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,458] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 93 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,458] WARN [QuorumController id=0] Performing controller 
activation. The metadata log appears to be empty. Appending 1 bootstrap 
record(s) at metadata.version 3.3-IV0 from bootstrap source 'test'. 
(org.apache.kafka.controller.QuorumController:108)
[2023-10-19 13:42:09,458] INFO [QuorumController id=0] Replayed a 
FeatureLevelRecord setting metadata version to 3.3-IV0 
(org.apache.kafka.controller.FeatureControlManager:400)
[2023-10-19 13:42:09,459] DEBUG append(batch=LocalRecordBatch(leaderEpoch=1, 
appendTimestamp=10, 
records=[ApiMessageAndVersion(FeatureLevelRecord(name='metadata.version', 
featureLevel=4) at version 0)]), nextEndOffset=1) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,459] DEBUG [QuorumController id=0] Creating in-memory 
snapshot 1 (org.apache.kafka.timeline.SnapshotRegistry:203)
[2023-10-19 13:42:09,459] DEBUG [QuorumController id=0] Read-write operation 
completeActivation[1](482477678) will be completed when the log reaches offset 
1. (org.apache.kafka.controller.QuorumController:849)
[2023-10-19 13:42:09,459] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,459] DEBUG [QuorumController id=0] Executing 
handleCommit[baseOffset=1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,459] DEBUG [QuorumController id=0] Completing purgatory 
items up to offset 1 and epoch 1. 
(org.apache.kafka.controller.QuorumController:1065)
[2023-10-19 13:42:09,459] DEBUG [QuorumController id=0] Deleting in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:247)
[2023-10-19 13:42:09,459] DEBUG [QuorumController id=0] completeUpTo(1): 
successfully completing completeActivation[1](482477678) 
(org.apache.kafka.deferred.DeferredEventQueue:61)
[2023-10-19 13:42:09,459] DEBUG [QuorumController id=0] Processed 
completeActivation[1](482477678) in 681 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,459] DEBUG [QuorumController id=0] Processed 
handleCommit[baseOffset=1] in 243 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,468] DEBUG [QuorumController id=0] Processed read 
migration state(419270958) in 22 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,472] INFO [QuorumController id=0] 
QuorumController#beginShutdown: shutting down event queue. 
(org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:09,472] DEBUG [QuorumController id=0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,475] INFO [QuorumController id=0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)
[2023-10-19 13:42:09,475] INFO [LocalLogManager 0] beginShutdown: shutting down 
event queue. (org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:09,476] DEBUG [LocalLogManager 0] Node 0: closing. 
(org.apache.kafka.metalog.LocalLogManager:653)
[2023-10-19 13:42:09,476] DEBUG [LocalLogManager 0] beginShutdown: Event queue 
is already shutting down. (org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,476] DEBUG [LocalLogManager 0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,486] DEBUG [LocalLogManager 0] Node 0: beginning shutdown. 
(org.apache.kafka.metalog.LocalLogManager:637)
[2023-10-19 13:42:09,486] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=2)), nextEndOffset=2) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,486] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=3). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,486] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=3)), nextEndOffset=3) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,486] DEBUG [QuorumController id=0] 
MetaLogManager.Listener: Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,491] INFO [LocalLogManager 0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)
[2023-10-19 13:42:09,494] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,494] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=1)), nextEndOffset=0) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,495] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,495] DEBUG [LocalLogManager 0] initialized local log 
manager for node 0 (org.apache.kafka.metalog.LocalLogManager:685)
[2023-10-19 13:42:09,505] DEBUG [QuorumController id=0] Creating in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:203)
[2023-10-19 13:42:09,505] INFO [QuorumController id=0] Creating new 
QuorumController with clusterId P32sDt5AQj-foG2jofGzjQ. ZK migration mode is 
enabled. (org.apache.kafka.controller.QuorumController:1912)
[2023-10-19 13:42:09,507] INFO [LocalLogManager 0] Node 0: registered 
MetaLogListener 1935252719 (org.apache.kafka.metalog.LocalLogManager:703)
[2023-10-19 13:42:09,507] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,507] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,507] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,507] INFO [QuorumController id=0] In the new epoch 1, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,507] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 25 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,507] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,507] INFO [QuorumController id=0] Becoming the active 
controller at epoch 1, next write offset 1. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,507] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[1] in 37 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,507] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,507] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[1](330313376). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,507] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[1]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 1 in 105 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,507] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=2)), nextEndOffset=1) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,507] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=3). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,507] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=3)), nextEndOffset=2) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,507] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,507] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=2) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=3) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,507] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=3) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,507] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,507] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,508] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[3]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,508] INFO [QuorumController id=0] In the new epoch 3, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,508] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[3] in 29 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,508] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[3]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,508] INFO [QuorumController id=0] Becoming the active 
controller at epoch 3, next write offset 3. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,508] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[3] in 118 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,508] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,508] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[3](1908568455). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,508] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[3]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 3 in 127 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,508] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=4)), nextEndOffset=3) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,508] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=5). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,508] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=5)), nextEndOffset=4) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,508] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,511] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,512] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=4) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=5) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,512] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=5) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,512] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[5]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,512] INFO [QuorumController id=0] In the new epoch 5, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[5] in 23 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[5]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,512] INFO [QuorumController id=0] Becoming the active 
controller at epoch 5, next write offset 5. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[5] in 43 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,512] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[5](154181525). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,512] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[5]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 5 in 127 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,512] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=6)), nextEndOffset=5) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,512] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=7). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,512] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=7)), nextEndOffset=6) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,512] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,512] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=6) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=7) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,512] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=7) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,512] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[7]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,512] INFO [QuorumController id=0] In the new epoch 7, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[7] in 32 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[7]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,512] INFO [QuorumController id=0] Becoming the active 
controller at epoch 7, next write offset 7. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[7] in 25 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,512] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,512] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[7](1351535182). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,512] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[7]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 7 in 76 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,512] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=8)), nextEndOffset=7) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,513] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=9). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,513] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,513] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=9)), nextEndOffset=8) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,513] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,513] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=8) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=9) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,513] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=9) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,513] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,513] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[9]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,513] INFO [QuorumController id=0] In the new epoch 9, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,513] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[9] in 72 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,513] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[9]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,513] INFO [QuorumController id=0] Becoming the active 
controller at epoch 9, next write offset 9. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,513] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[9] in 109 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,513] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,513] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[9](305401902). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,513] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[9]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 9 in 263 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,514] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=10)), nextEndOffset=9) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,514] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=11). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,514] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=11)), nextEndOffset=10) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,514] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,520] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,520] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=10) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=11) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,520] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=11) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,520] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,521] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[11]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,522] INFO [QuorumController id=0] In the new epoch 11, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,522] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[11] in 385 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,522] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[11]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,522] INFO [QuorumController id=0] Becoming the active 
controller at epoch 11, next write offset 11. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,522] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[11] in 115 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,522] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,522] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[11](278291950). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,522] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[11]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 11 in 274 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,522] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=12)), nextEndOffset=11) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,523] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=13). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,523] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=13)), nextEndOffset=12) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,523] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,527] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,527] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=12) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=13) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,527] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=13) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,527] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,527] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[13]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,527] INFO [QuorumController id=0] In the new epoch 13, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,527] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[13] in 412 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,527] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[13]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,527] INFO [QuorumController id=0] Becoming the active 
controller at epoch 13, next write offset 13. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,527] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[13] in 59 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,527] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,527] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[13](303556794). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,527] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[13]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 13 in 113 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,528] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=14)), nextEndOffset=13) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,528] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=15). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,528] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=15)), nextEndOffset=14) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,528] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,535] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,536] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=14) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=15) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,536] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=15) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,536] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,537] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[15]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,537] INFO [QuorumController id=0] In the new epoch 15, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,537] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[15] in 80 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,537] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[15]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,537] INFO [QuorumController id=0] Becoming the active 
controller at epoch 15, next write offset 15. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,537] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[15] in 47 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,537] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,538] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[15](1729737065). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,538] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[15]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 15 in 121 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,538] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=16)), nextEndOffset=15) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,538] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=17). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,538] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=17)), nextEndOffset=16) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,538] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,540] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,540] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=16) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=17) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,540] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=17) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,540] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,542] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[17]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,543] INFO [QuorumController id=0] In the new epoch 17, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,543] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[17] in 488 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,543] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[17]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,543] INFO [QuorumController id=0] Becoming the active 
controller at epoch 17, next write offset 17. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,543] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[17] in 112 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,543] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,543] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[17](1805130027). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,543] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[17]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 17 in 328 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,543] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=18)), nextEndOffset=17) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,544] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=19). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,544] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=19)), nextEndOffset=18) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,544] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,545] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,546] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=18) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=19) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,546] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=19) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,546] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,548] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[19]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,548] INFO [QuorumController id=0] In the new epoch 19, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,548] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[19] in 114 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,548] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[19]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,548] INFO [QuorumController id=0] Becoming the active 
controller at epoch 19, next write offset 19. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,548] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[19] in 98 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,548] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,548] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[19](492295970). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,548] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[19]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 19 in 280 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,548] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=20)), nextEndOffset=19) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,549] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=21). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,549] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=21)), nextEndOffset=20) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,549] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,551] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,551] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=20) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=21) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,551] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=21) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,551] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,557] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[21]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,557] INFO [QuorumController id=0] In the new epoch 21, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,557] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[21] in 175 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,557] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[21]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,557] INFO [QuorumController id=0] Becoming the active 
controller at epoch 21, next write offset 21. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,557] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[21] in 112 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,557] ERROR Encountered fatalFaultHandler fault: exception 
while completing controller activation 
(org.apache.kafka.server.fault.MockFaultHandler:44)
java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 does not 
support ZK migrations. Cannot continue with ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
[2023-10-19 13:42:09,558] DEBUG [QuorumController id=0] Got exception while 
running completeActivation[21](1265411926). Invoking handleException. 
(org.apache.kafka.queue.KafkaEventQueue:132)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,559] ERROR Encountered nonFatalFaultHandler fault: 
completeActivation[21]: event failed with FaultHandlerException (treated as 
UnknownServerException) at epoch 21 in 1233 microseconds. Renouncing leadership 
and reverting to the last committed offset -1. 
(org.apache.kafka.server.fault.MockFaultHandler:44)
org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while completing controller activation: The bootstrap 
metadata.version 3.3-IV0 does not support ZK migrations. Cannot continue with 
ZK migrations enabled.
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.recordsForEmptyLog(ActivationRecordsGenerator.java:103)
        at 
org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:226)
        at 
org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
        at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
        at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: The bootstrap metadata.version 3.3-IV0 
does not support ZK migrations. Cannot continue with ZK migrations enabled.
        ... 8 more
[2023-10-19 13:42:09,559] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=22)), nextEndOffset=21) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,559] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=23). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,559] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=23)), nextEndOffset=22) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,559] DEBUG [QuorumController id=0] Reverting to in-memory 
snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:213)
[2023-10-19 13:42:09,559] DEBUG [QuorumController id=0] Processed read 
migration state(18563510) in 6 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,560] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,560] DEBUG [LocalLogManager 0] Node 0: Ignoring 
LeaderAndEpoch(leaderId=OptionalInt.empty, epoch=22) since it doesn't match the 
latest known leader LeaderAndEpoch(leaderId=OptionalInt[0], epoch=23) 
(org.apache.kafka.metalog.LocalLogManager:585)
[2023-10-19 13:42:09,560] DEBUG [LocalLogManager 0] Node 0: Executing 
handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=23) 
(org.apache.kafka.metalog.LocalLogManager:578)
[2023-10-19 13:42:09,560] DEBUG [LocalLogManager 0] Node 0: running log check. 
(org.apache.kafka.metalog.LocalLogManager:536)
[2023-10-19 13:42:09,562] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[23]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,562] INFO [QuorumController id=0] In the new epoch 23, the 
leader is (none). (org.apache.kafka.controller.QuorumController:1179)
[2023-10-19 13:42:09,562] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[23] in 74 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,562] INFO [QuorumController id=0] 
QuorumController#beginShutdown: shutting down event queue. 
(org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:09,568] DEBUG [QuorumController id=0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,571] DEBUG [QuorumController id=0] Executing 
handleLeaderChange[23]. (org.apache.kafka.controller.QuorumController:577)
[2023-10-19 13:42:09,571] INFO [QuorumController id=0] Becoming the active 
controller at epoch 23, next write offset 23. 
(org.apache.kafka.controller.QuorumController:1175)
[2023-10-19 13:42:09,571] INFO [QuorumController id=0] completeActivation[23]: 
event unable to start processing because of RejectedExecutionException (treated 
as TimeoutException). (org.apache.kafka.controller.QuorumController:542)
[2023-10-19 13:42:09,571] DEBUG [QuorumController id=0] Processed 
handleLeaderChange[23] in 144 us 
(org.apache.kafka.controller.QuorumController:510)
[2023-10-19 13:42:09,574] INFO [QuorumController id=0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)
[2023-10-19 13:42:09,574] INFO [LocalLogManager 0] beginShutdown: shutting down 
event queue. (org.apache.kafka.queue.KafkaEventQueue:492)
[2023-10-19 13:42:09,576] DEBUG [LocalLogManager 0] Node 0: closing. 
(org.apache.kafka.metalog.LocalLogManager:653)
[2023-10-19 13:42:09,576] DEBUG [LocalLogManager 0] beginShutdown: Event queue 
is already shutting down. (org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,576] DEBUG [LocalLogManager 0] KafkaEventQueue#close: 
Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,576] DEBUG [LocalLogManager 0] Node 0: beginning shutdown. 
(org.apache.kafka.metalog.LocalLogManager:637)
[2023-10-19 13:42:09,577] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
 epoch=24)), nextEndOffset=23) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,577] INFO Elected new leader: 
LeaderAndEpoch(leaderId=OptionalInt[0], epoch=25). 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
[2023-10-19 13:42:09,577] DEBUG 
append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
 epoch=25)), nextEndOffset=24) 
(org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
[2023-10-19 13:42:09,577] DEBUG [QuorumController id=0] 
MetaLogManager.Listener: Event queue is already shutting down. 
(org.apache.kafka.queue.KafkaEventQueue:489)
[2023-10-19 13:42:09,578] INFO [LocalLogManager 0] closed event queue. 
(org.apache.kafka.queue.KafkaEventQueue:514)
{code}


> QuorumControllerTest#testBootstrapZkMigrationRecord is flaky
> ------------------------------------------------------------
>
>                 Key: KAFKA-15648
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15648
>             Project: Kafka
>          Issue Type: Bug
>          Components: controller, unit tests
>            Reporter: David Arthur
>            Priority: Minor
>              Labels: flaky-test, good-first-issue
>
> Noticed that this test failed on Jenkins with 
> {code}
> org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
> exception while completing controller activation: Should not have ZK 
> migrations enabled on a cluster running metadata.version 3.0-IV1
>       at 
> app//org.apache.kafka.controller.ActivationRecordsGenerator.recordsForNonEmptyLog(ActivationRecordsGenerator.java:154)
>       at 
> app//org.apache.kafka.controller.ActivationRecordsGenerator.generate(ActivationRecordsGenerator.java:229)
>       at 
> app//org.apache.kafka.controller.QuorumController$CompleteActivationEvent.generateRecordsAndResult(QuorumController.java:1237)
>       at 
> app//org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:784)
>       at 
> app//org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
>       at 
> app//org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>       at 
> app//org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>       at java.base@11.0.16.1/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.RuntimeException: Should not have ZK migrations enabled 
> on a cluster running metadata.version 3.0-IV1
>       ... 8 more
> {code}
> I think this exception is a red herring (it's expected from one of the 
> negative test cases). When trying to reproduce this issue locally, I do 
> occasionally see the following type of failure
> {code}
> [2023-10-19 13:42:10,091] INFO Elected new leader: 
> LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1). 
> (org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
> [2023-10-19 13:42:10,091] DEBUG 
> append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
>  epoch=1)), nextEndOffset=0) 
> (org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
> [2023-10-19 13:42:10,091] DEBUG [LocalLogManager 0] Node 0: running log 
> check. (org.apache.kafka.metalog.LocalLogManager:536)
> [2023-10-19 13:42:10,091] DEBUG [LocalLogManager 0] initialized local log 
> manager for node 0 (org.apache.kafka.metalog.LocalLogManager:685)
> [2023-10-19 13:42:10,091] DEBUG [QuorumController id=0] Creating in-memory 
> snapshot -1 (org.apache.kafka.timeline.SnapshotRegistry:203)
> [2023-10-19 13:42:10,091] INFO [QuorumController id=0] Creating new 
> QuorumController with clusterId 6xRUXZ_kQ1GfuaHK42AS9Q. ZK migration mode is 
> enabled. (org.apache.kafka.controller.QuorumController:1912)
> [2023-10-19 13:42:10,092] INFO [LocalLogManager 0] Node 0: registered 
> MetaLogListener 1082153924 (org.apache.kafka.metalog.LocalLogManager:703)
> [2023-10-19 13:42:10,092] DEBUG [LocalLogManager 0] Node 0: running log 
> check. (org.apache.kafka.metalog.LocalLogManager:536)
> [2023-10-19 13:42:10,092] DEBUG [LocalLogManager 0] Node 0: Executing 
> handleLeaderChange LeaderAndEpoch(leaderId=OptionalInt[0], epoch=1) 
> (org.apache.kafka.metalog.LocalLogManager:578)
> [2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Executing 
> handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
> [2023-10-19 13:42:10,092] INFO [QuorumController id=0] In the new epoch 1, 
> the leader is (none). (org.apache.kafka.controller.QuorumController:1179)
> [2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Processed 
> handleLeaderChange[1] in 50 us 
> (org.apache.kafka.controller.QuorumController:510)
> [2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Executing 
> handleLeaderChange[1]. (org.apache.kafka.controller.QuorumController:577)
> [2023-10-19 13:42:10,092] INFO [QuorumController id=0] Becoming the active 
> controller at epoch 1, next write offset 1. 
> (org.apache.kafka.controller.QuorumController:1175)
> [2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Processed 
> handleLeaderChange[1] in 77 us 
> (org.apache.kafka.controller.QuorumController:510)
> [2023-10-19 13:42:10,092] WARN [QuorumController id=0] Performing controller 
> activation. The metadata log appears to be empty. Appending 1 bootstrap 
> record(s) at metadata.version 3.4-IV0 from bootstrap source 'test'. Putting 
> the controller into pre-migration mode. No metadata updates will be allowed 
> until the ZK metadata has been migrated. 
> (org.apache.kafka.controller.QuorumController:108)
> [2023-10-19 13:42:10,092] INFO [QuorumController id=0] Replayed a 
> FeatureLevelRecord setting metadata version to 3.4-IV0 
> (org.apache.kafka.controller.FeatureControlManager:400)
> [2023-10-19 13:42:10,092] INFO [QuorumController id=0] Replayed a 
> ZkMigrationStateRecord changing the migration state from NONE to 
> PRE_MIGRATION. (org.apache.kafka.controller.FeatureControlManager:421)
> [2023-10-19 13:42:10,092] DEBUG [QuorumController id=0] Got exception while 
> running completeActivation[1](342832465). Invoking handleException. 
> (org.apache.kafka.queue.KafkaEventQueue:132)
> org.apache.kafka.raft.errors.NotLeaderException: Append failed because the 
> given epoch 0 is stale. Current leader epoch = 1
>       at 
> org.apache.kafka.metalog.LocalLogManager$SharedLogData.tryAppend(LocalLogManager.java:253)
>       at 
> org.apache.kafka.metalog.LocalLogManager$SharedLogData.tryAppend(LocalLogManager.java:233)
>       at 
> org.apache.kafka.metalog.LocalLogManager.scheduleAtomicAppend(LocalLogManager.java:795)
>       at 
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent$1.apply(QuorumController.java:839)
>       at 
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent$1.apply(QuorumController.java:816)
>       at 
> org.apache.kafka.controller.QuorumController.appendRecords(QuorumController.java:918)
>       at 
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:815)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>       at java.base/java.lang.Thread.run(Thread.java:833)
> [2023-10-19 13:42:10,092] INFO [QuorumController id=0] completeActivation[1]: 
> event failed with NotLeaderException (treated as NotControllerException) at 
> epoch 1 in 387 microseconds. Renouncing leadership and reverting to the last 
> committed offset -1. (org.apache.kafka.controller.QuorumController:542)
> [2023-10-19 13:42:10,092] ERROR Encountered fatalFaultHandler fault: 
> exception while renouncing leadership 
> (org.apache.kafka.server.fault.MockFaultHandler:44)
> java.lang.IllegalArgumentException: Attempt to resign from epoch 1 which is 
> larger than the current epoch 0
>       at 
> org.apache.kafka.metalog.LocalLogManager.resign(LocalLogManager.java:808)
>       at 
> org.apache.kafka.controller.QuorumController.renounce(QuorumController.java:1268)
>       at 
> org.apache.kafka.controller.QuorumController.handleEventException(QuorumController.java:545)
>       at 
> org.apache.kafka.controller.QuorumController.access$800(QuorumController.java:177)
>       at 
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:879)
>       at 
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:869)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>       at java.base/java.lang.Thread.run(Thread.java:833)
> [2023-10-19 13:42:10,093] DEBUG [QuorumController id=0] Processed read 
> migration state(176059074) in 4 us 
> (org.apache.kafka.controller.QuorumController:510)
> [2023-10-19 13:42:10,093] INFO [QuorumController id=0] 
> QuorumController#beginShutdown: shutting down event queue. 
> (org.apache.kafka.queue.KafkaEventQueue:492)
> [2023-10-19 13:42:10,093] DEBUG [QuorumController id=0] 
> KafkaEventQueue#close: Event queue is already shutting down. 
> (org.apache.kafka.queue.KafkaEventQueue:489)
> [2023-10-19 13:42:10,093] INFO [QuorumController id=0] closed event queue. 
> (org.apache.kafka.queue.KafkaEventQueue:514)
> [2023-10-19 13:42:10,093] INFO [LocalLogManager 0] beginShutdown: shutting 
> down event queue. (org.apache.kafka.queue.KafkaEventQueue:492)
> [2023-10-19 13:42:10,093] DEBUG [LocalLogManager 0] Node 0: closing. 
> (org.apache.kafka.metalog.LocalLogManager:653)
> [2023-10-19 13:42:10,094] DEBUG [LocalLogManager 0] beginShutdown: Event 
> queue is already shutting down. (org.apache.kafka.queue.KafkaEventQueue:489)
> [2023-10-19 13:42:10,094] DEBUG [LocalLogManager 0] KafkaEventQueue#close: 
> Event queue is already shutting down. 
> (org.apache.kafka.queue.KafkaEventQueue:489)
> [2023-10-19 13:42:10,094] DEBUG [LocalLogManager 0] Node 0: beginning 
> shutdown. (org.apache.kafka.metalog.LocalLogManager:637)
> [2023-10-19 13:42:10,094] DEBUG 
> append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt.empty,
>  epoch=2)), nextEndOffset=1) 
> (org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
> [2023-10-19 13:42:10,094] INFO Elected new leader: 
> LeaderAndEpoch(leaderId=OptionalInt[0], epoch=3). 
> (org.apache.kafka.metalog.LocalLogManager$SharedLogData:300)
> [2023-10-19 13:42:10,094] DEBUG 
> append(batch=LeaderChangeBatch(newLeader=LeaderAndEpoch(leaderId=OptionalInt[0],
>  epoch=3)), nextEndOffset=2) 
> (org.apache.kafka.metalog.LocalLogManager$SharedLogData:276)
> [2023-10-19 13:42:10,094] DEBUG [QuorumController id=0] 
> MetaLogManager.Listener: Event queue is already shutting down. 
> (org.apache.kafka.queue.KafkaEventQueue:489)
> [2023-10-19 13:42:10,094] INFO [LocalLogManager 0] closed event queue. 
> (org.apache.kafka.queue.KafkaEventQueue:514)
> org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
> exception while renouncing leadership: Attempt to resign from epoch 1 which 
> is larger than the current epoch 0
>       at 
> org.apache.kafka.metalog.LocalLogManager.resign(LocalLogManager.java:808)
>       at 
> org.apache.kafka.controller.QuorumController.renounce(QuorumController.java:1268)
>       at 
> org.apache.kafka.controller.QuorumController.handleEventException(QuorumController.java:545)
>       at 
> org.apache.kafka.controller.QuorumController.access$800(QuorumController.java:177)
>       at 
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:879)
>       at 
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:869)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:148)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:137)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
>       at 
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
>       at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: java.lang.IllegalArgumentException: Attempt to resign from epoch 1 
> which is larger than the current epoch 0
>       ... 11 more
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)