Youssef Ismail created KAFKA-20989:
--------------------------------------
Summary: KRaft dual-write does not clean up ZooKeeper config and
ACL notification znodes
Key: KAFKA-20989
URL: https://issues.apache.org/jira/browse/KAFKA-20989
Project: Kafka
Issue Type: Bug
Components: kraft
Affects Versions: 3.9.1
Reporter: Youssef Ismail
During ZooKeeper-to-KRaft migration, the KRaft controller creates persistent
sequential notifications for [config
changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkConfigMigrationClient.scala#L352-L357]
and [ACL
changes|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/zk/migration/ZkAclMigrationClient.scala#L41-L44].
These nodes are normally deleted after 15 minutes by
[{{ZkNodeChangeNotificationListener}}|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/common/ZkNodeChangeNotificationListener.scala#L125-L140],
which is started by [ZooKeeper-mode
brokers|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/KafkaServer.scala#L631-L633].
Once all brokers enter KRaft mode in dual-write phase no cleanup listener
remains, while the [migration controller continues
dual-writing|https://github.com/apache/kafka/blob/3.9.1/core/src/main/scala/kafka/server/ControllerServer.scala#L284-L314].
Consequently, {{{}/config/changes{}}}, {{{}/kafka-acl-changes{}}}, and
{{/kafka-acl-extended-changes}} keep growing without deletion.
Normally if the migration happens quickly this shouldn't be an issue However
we want to let our clusters bake for some time in this phase before finalizing
the migration which causes the backlog.
This can increase ZooKeeper load and eventually make {{getChildren}} exceed
{{{}jute.maxbuffer{}}}, potentially impairing rollback.
I think the simplest solution is to have the migration controller run an
equivalent cleanup while migration mode is enabled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
