David Arthur created KAFKA-16205: ------------------------------------ Summary: Reduce number of metadata requests during hybrid mode Key: KAFKA-16205 URL: https://issues.apache.org/jira/browse/KAFKA-16205 Project: Kafka Issue Type: Improvement Components: controller, kraft Affects Versions: 3.6.0, 3.5.0, 3.4.0, 3.7.0 Reporter: David Arthur
When migrating a cluster with a high number of brokers and partitions, it is possible for the controller channel manager queue to get backed up. This can happen when many small RPCs are generated in response to several small MetadataDeltas being handled MigrationPropagator. In the ZK controller, various optimizations have been made over the years to reduce the number of UMR and LISR sent during controlled shutdown or other large metadata events. For the ZK to KRaft migration, we use the MetadataLoader infrastructure to learn about and propagate metadata to ZK brokers. We need to improve the batching in MigrationPropagator to avoid performance issues during the migration of large clusters. -- This message was sent by Atlassian Jira (v8.20.10#820010)