Hi, We faced a similar problem when mirroring huge amounts of topics and partitions .
Kafka version 3.7.2 We saw an error of BoundedListTooLong <https://github.com/apache/kafka/blob/3.7.0/server-common/src/main/java/org/apache/kafka/server/mutable/BoundedListTooLongException.java> Excpetion in the kafka logs. According to kafka code <https://github.com/apache/kafka/blob/e3809682e2a0a312c6d5594fbc463b6cb0bb2273/metadata/src/main/java/org/apache/kafka/controller/QuorumController.java#L185C1-L186C1>, kraft allows only 10k operations per user in a single transaction. Though the catch here is, this only is a problem when MM2 initially creates these topics. There is no issue in mirroring more than 10k operations at once as MM2 threads are different. While creating the topic only 1 MM2 connection tries to create these topics instead of breaking them down into few different connections. As a simple resolution, we played around with the MM2 topic patterns to control the creation of topics in batches manually. on first deployment: topicsPattern = "[a-mA-M]" (This would only create topics from A-M and a-m) on second deployment: topicsPattern = ".*" (This would only create remaining topics. The previous topics would continue mirroring) You can split these into more chunks if a-m is also too large for you. Thanks On Tue, Apr 1, 2025 at 3:18 PM Alex Shapiro <bars...@yahoo.com.invalid> wrote: > > > > > > > Hi Team, > > > > We are trying to set MM2 syncbetween 2 clusters (Active\Passive) > > We have ~3000 topics > > Using CFK clusters 7.3.0 > > > > The problem > > MM2 tries to create in 1 batchtopics on target cluster (~3000) > > > > In the broker's logs, there is an error > > > > Could not create topicactive-north-2316. > > java.util.concurrent.CompletionException:org.apache.kafka.common.errors.PolicyViolationException: > Unable to performexcessively large batch operation. > > > > > > Possible Cause? > > The Kraft controller metadataimplementation has an 8kb limitation for more > detail: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions > . > > > > Suggested Resolution > > As a resolution, we got a suggestion to set a few MM2 instances. > > > > The question: > > Is there some way to limit the batch size by MM2 configuration only so > that topics are created in small batches? > > > > >
