Thanks, Vruttant, for sharing.
it means we need to have a few MM2 deployments, onlybecause the first step is
failing due to the batch limit.
It would be more nice if MM2 will have some approach forbatch size on this phase
On Tuesday, April 1, 2025 at 01:13:57 PM GMT+3, Vruttant Mankad
<vaman...@gmail.com> wrote:
Hi,
We faced a similar problem when mirroring huge amounts of topics and
partitions .
Kafka version 3.7.2
We saw an error of BoundedListTooLong
<https://github.com/apache/kafka/blob/3.7.0/server-common/src/main/java/org/apache/kafka/server/mutable/BoundedListTooLongException.java>
Excpetion in the kafka logs. According to kafka code
<https://github.com/apache/kafka/blob/e3809682e2a0a312c6d5594fbc463b6cb0bb2273/metadata/src/main/java/org/apache/kafka/controller/QuorumController.java#L185C1-L186C1>,
kraft allows only 10k operations per user in a single transaction.
Though the catch here is, this only is a problem when MM2 initially creates
these topics. There is no issue in mirroring more than 10k operations at
once as MM2 threads are different.
While creating the topic only 1 MM2 connection tries to create these topics
instead of breaking them down into few different connections.
As a simple resolution, we played around with the MM2 topic patterns to
control the creation of topics in batches manually.
on first deployment:
topicsPattern = "[a-mA-M]" (This would only create topics from A-M and a-m)
on second deployment:
topicsPattern = ".*" (This would only create remaining topics. The previous
topics would continue mirroring)
You can split these into more chunks if a-m is also too large for you.
Thanks
On Tue, Apr 1, 2025 at 3:18 PM Alex Shapiro <bars...@yahoo.com.invalid>
wrote:
>
>
>
>
>
>
> Hi Team,
>
>
>
> We are trying to set MM2 syncbetween 2 clusters (Active\Passive)
>
> We have ~3000 topics
>
> Using CFK clusters 7.3.0
>
>
>
> The problem
>
> MM2 tries to create in 1 batchtopics on target cluster (~3000)
>
>
>
> In the broker's logs, there is an error
>
>
>
> Could not create topicactive-north-2316.
>
> java.util.concurrent.CompletionException:org.apache.kafka.common.errors.PolicyViolationException:
> Unable to performexcessively large batch operation.
>
>
>
>
>
> Possible Cause?
>
> The Kraft controller metadataimplementation has an 8kb limitation for more
> detail:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions
> .
>
>
>
> Suggested Resolution
>
> As a resolution, we got a suggestion to set a few MM2 instances.
>
>
>
> The question:
>
> Is there some way to limit the batch size by MM2 configuration only so
> that topics are created in small batches?
>
>
>
>
>
