Jarkko Jaakola created KAFKA-18125:
--------------------------------------

             Summary: Implement MirrorCheckpointConnector max task limit
                 Key: KAFKA-18125
                 URL: https://issues.apache.org/jira/browse/KAFKA-18125
             Project: Kafka
          Issue Type: Improvement
          Components: mirrormaker
            Reporter: Jarkko Jaakola


Mirrormaker 2 MirrorCheckpointConnector currently can create as many tasks as 
globally configured. Primary limiting factor for number of tasks up to the 
configured maximum is the number of known consumer groups. On larger mirrored 
environments where known consumers groups is large this leads to excessive 
amount of checkpoint tasks.

As the offset sync topic is single partition topic there will be excessive 
amount of tasks reading from and writing to this topic. The targeted Kafka 
broker can and will be under heavy burden.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to