Jarkko Jaakola created KAFKA-18125: -------------------------------------- Summary: Implement MirrorCheckpointConnector max task limit Key: KAFKA-18125 URL: https://issues.apache.org/jira/browse/KAFKA-18125 Project: Kafka Issue Type: Improvement Components: mirrormaker Reporter: Jarkko Jaakola
Mirrormaker 2 MirrorCheckpointConnector currently can create as many tasks as globally configured. Primary limiting factor for number of tasks up to the configured maximum is the number of known consumer groups. On larger mirrored environments where known consumers groups is large this leads to excessive amount of checkpoint tasks. As the offset sync topic is single partition topic there will be excessive amount of tasks reading from and writing to this topic. The targeted Kafka broker can and will be under heavy burden. -- This message was sent by Atlassian Jira (v8.20.10#820010)