twobeeb opened a new pull request #9589: URL: https://github.com/apache/kafka/pull/9589
As described in JIRA KAFKA-10710, Running mirror maker 2 with multiple clusters defined in the clusters properties creates a herder for all the possible combinations even if the "link" are not wanted. If my use-case, most of topics must be replicated from replica_CENTRAL to replica_<other>. Example config file ```` clusters = replica_CENTRAL, replica_OLS, replica_HBG # and lots more # cluster connection informations replica_CENTRAL.bootstrap.servers = ... replica_OLS.bootstrap.servers = ... replica_HBG.bootstrap.servers = ... ... replica_CENTRAL->replica_OLS.enabled = true replica_CENTRAL->replica_OLS.topics = _schemas replica_CENTRAL->replica_HBG.enabled = true replica_CENTRAL->replica_HBG.topics = _schemas ... ```` This leads to an incredible amount of connections, file descriptors and configuration topics created in every target clusters that are not required for this use-case. The number of opened file descriptors is more than 10 000 for only one topic replicated from CENTRAL cluster to 10 other clusters The following commit is a raw suggestion of what should be done to limit the number of herders created on startup. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
