[
https://issues.apache.org/jira/browse/KAFKA-10710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien C updated KAFKA-10710:
-----------------------------
Description:
We are using MM2 distributed to synchronize topics from a "Central" broker down
to multiple "Local" brokers.
{quote}replica_CENTRAL->replica_OLS.enabled = true
replica_CENTRAL->replica_OLS.topics = _schemas
replica_CENTRAL->replica_OLS.replication.factor = 3
replica_CENTRAL->replica_HBG.enabled = true
replica_CENTRAL->replica_HBG.topics = _schemas
replica_CENTRAL->replica_HBG.replication.factor = 3
...
many more
...
replica_CENTRAL->replica_VIT.enabled = true
replica_CENTRAL->replica_VIT.topics = _schemas
replica_CENTRAL->replica_VIT.replication.factor = 3
replica_CENTRAL->replica_UGO.enabled = true
replica_CENTRAL->replica_UGO.topics = _schemas
replica_CENTRAL->replica_UGO.replication.factor = 3
{quote}
When looking into the Mirror Maker logs, we discover that a herder is created
for each combination even if we specifically don't describe a link between 2
clusters
Exemples:
{quote}{{[2020-11-12 08:43:30,351] INFO creating herder for
replica_VIT->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)}}
{{ [2020-11-12 08:43:33,697] INFO creating herder for replica_CNO->replica_UGO
(org.apache.kafka.connect.mirror.MirrorMaker)}}
{{ [2020-11-12 08:43:38,508] INFO creating herder for replica_UMO->replica_UGO
(org.apache.kafka.connect.mirror.MirrorMaker)}}
{quote}
... And many many more.
So much that we reached the limit of our user property LimitNOFILE recently
when trying to add a new "Local" cluster.
I believe this behavior leads to unecessary connections and resource usage that
could be easily avoided by just limiting the herder creation only to elements
specifically described in the mirrormaker.properties file
[https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java#L130-L136]
was:
We are using MM2 distributed to synchronize topics from a "Central" broker down
to multiple "Local" brokers.
{quote}# enable and configure individual replication flows
replica_CENTRAL->replica_OLS.enabled = true
replica_CENTRAL->replica_OLS.topics = _schemas
replica_CENTRAL->replica_OLS.replication.factor = 3
replica_CENTRAL->replica_HBG.enabled = true
replica_CENTRAL->replica_HBG.topics = _schemas
replica_CENTRAL->replica_HBG.replication.factor = 3
# many more
replica_CENTRAL->replica_VIT.enabled = true
replica_CENTRAL->replica_VIT.topics = _schemas
replica_CENTRAL->replica_VIT.replication.factor = 3
replica_CENTRAL->replica_UGO.enabled = true
replica_CENTRAL->replica_UGO.topics = _schemas
replica_CENTRAL->replica_UGO.replication.factor = 3
{quote}
When looking into the Mirror Maker logs, we discover that a herder is created
for each combination even if we specifically don't describe a link between 2
clusters
Exemples:
{quote}{{[2020-11-12 08:43:30,351] INFO creating herder for
replica_VIT->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)}}
{{ [2020-11-12 08:43:33,697] INFO creating herder for replica_CNO->replica_UGO
(org.apache.kafka.connect.mirror.MirrorMaker)}}
{{ [2020-11-12 08:43:38,508] INFO creating herder for replica_UMO->replica_UGO
(org.apache.kafka.connect.mirror.MirrorMaker)}}
{quote}
... And many many more.
So much that we reached the limit of our user property LimitNOFILE recently
when trying to add a new "Local" cluster.
I believe this behavior leads to unecessary connections and resource usage that
could be easily avoided by just limiting the herder creation only to elements
specifically described in the mirrormaker.properties file
[https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java#L130-L136]
> MirrorMaker 2 creates all combinations of herders
> -------------------------------------------------
>
> Key: KAFKA-10710
> URL: https://issues.apache.org/jira/browse/KAFKA-10710
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.5.1
> Reporter: Julien C
> Priority: Major
>
> We are using MM2 distributed to synchronize topics from a "Central" broker
> down to multiple "Local" brokers.
> {quote}replica_CENTRAL->replica_OLS.enabled = true
> replica_CENTRAL->replica_OLS.topics = _schemas
> replica_CENTRAL->replica_OLS.replication.factor = 3
> replica_CENTRAL->replica_HBG.enabled = true
> replica_CENTRAL->replica_HBG.topics = _schemas
> replica_CENTRAL->replica_HBG.replication.factor = 3
> ...
> many more
> ...
> replica_CENTRAL->replica_VIT.enabled = true
> replica_CENTRAL->replica_VIT.topics = _schemas
> replica_CENTRAL->replica_VIT.replication.factor = 3
> replica_CENTRAL->replica_UGO.enabled = true
> replica_CENTRAL->replica_UGO.topics = _schemas
> replica_CENTRAL->replica_UGO.replication.factor = 3
> {quote}
>
> When looking into the Mirror Maker logs, we discover that a herder is created
> for each combination even if we specifically don't describe a link between 2
> clusters
>
> Exemples:
> {quote}{{[2020-11-12 08:43:30,351] INFO creating herder for
> replica_VIT->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)}}
> {{ [2020-11-12 08:43:33,697] INFO creating herder for
> replica_CNO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)}}
> {{ [2020-11-12 08:43:38,508] INFO creating herder for
> replica_UMO->replica_UGO (org.apache.kafka.connect.mirror.MirrorMaker)}}
> {quote}
> ... And many many more.
> So much that we reached the limit of our user property LimitNOFILE recently
> when trying to add a new "Local" cluster.
> I believe this behavior leads to unecessary connections and resource usage
> that could be easily avoided by just limiting the herder creation only to
> elements specifically described in the mirrormaker.properties file
> [https://github.com/apache/kafka/blob/trunk/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorMaker.java#L130-L136]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
