Robert Martin created KAFKA-10582: ------------------------------------- Summary: Mirror Maker 2 not replicating new topics until restart Key: KAFKA-10582 URL: https://issues.apache.org/jira/browse/KAFKA-10582 Project: Kafka Issue Type: Bug Components: mirrormaker Affects Versions: 2.5.1 Environment: RHEL 7 Linux. Reporter: Robert Martin
We are using Mirror Maker 2 from the 2.5.1 release for replication on some clusters. Replication is working as expected for existing topics. When we create a new topic, however, Mirror Maker 2 creates the replicated topic as expected but never starts replicating it. If we restart Mirror Maker 2 within 2-3 minutes the topic starts replicating as expected. From documentation we haveve seen it appears this should start replicating without a restart based on the settings we have. *Example:* Create topic "mytesttopic" on source cluster MirrorMaker 2 creates "source.mytesttopioc" on target cluster with no issue MirrorMaker 2 does not replicate "mytesttopic" -> "source.mytesttopic" Restart MirrorMaker 2 and now replication works for "mytesttopic" -> "source.mytesttopic" *Example config:* name = source->target group.id = source-to-target # specify any number of cluster aliases clusters = source, target # connection information for each cluster # This is a comma separated host:port pairs for each cluster # for e.g. "A_host1:9092, A_host2:9092, A_host3:9092" source.bootstrap.servers = sourcehosts:9092 target.bootstrap.servers = targethosts:9092 # enable and configure individual replication flows source->target.enabled = true # regex which defines which topics gets replicated. For eg "foo-.*" source->target.topics = .* target->source = false target->source.topics = .* # Setting replication factor of newly created remote topics replication.factor=3 ############################# Internal Topic Settings ############################# # The replication factor for mm2 internal topics "heartbeats", "B.checkpoints.internal" and # "mm2-offset-syncs.B.internal" # For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3. checkpoints.topic.replication.factor=3 heartbeats.topic.replication.factor=3 offset-syncs.topic.replication.factor=3 # The replication factor for connect internal topics "mm2-configs.B.internal", "mm2-offsets.B.internal" and # "mm2-status.B.internal" # For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3. offset.storage.replication.factor=3 status.storage.replication.factor=3 config.storage.replication.factor=3 # customize as needed # replication.policy.separator = _ # sync.topic.acls.enabled = false # emit.heartbeats.interval.seconds = 5 # tasks.max results in parallelism tasks.max = 16 refresh.topics.enabled = true sync.topic.configs.enabled = true # Setting the below too low can result in performance issues refresh.topics.interval.seconds = 300 refresh.groups.interval.seconds = 300 readahead.queue.capacity = 100 emit.checkpoints.enabled = true emit.checkpoints.interval.seconds = 5 -- This message was sent by Atlassian Jira (v8.3.4#803005)