[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929057#comment-16929057
]
Christian Hagel commented on KAFKA-7500:
----------------------------------------
Dear [~ryannedolan] , thanks also from my side for your efforts pushing MM2.
I was trying out MM2 in order to mirror one cluster completely to a downstream
one in a distributed connector setup.
When running the MM2, I saw at the beginning lot's of
```WorkerSourceTask\{id=mirror-maker-49} Failed to flush, timed out while
waiting for producer to flush outstanding 5942 messages``` &
```WorkerSourceTask\{id=mirror-maker-49} Failed to commit offsets```
Error messages.
I would guess they are happening when the MM2 replicates faster, than it can
write on the connect.offsets.topic. This topic is configured with 25
partitions, which I thought should provide enough throughput to handle the load.
In order to test the robustness of the MM2, we also replaced the containers
several times during the replication and observed a surprising amount of
duplicated messages, which I guess happen because of the lagging offset commit.
Any ideas, how the setup can be improved so it becomes more robust.
> MirrorMaker 2.0 (KIP-382)
> -------------------------
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
> Affects Versions: 2.4.0
> Reporter: Ryanne Dolan
> Assignee: Manikumar
> Priority: Major
> Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
