Hi all, I'm exploring using the MirrorSourceConnector and MirrorCheckpointConnector on Kafka Connect to setup active/active replication between two Kafka clusters. Using the DefaultReplicationPolicy replication policy class, messages originating from the source cluster get replicated as expected to the cluster-prefixed topic in the target cluster. Consumergroup offsets from the source to target cluster are replicated likewise. However, once the consumer group migrates from the source to the target cluster, its offsets are not replicated from the target back to the source cluster. For an active/active setup I'd want consumer group offsets for topic <source-cluster-alias>.<topic-name> in the target cluster to be replicated back to <topic-name> in the source cluster. This would allow consumers to failover & failback between clusters with minimal duplicate message consumption.
To clarify my setup a bit; I'm running two single-broker Kafka clusters in Docker (cluster A & B), along with a single Connect instance on which I've provisioned four source connectors: - A MirrorSourceConnector replicating topics from cluster A to cluster B - A MirrorSourceConnector replicating topics from cluster B to cluster A - A MirrorCheckpointConnector translating & replicating offsets from cluster A to cluster B - A MirrorCheckpointConnector translating & replicating offsets from cluster B to cluster A I'm not sure whether this is by design, or maybe I'm missing something. I've seen a similar question posted to KAFKA-9076 [1] without a resolution. Regards, Jeroen [1] https://issues.apache.org/jira/browse/KAFKA-9076?focusedCommentId=17268908&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17268908
