Mohit, PR-6295 and KIP-382 introduce MirorrMaker 2.0 which was designed to support this operation.
In a nutshell, MM2 maintains a sparse offset sync stream while replicating records between clusters. The offset syncs are used to translate consumer offsets for periodic cross-cluster checkpoints. Leveraging these checkpoints, you can resume processing in a new cluster via kafka-consumer-groups.sh --reset-offsets, and you are guaranteed to never skip records. The RemoteClusterUtils.translateOffsets() method in the PR is a good place to start. It will return the latest translated offsets for a particular consumer group on a remote cluster. You can build tooling around this, e.g. to migrate all your consumers at once. https://github.com/apache/kafka/pull/6295 https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 Ryanne On Mon, Jun 24, 2019 at 10:34 AM Mohit Kumar <m...@imohit.me> wrote: > Is there anyway to copy consumers offset while moving data from one cluster > to another cluster? >
