[ https://issues.apache.org/jira/browse/KAFKA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mickael Maison resolved KAFKA-12937. ------------------------------------ Resolution: Duplicate > Mirrormaker2 can only start from the beginning of a topic > ---------------------------------------------------------- > > Key: KAFKA-12937 > URL: https://issues.apache.org/jira/browse/KAFKA-12937 > Project: Kafka > Issue Type: Bug > Components: mirrormaker > Affects Versions: 2.8.0 > Environment: Dockerized environment > Reporter: Daan Bosch > Priority: Major > > *Goal*: > I want to replace Mirrormaker version 1 with Mirrormaker2. > To do this I want to: > start Mirrormaker2 from the latest offset of every topic > stop Mirrormaker1 > There should only be a couple of double messages. > What happened: > Mirrormaker2 starts replicating from the start of all topics > *How to reproduce:* > Start two Kafka clusters, A and B > I produce 3000 messages to cluster A on a topic (TOPIC1) > Kafka Connect is running and connected to cluster B > Start a Mirrormaker2 task in connect to replicate messages from cluster A. > Wit the option: > consumer auto.offset.reset to latest > Produce another 3000 messages to cluster A on the same topic (TOPIC1) > *Expected result:* > Cluster B will only contain the messages produced the second time (3000 in > total) on TOPIC1 > Actual result: > The mirror picks up all messages from the start (6000 in total) and > replicates them to cluster B > *Additional logs:* > Logs from the consumer of the Mirrormaker task: > mirrormaker.log:7581:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer > clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition > perf-test-8 (org.apache.kafka.clients.consumer.KafkaConsumer:1582) > mirrormaker.log:7583:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO > [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for > partition perf-test-3 (org.apache.kafka.clients.consumer.KafkaConsumer:1582) > mirrormaker.log:7585:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO > [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for > partition perf-test-2 (org.apache.kafka.clients.consumer.KafkaConsumer:1582) > mirrormaker.log:7587:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO > [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for > partition perf-test-1 (org.apache.kafka.clients.consumer.KafkaConsumer:1582) > mirrormaker.log:7589:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO > [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for > partition perf-test-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1582) > mirrormaker.log:7591:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO > [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for > partition perf-test-7 (org.apache.kafka.clients.consumer.KafkaConsumer:1582) > mirrormaker.log:7593:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO > [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for > partition perf-test-6 (org.apache.kafka.clients.consumer.KafkaConsumer:1582) > mirrormaker.log:7595:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO > [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for > partition perf-test-5 (org.apache.kafka.clients.consumer.KafkaConsumer:1582) > mirrormaker.log:7597:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO > [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for > partition perf-test-4 > (org.apache.kafka.clients.consumer.KafkaConsumer:1582)You can see they are > trying to seek to a position and thus overriding the latest offset > > You can see it is doing a seek to position 0 for every partition. which is > not what I expected -- This message was sent by Atlassian Jira (v8.20.10#820010)