[
https://issues.apache.org/jira/browse/KAFKA-12937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mickael Maison resolved KAFKA-12937.
------------------------------------
Resolution: Duplicate
> Mirrormaker2 can only start from the beginning of a topic
> ----------------------------------------------------------
>
> Key: KAFKA-12937
> URL: https://issues.apache.org/jira/browse/KAFKA-12937
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.8.0
> Environment: Dockerized environment
> Reporter: Daan Bosch
> Priority: Major
>
> *Goal*:
> I want to replace Mirrormaker version 1 with Mirrormaker2.
> To do this I want to:
> start Mirrormaker2 from the latest offset of every topic
> stop Mirrormaker1
> There should only be a couple of double messages.
> What happened:
> Mirrormaker2 starts replicating from the start of all topics
> *How to reproduce:*
> Start two Kafka clusters, A and B
> I produce 3000 messages to cluster A on a topic (TOPIC1)
> Kafka Connect is running and connected to cluster B
> Start a Mirrormaker2 task in connect to replicate messages from cluster A.
> Wit the option:
> consumer auto.offset.reset to latest
> Produce another 3000 messages to cluster A on the same topic (TOPIC1)
> *Expected result:*
> Cluster B will only contain the messages produced the second time (3000 in
> total) on TOPIC1
> Actual result:
> The mirror picks up all messages from the start (6000 in total) and
> replicates them to cluster B
> *Additional logs:*
> Logs from the consumer of the Mirrormaker task:
> mirrormaker.log:7581:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO [Consumer
> clientId=consumer-null-4, groupId=null] Seeking to offset 0 for partition
> perf-test-8 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
> mirrormaker.log:7583:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO
> [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for
> partition perf-test-3 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
> mirrormaker.log:7585:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO
> [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for
> partition perf-test-2 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
> mirrormaker.log:7587:mirrormaker_1 | [2021-06-11 09:31:40,403] INFO
> [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for
> partition perf-test-1 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
> mirrormaker.log:7589:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO
> [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for
> partition perf-test-0 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
> mirrormaker.log:7591:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO
> [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for
> partition perf-test-7 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
> mirrormaker.log:7593:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO
> [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for
> partition perf-test-6 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
> mirrormaker.log:7595:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO
> [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for
> partition perf-test-5 (org.apache.kafka.clients.consumer.KafkaConsumer:1582)
> mirrormaker.log:7597:mirrormaker_1 | [2021-06-11 09:31:40,404] INFO
> [Consumer clientId=consumer-null-4, groupId=null] Seeking to offset 0 for
> partition perf-test-4
> (org.apache.kafka.clients.consumer.KafkaConsumer:1582)You can see they are
> trying to seek to a position and thus overriding the latest offset
>
> You can see it is doing a seek to position 0 for every partition. which is
> not what I expected
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
