[
https://issues.apache.org/jira/browse/KAFKA-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841072#comment-17841072
]
Greg Harris commented on KAFKA-16622:
-------------------------------------
[~ecomar] Do you know if the MirrorCheckpointTask started/restarted after the
MirrorSourceTask replicated the data? This sounds like the known issue
https://issues.apache.org/jira/browse/KAFKA-15905 .
Or can you post some TRACE logging from OffsetSyncStore, to show how it decided
to translate using 19999?
I would expect that the offsets are translated at some coarse steps between 0
and 10000. With a 10 second translation latency(5+5), you would get translation
about every 500 records of consumer progress. 10000 records is enough to see
the 512, 1024, 2048, 4096, and 8192 offset translation steps, so there should
be ~5 checkpoints if it is working optimally.
> Mirromaker2 first Checkpoint not emitted until consumer group fully catches
> up once
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-16622
> URL: https://issues.apache.org/jira/browse/KAFKA-16622
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.7.0, 3.6.2, 3.8.0
> Reporter: Edoardo Comar
> Priority: Major
> Attachments: edo-connect-mirror-maker-sourcetarget.properties
>
>
> We observed an excessively delayed emission of the MM2 Checkpoint record.
> It only gets created when the source consumer reaches the end of a topic.
> This does not seem reasonable.
> In a very simple setup :
> Tested with a standalone single process MirrorMaker2 mirroring between two
> single-node kafka clusters(mirromaker config attached) with quick refresh
> intervals (eg 5 sec) and a small offset.lag.max (eg 10)
> create a single topic in the source cluster
> produce data to it (e.g. 10000 records)
> start a slow consumer - e.g. fetching 50records/poll and pausing 1 sec
> between polls which commits after each poll
> watch the Checkpoint topic in the target cluster
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 \
> --topic source.checkpoints.internal \
> --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter \
> --from-beginning
> -> no record appears in the checkpoint topic until the consumer reaches the
> end of the topic (ie its consumer group lag gets down to 0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
