[
https://issues.apache.org/jira/browse/KAFKA-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841137#comment-17841137
]
Edoardo Comar edited comment on KAFKA-16622 at 4/26/24 6:04 PM:
----------------------------------------------------------------
if after the consumer reaches 10000 and the 1st checkpoint is emitted, MM2
restarts before the other 10000 messages are produced,
then bug https://issues.apache.org/jira/browse/KAFKA-15905 hits and we end up
with just two checkpoints, at 10000 and 20000.
but the problem here is that if the consumer never fully catches up once, we
will never have a checkpoint.
If the {color:#000000}OffsetSyncStore.{color}{color:#871094}offsetSyncs
{color}contained a distribution of {color:#000000}OffsetSync rather than just
multiple copies of the last {color}{color:#000000}OffsetSync , Checkpoints
would be computed before
{color}
{color:#000000} {color}
was (Author: ecomar):
if after the consumer reaches 10000 and the 1st checkpoint is emitted, MM2
restarts before the other 10000 messages are produced,
then bug https://issues.apache.org/jira/browse/KAFKA-15905 hits and we end up
with just two checkpoints, at 10000 and 20000.
but the problem here is that if the consumer never fully cathces up once, we
will never have a checkpoint
> Mirromaker2 first Checkpoint not emitted until consumer group fully catches
> up once
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-16622
> URL: https://issues.apache.org/jira/browse/KAFKA-16622
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.7.0, 3.6.2, 3.8.0
> Reporter: Edoardo Comar
> Priority: Major
> Attachments: connect.log.2024-04-26-10.zip,
> edo-connect-mirror-maker-sourcetarget.properties
>
>
> We observed an excessively delayed emission of the MM2 Checkpoint record.
> It only gets created when the source consumer reaches the end of a topic.
> This does not seem reasonable.
> In a very simple setup :
> Tested with a standalone single process MirrorMaker2 mirroring between two
> single-node kafka clusters(mirromaker config attached) with quick refresh
> intervals (eg 5 sec) and a small offset.lag.max (eg 10)
> create a single topic in the source cluster
> produce data to it (e.g. 10000 records)
> start a slow consumer - e.g. fetching 50records/poll and pausing 1 sec
> between polls which commits after each poll
> watch the Checkpoint topic in the target cluster
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 \
> --topic source.checkpoints.internal \
> --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter \
> --from-beginning
> -> no record appears in the checkpoint topic until the consumer reaches the
> end of the topic (ie its consumer group lag gets down to 0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
