[
https://issues.apache.org/jira/browse/KAFKA-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17489633#comment-17489633
]
Mickael Maison commented on KAFKA-13659:
----------------------------------------
Is this a DUP of https://issues.apache.org/jira/browse/KAFKA-12635 ?
Overall I agree with the proposed approach to not commit any offsets for a
partition until some records have been mirrored.
> MM2 should read all offset syncs at start up and should not set consumer
> offset higher than the end offset
> ----------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-13659
> URL: https://issues.apache.org/jira/browse/KAFKA-13659
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Reporter: Kanalas Vidor
> Assignee: Kanalas Vidor
> Priority: Major
>
> - MirrorCheckpointTask uses OffsetSyncStore, and does not check whether
> OffsetSyncStore managed to read to the "end" of the offset-syncs topic.
> OffsetSyncStore should fetch the endoffset of the topic at startup, and set a
> flag when it finally reaches the endoffset in consumption.
> MirrorCheckpointTask.poll should wait for this flag to be true before doing
> any in-memory updates and group offset management.
> - MirrorCheckpointTask can create checkpoints which point into the "future"
> - meaning it sometimes translates consumer offsets in a way that the target
> offset is greater than the endoffset of the replica topic partition.
> MirrorCheckpointTask should fetch the endoffsets of the affected topics, and
> make sure that it does not try to set the consumer offset to anything higher
> than the endoffset.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
