[ https://issues.apache.org/jira/browse/KAFKA-16641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848631#comment-17848631 ]
Adrian Preston edited comment on KAFKA-16641 at 5/23/24 11:38 AM: ------------------------------------------------------------------ -A further (related) question: assuming exactly once is enabled, can you foresee situations when the downstream offset can advance without at least an equal advance in the upstream offset?- Actually, I think I've found the answer my own question :) Enabling exactly once will result in transactions being used with the downstream mirrored topic, so the markers will cause the downstream offset to advance each time a transaction is committed / rolled back (without an equal advance in the upstream offset). was (Author: prestona): A further (related) question: assuming exactly once is enabled, can you foresee situations when the downstream offset can advance without at least an equal advance in the upstream offset? > MM2 offset translation should interpolate between sparse OffsetSyncs > -------------------------------------------------------------------- > > Key: KAFKA-16641 > URL: https://issues.apache.org/jira/browse/KAFKA-16641 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker > Reporter: Greg Harris > Priority: Major > > Right now, the OffsetSyncStore keeps a sparse offset store, with exponential > spacing between syncs. This can leave large gaps in translation, where > offsets are translated much more conservatively than necessary. > The dominant way to use MirrorMaker2 is in a "single writer" fashion, where > the target topic is only written to by a single mirror maker 2. When a topic > without gaps is replicated, contiguous blocks of offsets are preserved. For > example: > Say that MM2 mirrors 100 records, and emits two syncs: 0:100 and 100:200. We > can detect when the gap between the upstream and downstream offsets is the > same using subtraction, and then assume that 50:150 is also a valid > translation. If the source topic has gaps, or goes through a restart, we > should expect a discontinuity in the offset syncs, like 0:100 and 100:250 or > 0:100 and 100:150. > This may allow us to restore much of the offset translation precision that > was lost for simple contiguous topics, without additional memory usage, but > at the risk of mis-translating some pathological situations when the source > topic has gaps. This might be able to be enabled unconditionally, or enabled > via a configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)