Greg Harris created KAFKA-14666:
-----------------------------------
Summary: MM2 should translate consumer group offsets behind
replication flow
Key: KAFKA-14666
URL: https://issues.apache.org/jira/browse/KAFKA-14666
Project: Kafka
Issue Type: Improvement
Components: mirrormaker
Affects Versions: 3.5.0
Reporter: Greg Harris
MirrorMaker2 includes an offset translation feature which can translate the
offsets for an upstream consumer group to a corresponding downstream consumer
group. It does this by keeping a topic of offset-syncs to correlate upstream
and downstream offsets, and translates any source offsets which are ahead of
the replication flow.
However, if a replication flow is closer to the end of a topic than the
consumer group, then the offset translation feature will refuse to translate
the offset for correctness reasons. This is because the MirrorCheckpointTask
only keeps the latest offset correlation between source and target, it does not
have sufficient information to translate older offsets.
The workarounds for this issue are to:
1. Pause the replication flow occasionally to allow the source to get ahead of
MM2
2. Increase the offset.lag.max to delay offset syncs, increasing the window for
translation to happen. With the fix for KAFKA-12468, this will also increase
the lag of applications that are ahead of the replication flow, so this is a
tradeoff.
Instead, the MirrorCheckpointTask should provide correct and best-effort
translation for consumer groups behind the replication flow by keeping
additional state, or re-reading the offset-syncs topic. This should be a
substantial improvement for use-cases where applications have a higher latency
to commit than the replication flow, or where applications are reading from the
earliest offset.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
