Daniel Urban created KAFKA-13452:
------------------------------------
Summary: MM2 creates invalid checkpoint when offset mapping is not
available
Key: KAFKA-13452
URL: https://issues.apache.org/jira/browse/KAFKA-13452
Project: Kafka
Issue Type: Improvement
Components: mirrormaker
Reporter: Daniel Urban
MM2 checkpointing reads the offset-syncs topic to create offset mappings for
committed consumer group offsets. In some corner cases, it is possible that a
mapping is not available in offset-syncs - in that case, MM2 simply copies the
source offset, which might not be a valid offset in the replica topic at all.
One possible situation is if there is an empty topic in the source cluster with
a non-zero endoffset (e.g. retention already removed the records), and a
consumer group which has a committed offset set to the end offset. If
replication is configured to start replicating this topic, it will not have an
offset mapping available in offset-syncs (as the topic is empty), causing MM2
to copy the source offset.
This can cause issues when auto offset sync is enabled, as the consumer group
offset can be potentially set to a high number. MM2 never rewinds these
offsets, so even when there is a correct offset mapping available, the offset
will not be updated correctly.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
