Omnia Ibrahim created KAFKA-16254:
-------------------------------------
Summary: Allow MM2 to fully disable offset sync feature
Key: KAFKA-16254
URL: https://issues.apache.org/jira/browse/KAFKA-16254
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 3.6.0, 3.5.0, 3.7.0
Reporter: Omnia Ibrahim
Assignee: Omnia Ibrahim
*Background:*
At the moment syncing offsets feature in MM2 is broken to 2 parts
# One is in `MirrorSourceTask` where we store the new recored's offset on
target cluster to {{offset_syncs}} internal topic after mirroring the record.
Before KAFKA-14610 in 3.5 MM2 used to just queue the offsets and publish them
later but since 3.5 this behaviour changed we now publish any offset syncs that
we've queued up, but have not yet been able to publish when
`MirrorSourceTask.commit` get invoked. This introduced an over head to commit
process.
# The second part is in checkpoints source task where we use the new record
offsets from {{offset_syncs}} and update {{checkpoints}} and
{{__consumer_offsets}} topics.
*Problem:*
For customers who only use MM2 for mirroring data and not interested in syncing
offsets feature they now can disable the second part of this feature which is
by disabling {{emit.checkpoints.enabled}} and/or {{sync.group.offsets.enabled}}
to disable emitting {{__consumer_offsets}} topic but nothing disabling 1st part
of the feature.
The problem get worse if they disabled MM2 from creating offset syncs internal
topic as
1. this will increase throughput as MM2 will try to force trying to update the
offset with every mirrored batch which impacting the performance of our MM2.
2. Get too many error logs because they don't create the sync offset topic as
they don't use the feature.
*Possible solution:*
Allow customers to fully disable the feature if they don't really need it
similar to how we fully can disable other MM2 features like heartbeat feature
by adding a new config.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
