[
https://issues.apache.org/jira/browse/KAFKA-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton resolved KAFKA-14663.
-----------------------------------
Resolution: Duplicate
> High throughput topics can starve low-throughput MM2 offset syncs
> -----------------------------------------------------------------
>
> Key: KAFKA-14663
> URL: https://issues.apache.org/jira/browse/KAFKA-14663
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.1.0, 3.0.0, 3.3.0, 3.4.0, 3.5.0
> Reporter: Greg Harris
> Assignee: Greg Harris
> Priority: Major
>
> In MM2, a semaphore is used to throttle the number of offset syncs written to
> the offset-syncs topic. If too many offset writes are requested (for example,
> from high-throughput topics) then some are silently dropped and never
> retried. This is acceptable for a single topic-partition, where a later
> record may re-trigger the offset-sync and write the sync successfully.
> However, if there is a large variance between throughput in the topics
> emitted by an MM2 instance, it is possible for high-throughput topics to
> trigger many offset syncs, and cause the offset-syncs for a co-located
> low-throughput topic to be unfairly dropped.
> This can cause the downstream offsets for the starved topic to lag behind
> significantly, or be prevented completely.
> Instead, we should have some sort of fairness mechanism where low-thoughput
> topics are given similar priority to high-throughput topics in producing
> offset syncs, and cause excess sync messages from high-throughput topics to
> be dropped instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
