[ https://issues.apache.org/jira/browse/KAFKA-15906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Harris updated KAFKA-15906: -------------------------------- Description: Right now, the offset.lag.max configuration limits the number of offset syncs are emitted by the MirrorSourceTask, along with a fair rate-limiting semaphore. After 100 records have been emitted for a partition, _and_ the semaphore is available, an offset sync can be emitted. For low-volume topics, the `offset.lag.max` default of 100 is much more restrictive than the rate-limiting semaphore. For example, a topic which mirrors at the rate of 1 record/sec may take 100 seconds to receive an offset sync. If the topic is actually finite, the last offset sync will never arrive, and the translation will have a persistent lag. Instead, we can periodically flush the offset syncs for partitions that are under the offset.lag.max limit, but have not received an offset sync recently. This could be a new configuration, be a hard-coded time, or be based on the existing emit.checkpoints.interval.seconds and sync.group.offsets.interval.seconds configurations. Alternatively, we could decrease the default `offset.lag.max` value to 0, and rely on the fair semaphore to limit the number of syncs emitted for high-throughput partitions. The semaphore is not currently configurable, so users wanting lower throughput on the offset-syncs topic will still need an offset.lag.max > 0. was: Right now, the offset.lag.max configuration limits the number of offset syncs are emitted by the MirrorSourceTask, along with a fair rate-limiting semaphore. After 100 records have been emitted for a partition, _and_ the semaphore is available, an offset sync can be emitted. For low-volume topics, the `offset.lag.max` default of 100 is much more restrictive than the rate-limiting semaphore. For example, a topic which mirrors at the rate of 1 record/sec may take 100 seconds to receive an offset sync. If the topic is actually finite, the last offset sync will never arrive, and the translation will have a persistent lag. Instead, we can periodically flush the offset syncs for partitions that are under the offset.lag.max limit, but have not received an offset sync recently. This could be a new configuration, be a hard-coded time, or be based on the existing emit.checkpoints.interval.seconds and sync.group.offsets.interval.seconds configurations. > Emit offset syncs more often than offset.lag.max for low-throughput/finite > partitions > ------------------------------------------------------------------------------------- > > Key: KAFKA-15906 > URL: https://issues.apache.org/jira/browse/KAFKA-15906 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker > Reporter: Greg Harris > Priority: Minor > > Right now, the offset.lag.max configuration limits the number of offset syncs > are emitted by the MirrorSourceTask, along with a fair rate-limiting > semaphore. After 100 records have been emitted for a partition, _and_ the > semaphore is available, an offset sync can be emitted. > For low-volume topics, the `offset.lag.max` default of 100 is much more > restrictive than the rate-limiting semaphore. For example, a topic which > mirrors at the rate of 1 record/sec may take 100 seconds to receive an offset > sync. If the topic is actually finite, the last offset sync will never > arrive, and the translation will have a persistent lag. > Instead, we can periodically flush the offset syncs for partitions that are > under the offset.lag.max limit, but have not received an offset sync > recently. This could be a new configuration, be a hard-coded time, or be > based on the existing emit.checkpoints.interval.seconds and > sync.group.offsets.interval.seconds configurations. > > Alternatively, we could decrease the default `offset.lag.max` value to 0, and > rely on the fair semaphore to limit the number of syncs emitted for > high-throughput partitions. The semaphore is not currently configurable, so > users wanting lower throughput on the offset-syncs topic will still need an > offset.lag.max > 0. -- This message was sent by Atlassian Jira (v8.20.10#820010)