Greg Harris created KAFKA-15202:
-----------------------------------
Summary: MM2 OffsetSyncStore clears too many syncs when sync
spacing is variable
Key: KAFKA-15202
URL: https://issues.apache.org/jira/browse/KAFKA-15202
Project: Kafka
Issue Type: Bug
Components: mirrormaker
Affects Versions: 3.4.1, 3.5.0, 3.3.3
Reporter: Greg Harris
The spacing between OffsetSyncs can vary significantly, due to conditions in
the upstream topic and in the replication rate of the MirrorSourceTask.
The OffsetSyncStore attempts to keep a maximal number of distinct syncs
present, and for regularly spaced syncs it does not allow an incoming sync to
expire more than one other unique sync. There are tests to enforce this
property.
For variable spaced syncs, there is no such guarantee, because multiple
fine-grained syncs may need to be expired at the same time. However, instead of
only those fine-grained syncs being expired, the store may also expire
coarser-grained syncs. This causes a large decrease in the number of unique
syncs.
This is an extremely simple example:
* Syncs: 0 (start), 1, 2, 4.
The result:
```
TRACE New sync OffsetSync\{topicPartition=topic1-2, upstreamOffset=1,
downstreamOffset=1} applied, new state is [1:1,0:0]
(org.apache.kafka.connect.mirror.OffsetSyncStore:194)
TRACE New sync OffsetSync\{topicPartition=topic1-2, upstreamOffset=2,
downstreamOffset=2} applied, new state is [2:2,1:1,0:0]
(org.apache.kafka.connect.mirror.OffsetSyncStore:194)
TRACE New sync OffsetSync\{topicPartition=topic1-2, upstreamOffset=4,
downstreamOffset=4} applied, new state is [4:4,0:0]
(org.apache.kafka.connect.mirror.OffsetSyncStore:194)
```
Instead of being expired, the `2:2` sync should still be present in the final
state, allowing the store to maintain 3 unique syncs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
