Charles-Eddy created KAFKA-21314:
------------------------------------
Summary: Share partition acquire throws "inconsistent range" at
the in-flight limit on a compacted topic
Key: KAFKA-21314
URL: https://issues.apache.org/jira/browse/KAFKA-21314
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 4.5.0
Reporter: Charles-Eddy
Attachments: 2-inconsistent-range-at-in-flight-limit.patch
I found this while prototyping key-based ordering for share groups. It
reproduces on plain trunk.
When a share partition is at share.partition.max.record.locks,
lastOffsetAndMaxRecordsToAcquire caps lastOffsetToAcquire to the end offset. On
a compacted topic the records between the fetch offset and the end offset may
be gone, so the first fetched batch can start after the end offset, and
acquire() calls cachedState.subMap(from, to) with from > to:
{code:java}
java.lang.IllegalArgumentException: inconsistent range
at
java.base/java.util.concurrent.ConcurrentSkipListMap.subMap(ConcurrentSkipListMap.java:1912)
at kafka.server.share.SharePartition.acquire(SharePartition.java:842){code}
In the simple case maybeArchiveStaleBatches has already archived the missing
offsets, so one ShareFetch request fails with UNKNOWN_SERVER_ERROR and the
partition recovers. After a re-initialization with a persister read gap at the
start of the window, the gap window does not advance and every fetch at the gap
start fails the same way (I checked 3 consecutive fetches).
The attached patch adds a test for the simple
case:SharePartitionTest#testAcquireAtMaxInFlightRecordsWhenAvailableRecordsAreCompactedAway.
Returning an empty result when the first fetched batch starts after
lastOffsetToAcquire, and moving the gap start past the compacted range, should
fix it, I don't know if it creates other issues though, I let you decide how to
solve this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
