GiminKim created KAFKA-20932:
--------------------------------
Summary: RemoteLogLeaderEpochState loses replaced segments from
leader epoch listings
Key: KAFKA-20932
URL: https://issues.apache.org/jira/browse/KAFKA-20932
Project: Kafka
Issue Type: Bug
Components: Tiered-Storage
Reporter: GiminKim
Assignee: GiminKim
RemoteLogLeaderEpochState tracks COPY_SEGMENT_FINISHED segments in an
offsetToId map and keeps overwritten segments in unreferencedSegmentIds so that
leader-epoch listings can return them for cleanup.
Reproduction
1. Add S0 for offsets [0, 100].
2. Add S1 for offsets [101, 200].
3. Add a later overlapping segment S2 for offsets [0, 150].
After S1, highestLogOffset is 200. While processing S2, the removal loop is
skipped because 200 <= 150 is false. offsetToId.put(0, S2) then replaces S0,
but the returned S0 id is ignored.
Expected
S0 is moved to unreferencedSegmentIds. The leader-epoch listing contains S0,
S1, and S2, with S0 available for retention cleanup.
Actual
S0 is in neither offsetToId nor unreferencedSegmentIds. It remains in the
partition-wide idToSegmentMetadata map but disappears from
listRemoteLogSegments(topicIdPartition, leaderEpoch).
Impact
RemoteLogManager retention iterates leader-epoch listings. The lost segment is
not considered for normal retention or leader-epoch cleanup, so it can remain
in remote storage for the lifetime of the partition. It also keeps the
partition-wide metadata and per-epoch views inconsistent.
Production reachability
Overlapping segments are expected during leader changes (KAFKA-15168). The
exact ordering can occur when an old leader starts copying S1 while
highestLogOffset is 100, a new leader selects a local segment S2 containing
offset 101, and the COPY_SEGMENT_FINISHED events are applied in S1 then S2
order. KAFKA-16890 also documents same-start remote segments with different end
offsets after broker failover.
Regression
Before PR #14004, handleSegmentWithCopySegmentFinishedState used the value
returned by offsetToId.put and added the replaced id to unreferencedSegmentIds.
PR #14004 removed that fallback while adding overlap handling.
Proposed fix
Add the distinct segment id returned by offsetToId.put to
unreferencedSegmentIds and add a regression test for the S0/S1/S2 ordering.
This restores tracking without changing the existing lookup selection policy.
Related:
- KAFKA-15168
- https://github.com/apache/kafka/pull/14004
- KAFKA-16890
- https://github.com/apache/kafka/pull/16237
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
