Alan Lau created KAFKA-20876:
--------------------------------
Summary: Windowed restore optimisation gives up when `endOffset-1`
is a transaction control record
Key: KAFKA-20876
URL: https://issues.apache.org/jira/browse/KAFKA-20876
Project: Kafka
Issue Type: Bug
Reporter: Alan Lau
Assignee: Alan Lau
When restoring a windowed store with no checkpoint, {{StoreChangelogReader}}
seeks to {{latestTimestamp - retentionPeriod}} instead of {{{}log-start{}}},
since data older than the retention is discarded on write. It learns
{{latestTimestamp}} by seeking to {{endOffset - 1}} and polling once in
{{{}seekNewPartitions{}}}; an empty poll falls back to {{{}seekToBeginning{}}}.
On a 4.3 EOS-v2 soak, {{endOffset - 1}} returned no record on 195 of 195
probes. Dumping the segment confirms the last batch is a transaction control
record (isControl: true, count: 1, size: 78), and the pattern recurs throughout
— every transaction ends with a marker occupying an offset that is never
delivered to a consumer. The poll therefore returns nothing and the
optimisation is abandoned, leaving the restore at log-start with no margin
against retention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
