[
https://issues.apache.org/jira/browse/KAFKA-20876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101109#comment-18101109
]
Yunseop Eom commented on KAFKA-20876:
-------------------------------------
Opened PR #23029: https://github.com/apache/kafka/pull/23029
The windowed changelog restore optimization previously probed only endOffset -
1. When that offset is a transaction control record, the consumer filters it
from poll results and the optimization incorrectly falls back to restoring from
the beginning.
The PR retries from the preceding offset before falling back. TDD verification
reproduced the failure on the baseline branch and passed after the
implementation change. The affected StoreChangelogReaderTest, streams
checkstyle, SpotBugs, and git diff --check pass locally. The PR is open and
awaiting review.
> Windowed restore optimisation gives up when `endOffset-1` is a transaction
> control record
> -----------------------------------------------------------------------------------------
>
> Key: KAFKA-20876
> URL: https://issues.apache.org/jira/browse/KAFKA-20876
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.4.0
> Reporter: Alan Lau
> Assignee: Alan Lau
> Priority: Blocker
>
> When restoring a windowed store with no checkpoint, {{StoreChangelogReader}}
> seeks to {{latestTimestamp - retentionPeriod}} instead of {{{}log-start{}}},
> since data older than the retention is discarded on write. It learns
> {{latestTimestamp}} by seeking to {{endOffset - 1}} and polling once in
> {{{}seekNewPartitions{}}}; an empty poll falls back to
> {{{}seekToBeginning{}}}.
> On a 4.3 EOS-v2 soak, {{endOffset - 1}} returned no record on 195 of 195
> probes. Dumping the segment confirms the last batch is a transaction control
> record (isControl: true, count: 1, size: 78), and the pattern recurs
> throughout — every transaction ends with a marker occupying an offset that is
> never delivered to a consumer. The poll therefore returns nothing and the
> optimisation is abandoned, leaving the restore at log-start with no margin
> against retention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
