Matthias J. Sax created KAFKA-21550:
---------------------------------------
Summary: Standby task on a source-topic changelog reports end
offset 0 if its committed offset cannot be fetched
Key: KAFKA-21550
URL: https://issues.apache.org/jira/browse/KAFKA-21550
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 4.4.0
Reporter: Matthias J. Sax
StoreChangelogReader#register() sets restoreEndOffset = 0L for a standby task
on a source-topic changelog. This disables restoration until the committed
offset is known.
In initializeChangelogs(), only an active changelog is removed from
newPartitionsToRestore when its offset lookup fails. If the committed-offset
lookup fails for a standby, committedOffsetForChangelogs() returns an empty
map, so updateLimitOffsetsForStandbyChangelogs() is skipped:
{code:java}
if (!committedOffsets.isEmpty()) {
updateLimitOffsetsForStandbyChangelogs(committedOffsets);
}{code}
The changelog still moves to RESTORING with restoreEndOffset == 0.
initializeChangelogs() only handles REGISTERED changelogs, so it never retries
this one. The value is only corrected by
maybeUpdateLimitOffsetsForStandbyChangelogs(), which requires STANDBY_UPDATING,
buffered records and an elapsed commit interval. It is never corrected while
active tasks restore.
Meanwhile logicalChangelogEndOffsets() reports an end offset of 0 for this
changelog. The broker computes the lag as endOffset - offset and treats the
task as caught up.
A standby task on a source-topic changelog should stay REGISTERED until its
committed offset is known. Until then, its end offset should be reported as
unknown, because logicalChangelogEndOffsets() also reports REGISTERED
changelogs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)