Matthias J. Sax created KAFKA-21550:
---------------------------------------

             Summary: Standby task on a source-topic changelog reports end 
offset 0 if its committed offset cannot be fetched
                 Key: KAFKA-21550
                 URL: https://issues.apache.org/jira/browse/KAFKA-21550
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 4.4.0
            Reporter: Matthias J. Sax


StoreChangelogReader#register() sets restoreEndOffset = 0L for a standby task 
on a source-topic changelog. This disables restoration until the committed 
offset is known.

In initializeChangelogs(), only an active changelog is removed from 
newPartitionsToRestore when its offset lookup fails. If the committed-offset 
lookup fails for a standby, committedOffsetForChangelogs() returns an empty 
map, so updateLimitOffsetsForStandbyChangelogs() is skipped:

 
{code:java}
if (!committedOffsets.isEmpty()) {
    updateLimitOffsetsForStandbyChangelogs(committedOffsets);
}{code}
 

The changelog still moves to RESTORING with restoreEndOffset == 0. 
initializeChangelogs() only handles REGISTERED changelogs, so it never retries 
this one. The value is only corrected by 
maybeUpdateLimitOffsetsForStandbyChangelogs(), which requires STANDBY_UPDATING, 
buffered records and an elapsed commit interval. It is never corrected while 
active tasks restore.

Meanwhile logicalChangelogEndOffsets() reports an end offset of 0 for this 
changelog. The broker computes the lag as endOffset - offset and treats the 
task as caught up.

A standby task on a source-topic changelog should stay REGISTERED until its 
committed offset is known. Until then, its end offset should be reported as 
unknown, because logicalChangelogEndOffsets() also reports REGISTERED 
changelogs.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to