Matthias J. Sax created KAFKA-21567:
---------------------------------------

             Summary: StoreChangelogReader retries a missing committed offset 
in a busy loop and never times out
                 Key: KAFKA-21567
                 URL: https://issues.apache.org/jira/browse/KAFKA-21567
             Project: Kafka
          Issue Type: Bug
          Components: streams
            Reporter: Matthias J. Sax


StoreChangelogReader keeps a changelog REGISTERED until its offset lookups 
return it. That covers the end offset, plus the committed offset for a 
source-topic changelog. It retries in initializeChangelogs() on every restore() 
call. The affected changelogs are active changelogs, and, since KAFKA-21550, 
standby changelogs on a source topic.

ListConsumerGroupOffsetsHandler skips a partition with a partition-level error 
and logs a WARN, without failing. One example is UNSTABLE_OFFSET_COMMIT, while 
a transactional offset commit is pending. committedOffsetForChangelogs() then 
returns without that partition, so the changelog stays REGISTERED. Two problems 
follow:
 # *Busy loop.* If no other changelog is restoring, restore() returns without 
polling the restore consumer. DefaultStateUpdater doesn't wait either, because 
allWorkDone() is false while the task is still updating. The state updater 
thread sends one listConsumerGroupOffsets request per round trip, logging a 
WARN each time, until the partition comes back.
 # *No task timeout.* committedOffsetForChangelogs() calls clearTaskTimeout() 
for the tasks of all requested partitions, including those missing from the 
result. A persistent partition error therefore never fails the task after 
task.timeout.ms; it retries forever. The end-offset lookup doesn't have this 
problem, since listOffsets(...).all() throws on a partition error.

Proposed fix:
 * After a missing partition, wait retry.backoff.ms before looking up that 
changelog again.
 * Treat a missing partition as a failed lookup for its task: start or advance 
its task timeout, and clear the timeout only for tasks whose partitions all 
came back.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to