Matthias J. Sax created KAFKA-21567:
---------------------------------------
Summary: StoreChangelogReader retries a missing committed offset
in a busy loop and never times out
Key: KAFKA-21567
URL: https://issues.apache.org/jira/browse/KAFKA-21567
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: Matthias J. Sax
StoreChangelogReader keeps a changelog REGISTERED until its offset lookups
return it. That covers the end offset, plus the committed offset for a
source-topic changelog. It retries in initializeChangelogs() on every restore()
call. The affected changelogs are active changelogs, and, since KAFKA-21550,
standby changelogs on a source topic.
ListConsumerGroupOffsetsHandler skips a partition with a partition-level error
and logs a WARN, without failing. One example is UNSTABLE_OFFSET_COMMIT, while
a transactional offset commit is pending. committedOffsetForChangelogs() then
returns without that partition, so the changelog stays REGISTERED. Two problems
follow:
# *Busy loop.* If no other changelog is restoring, restore() returns without
polling the restore consumer. DefaultStateUpdater doesn't wait either, because
allWorkDone() is false while the task is still updating. The state updater
thread sends one listConsumerGroupOffsets request per round trip, logging a
WARN each time, until the partition comes back.
# *No task timeout.* committedOffsetForChangelogs() calls clearTaskTimeout()
for the tasks of all requested partitions, including those missing from the
result. A persistent partition error therefore never fails the task after
task.timeout.ms; it retries forever. The end-offset lookup doesn't have this
problem, since listOffsets(...).all() throws on a partition error.
Proposed fix:
* After a missing partition, wait retry.backoff.ms before looking up that
changelog again.
* Treat a missing partition as a failed lookup for its task: start or advance
its task timeout, and clear the timeout only for tasks whose partitions all
came back.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)