Bill Bejeck created KAFKA-20805:
-----------------------------------
Summary: State directory cleaned prematurely under KIP-1035,
forcing from-scratch restore and OffsetOutOfRange after a StreamThread failure
Key: KAFKA-20805
URL: https://issues.apache.org/jira/browse/KAFKA-20805
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 4.4.0
Reporter: Bill Bejeck
Assignee: Bill Bejeck
Fix For: 4.4.0
After a StreamThread dies and releases its task locks, the background
state-directory cleaner can delete a still-valid task directory whose task
committed only seconds earlier. The reassigned task then finds no local
offsets, restores its changelog from the beginning, and — for a
compacted/retention-deleted changelog — possibly the restore is overtaken by
retention mid-flight, producing \{{OffsetOutOfRangeException}} →
\{{TaskCorruptedException}} → rebalance.
The cleaner decides obsolescence from the task directory's filesystem mtime:
\{{now - taskDir.lastModified() > state.cleanup.delay.ms}}
(StateDirectory.cleanRemovedTasks). This relied on the pre-KIP-1035 behavior
where the per-commit \{{.checkpoint}} file (a direct child of the task dir)
refreshed the dir mtime on every commit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
