Bill Bejeck created KAFKA-21159:
-----------------------------------
Summary: Clear next offsets to consume on task close to avoid
stale commit after revive
Key: KAFKA-21159
URL: https://issues.apache.org/jira/browse/KAFKA-21159
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: Bill Bejeck
Fix For: 4.5.0, 4.4.1
A revived active task (`closeDirtyAndRevive`) kept
`nextOffsetsAndMetadataToBeConsumed`
from before the corruption. After the consumer is seeked back to its committed
offset, a
commit driven by processor metadata (`needsCommit`, which commits all input
partitions)
could fall back to that stale next offset and commit past records the task
hasn't
reprocessed — skipping them if the task later moves, and purging them for a
repartition
topic. The KAFKA-20416 `consumedOffsets.clear()` fix does not cover this path.
Fix: clear `nextOffsetsAndMetadataToBeConsumed` in `StreamTask.close()`,
alongside
`consumedOffsets`, so a revived task has no stale fallback for a partition it
hasn't
re-read.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
