bbejeck opened a new pull request, #23339:
URL: https://github.com/apache/kafka/pull/23339
Under EOS, a source-topic-backed KTable store can be silently emptied by a
double race during a broker rolling restart:
1. An `AddOffsetsToTxn` timeout raises `TaskCorruptedException`, so the task
is closed dirty and its state store is wiped.
2. On the re-restore, `StoreChangelogReader` fetches the group's committed
offset to compute the restore ceiling min(endOffset, committedOffset). A
resigning/unavailable coordinator can transiently answer -1/NONE ("no committed
offset"), which was coerced to 0 → ceiling 0 → the wiped store is declared
fully restored while empty, returning null for keys that exist in the topic.
The trouble is that -1 is also the legitimate answer for a brand-new group,
where 0 is correct — so we can't just retry every missing offset (that would
stall every fresh source-table startup).
The fix is to distinguish the two cases by whether the changelog has
restored before in this process. Fresh changelog → take 0 immediately
(unchanged fast path); a previously-restored (i.e. wiped) changelog → treat a
missing committed offset as possibly-transient and retry it, bounded by
task.timeout.ms, before falling back to 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
