cheeeee opened a new pull request, #24072:
URL: https://github.com/apache/kafka/pull/24072
When a cleaning pass fills its offset map, it stops in the middle of a
segment and the next pass starts at that offset. `buildOffsetMap` seeds
`CleanedTransactionMetadata` with `log.collectAbortedTransactions(start,
end)`, which uses the log start offset as its lower bound, so it also
returns aborted transactions of that segment that ended before `start`.
The pass begins reading at the offset index entry at or below `start`
and may never see their ABORT markers. Every later transactional batch
of such a producer, including committed ones, is then treated as
aborted and its keys are left out of the offset map. A committed
tombstone is ignored: the older value of the key is kept, the tombstone
is removed after `delete.retention.ms`, and the deleted key comes back
(KAFKA-21410).
This change keeps only the aborted transactions with `lastOffset() >=
start` when building the offset map. A transaction that began before
`start` and ends at or after it is still seeded. `cleanSegments` still
collects the aborted transactions of the whole segment, which it needs
to rebuild the transaction index (KAFKA-13727), so
`UnifiedLog.collectAbortedTransactions` does not change.
Tests in `LogCleanerTest`:
* `testCommittedTombstoneAfterAbortedTransactionBeforeCleaningStart`:
the deleted key does not come back; fails without the fix.
* `testBuildOffsetMapSkipsAbortedTransactionSpanningCleaningStart`: a
transaction aborted after `start` that began before it stays out of
the offset map.
Generated-by: Claude (Anthropic)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
