[
https://issues.apache.org/jira/browse/KAFKA-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jun Rao resolved KAFKA-17076.
-----------------------------
Fix Version/s: 4.0.0
Assignee: Vincent Jiang
Resolution: Fixed
Merged the PR to trunk.
> logEndOffset could be lost due to log cleaning
> ----------------------------------------------
>
> Key: KAFKA-17076
> URL: https://issues.apache.org/jira/browse/KAFKA-17076
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Jun Rao
> Assignee: Vincent Jiang
> Priority: Major
> Fix For: 4.0.0
>
>
> It's possible for the log cleaner to remove all records in the suffix of the
> log. If the partition is then reassigned, the new replica won't be able to
> see the true logEndOffset since there is no record batch associated with it.
> If this replica becomes the leader, it will assign an already used offset to
> a newly produced record, which is incorrect.
>
> It's relatively rare to trigger this issue since the active segment is never
> cleaned and typically is not empty. However, the following is one possibility.
> # records with offset 100-110 are produced and fully replicated to all ISR.
> All those records are delete records for certain keys.
> # record with offset 111 is produced. It forces the roll of a new segment in
> broker b1 and is added to the log. The record is not committed and is later
> truncated from the log, leaving an empty active segment in this log. b1 at
> some point becomes the leader.
> # log cleaner kicks in and removes records 100-110.
> # The partition is reassigned to another broker b2. b2 replicates all
> records from b1 up to offset 100 and marks its logEndOffset at 100. Since
> there is no record to replicate after offset 100 in b1, b2's logEndOffset
> stays at 100 and b2 can join the ISR.
> # b2 becomes the leader and assign offset 100 to a new record.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
