Dhruvil Shah edited comment on KAFKA-6264 at 3/8/18 12:57 AM:

[~becket_qin] if you don't have a working fix yet, would you mind if I picked 
this issue up?

was (Author: dhruvilshah):
[~becket_qin] would you mind if I picked this issue up?

> Log cleaner thread may die on legacy segment containing messages whose 
> offsets are too large
> --------------------------------------------------------------------------------------------
>                 Key: KAFKA-6264
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6264
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions:, 1.0.0,
>            Reporter: Jiangjie Qin
>            Assignee: Jiangjie Qin
>            Priority: Critical
>             Fix For: 1.2.0
> We encountered a problem that some of the legacy log segments contains 
> messages whose offsets are larger than {{SegmentBaseOffset + Int.MaxValue}}.
> Prior to, we do not assert the offset of the messages when appending 
> them to the log segments. Due to KAFKA-5413, the log cleaner may append 
> messages whose offset is greater than {{base_offset + Int.MaxValue}} into the 
> segment during the log compaction.
> After the brokers are upgraded, those log segments cannot be compacted 
> anymore because the compaction will fail immediately due to the offset range 
> assertion we added to the LogSegment.
> We have seen this issue in the __consumer_offsets topic so it could be a 
> general problem. There is no easy solution for the users to recover from this 
> case. 
> One solution is to split such log segments in the log cleaner once it sees a 
> message with problematic offset and append those messages to a separate log 
> segment with a larger base_offset.
> Due to the impact of the issue. We may want to consider backporting the fix 
> to previous affected versions.

This message was sent by Atlassian JIRA

Reply via email to