Ismael Juma updated KAFKA-4298:
    Summary: LogCleaner writes inconsistent compressed message set if topic 
message format != message format  (was: LogCleaner does not convert compressed 
message sets properly)

> LogCleaner writes inconsistent compressed message set if topic message format 
> != message format
> -----------------------------------------------------------------------------------------------
>                 Key: KAFKA-4298
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4298
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions:
>            Reporter: Jason Gustafson
>            Assignee: Jason Gustafson
>            Priority: Critical
>             Fix For:,
> When cleaning the log, we don't want to convert messages to the format 
> configured for the topic due to KAFKA-3915. However, the cleaner logic for 
> writing compressed messages (in case some messages in the message set were 
> not retained) writes the topic message format version in the magic field of 
> the outer message instead of the actual message format. The choice of the 
> absolute/relative offset for the inner messages will also be based on the 
> topic message format version.
> For example, if there is an old compressed message set with magic=0 in the 
> log and the topic is configured for magic=1, then after cleaning, the new 
> message set will have a wrapper with magic=1, the nested messages will still 
> have magic=0, but the message offsets will be relative. If this happens, 
> there does not seem to be an easy way to recover without manually fixing up 
> the log.
> The offsets still work correctly as both the clients and broker use the outer 
> message format version to decide if the relative offset needs to be converted 
> to an absolute offset. So the main problem turns out to be that 
> `ByteBufferMessageSet.deepIterator` throws an exception if there is a 
> mismatch between outer and inner message format version.
> {code}
> if (newMessage.magic != wrapperMessage.magic)
>           throw new IllegalStateException(s"Compressed message has magic 
> value ${wrapperMessage.magic} " +
>             s"but inner message has magic value ${newMessage.magic}")
> {code}

This message was sent by Atlassian JIRA

Reply via email to