[ https://issues.apache.org/jira/browse/KAFKA-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Gustafson resolved KAFKA-4298. ------------------------------------ Resolution: Fixed Fix Version/s: (was: 0.10.0.2) Issue resolved by pull request 2019 [https://github.com/apache/kafka/pull/2019] > LogCleaner writes inconsistent compressed message set if topic message format > != message format > ----------------------------------------------------------------------------------------------- > > Key: KAFKA-4298 > URL: https://issues.apache.org/jira/browse/KAFKA-4298 > Project: Kafka > Issue Type: Bug > Affects Versions: 0.10.0.1 > Reporter: Jason Gustafson > Assignee: Jason Gustafson > Priority: Critical > Fix For: 0.10.1.0 > > > When cleaning the log, we don't want to convert messages to the format > configured for the topic due to KAFKA-3915. However, the cleaner logic for > writing compressed messages (in case some messages in the message set were > not retained) writes the topic message format version in the magic field of > the outer message instead of the actual message format. The choice of the > absolute/relative offset for the inner messages will also be based on the > topic message format version. > For example, if there is an old compressed message set with magic=0 in the > log and the topic is configured for magic=1, then after cleaning, the new > message set will have a wrapper with magic=1, the nested messages will still > have magic=0, but the message offsets will be relative. If this happens, > there does not seem to be an easy way to recover without manually fixing up > the log. > The offsets still work correctly as both the clients and broker use the outer > message format version to decide if the relative offset needs to be converted > to an absolute offset. So the main problem turns out to be that > `ByteBufferMessageSet.deepIterator` throws an exception if there is a > mismatch between outer and inner message format version. > {code} > if (newMessage.magic != wrapperMessage.magic) > throw new IllegalStateException(s"Compressed message has magic > value ${wrapperMessage.magic} " + > s"but inner message has magic value ${newMessage.magic}") > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)