[jira] [Commented] (KAFKA-4298) LogCleaner does not convert compressed message sets properly

2016-10-13 Thread Ismael Juma (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571432#comment-15571432
 ] 

Ismael Juma commented on KAFKA-4298:


Great catch. I'm a bit unsure why this hasn't been reported so far. I think the 
correct description is:

"When cleaning the log, we don't want to convert messages to the format 
configured for the topic due to KAFKA-3915. However, the cleaner logic for 
writing compressed messages (in case some messages in the message set were not 
retained) writes the topic message format version in the magic field of the 
outer message instead of the actual message format. The choice of the 
absolute/relative offset for the inner messages will also be based on the topic 
message format version.

For example, if there is an old compressed message set with magic=0 in the log 
and the topic is configured for magic=1, then after cleaning, the new message 
set will have a wrapper with magic=1, the nested messages will still have 
magic=0, but the message offsets will be relative. If this happens, there does 
not seem to be an easy way to recover without manually fixing up the log."

> LogCleaner does not convert compressed message sets properly
> 
>
> Key: KAFKA-4298
> URL: https://issues.apache.org/jira/browse/KAFKA-4298
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 0.10.1.0, 0.10.0.2
>
>
> When cleaning the log, we attempt to write the cleaned messages using the 
> message format configured for the topic, but as far as I can tell, we do not 
> convert the wrapped messages in compressed message sets. For example, if 
> there is an old compressed message set with magic=0 in the log and the topic 
> is configured for magic=1, then after cleaning, the new message set will have 
> a wrapper with magic=1, but the nested messages will still have magic=0. If 
> this happens, there does not seem to be an easy way to recover without 
> manually fixing up the log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4298) LogCleaner does not convert compressed message sets properly

2016-10-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15570935#comment-15570935
 ] 

ASF GitHub Bot commented on KAFKA-4298:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/2019

KAFKA-4298: Ensure compressed message sets are converted when cleaning the 
log



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-4298

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2019.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2019


commit af3b31b4d94ece1603ac470bfd8d781558987501
Author: Jason Gustafson 
Date:   2016-10-13T04:56:52Z

KAFKA-4298: Ensure compressed message sets are converted when cleaning the 
log




> LogCleaner does not convert compressed message sets properly
> 
>
> Key: KAFKA-4298
> URL: https://issues.apache.org/jira/browse/KAFKA-4298
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 0.10.1.1
>
>
> When cleaning the log, we attempt to write the cleaned messages using the 
> message format configured for the topic, but as far as I can tell, we do not 
> convert the wrapped messages in compressed message sets. For example, if 
> there is an old compressed message set with magic=0 in the log and the topic 
> is configured for magic=1, then after cleaning, the new message set will have 
> a wrapper with magic=1, but the nested messages will still have magic=0. If 
> this happens, there does not seem to be an easy way to recover without 
> manually fixing up the log.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)