Jay Kreps created KAFKA-546:
-------------------------------
Summary: Fix commit() in zk consumer
Key: KAFKA-546
URL: https://issues.apache.org/jira/browse/KAFKA-546
Project: Kafka
Issue Type: New Feature
Affects Versions: 0.8
Reporter: Jay Kreps
In 0.7.x and earlier versions offsets were assigned by the byte location in the
file. Because it wasn't possible to directly decompress from the middle of a
compressed block, messages inside a compressed message set effectively had no
offset. As a result the offset given to the consumer was always the offset of
the wrapper message set.
In 0.8 after the logical offsets patch messages in a compressed set do have
offsets. However the server still needs to fetch from the beginning of the
compressed messageset (otherwise it can't be decompressed). As a result a
commit() which occurs in the middle of a message set will still result in some
duplicates.
This can be fixed in the ConsumerIterator by discarding messages smaller than
the fetch offset rather than giving them to the consumer. This will make commit
work correctly in the presence of compressed messages (finally).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
