Jason Gustafson created KAFKA-5443:
--------------------------------------
Summary: Consumer should use last offset from batch to set next
fetch offset
Key: KAFKA-5443
URL: https://issues.apache.org/jira/browse/KAFKA-5443
Project: Kafka
Issue Type: Sub-task
Reporter: Jason Gustafson
Assignee: Jason Gustafson
Fix For: 0.11.0.0
With message format v2, the log cleaner preserves the last offset in each batch
even if the last record is removed. Currently when the batch is consumed by the
consumer, we use the last record in the batch to determine the next offset to
fetch. So if the last record in the batch was removed through compaction, the
next fetch offset will still point to an offset in the current batch and it
will be refetched. In the worst case, if the fetch size has room for that
batch, the consumer will not be able to make progress. To fix this, we should
advance the next fetch offset to the last offset from the batch once we have
consumed that batch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
