[
https://issues.apache.org/jira/browse/KAFKA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16048688#comment-16048688
]
ASF GitHub Bot commented on KAFKA-5443:
---------------------------------------
GitHub user hachikuji opened a pull request:
https://github.com/apache/kafka/pull/3331
KAFKA-5443: Consumer should use last offset from batch to set next fetch
offset
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hachikuji/kafka KAFKA-5443
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3331.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3331
----
commit b991c8185c5bce1cedb39dc5158afaed1b8db2f4
Author: Jason Gustafson <[email protected]>
Date: 2017-06-14T04:17:33Z
KAFKA-5443: Consumer should use last offset from batch to set next fetch
offset
----
> Consumer should use last offset from batch to set next fetch offset
> -------------------------------------------------------------------
>
> Key: KAFKA-5443
> URL: https://issues.apache.org/jira/browse/KAFKA-5443
> Project: Kafka
> Issue Type: Sub-task
> Components: clients, core, producer
> Reporter: Jason Gustafson
> Assignee: Jason Gustafson
> Fix For: 0.11.0.0
>
>
> With message format v2, the log cleaner preserves the last offset in each
> batch even if the last record is removed. Currently when the batch is
> consumed by the consumer, we use the last record in the batch to determine
> the next offset to fetch. So if the last record in the batch was removed
> through compaction, the next fetch offset will still point to an offset in
> the current batch and it will be refetched. In the worst case, if the fetch
> size has room for that batch, the consumer will not be able to make progress.
> To fix this, we should advance the next fetch offset to the last offset from
> the batch once we have consumed that batch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
