[jira] [Commented] (KAFKA-5443) Consumer should use last offset from batch to set next fetch offset

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049371#comment-16049371
 ] 

ASF GitHub Bot commented on KAFKA-5443:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3331


> Consumer should use last offset from batch to set next fetch offset
> ---
>
> Key: KAFKA-5443
> URL: https://issues.apache.org/jira/browse/KAFKA-5443
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.11.0.0
>
>
> With message format v2, the log cleaner preserves the last offset in each 
> batch even if the last record is removed. Currently when the batch is 
> consumed by the consumer, we use the last record in the batch to determine 
> the next offset to fetch. So if the last record in the batch was removed 
> through compaction, the next fetch offset will still point to an offset in 
> the current batch and it will be refetched. In the worst case, if the fetch 
> size has room for that batch, the consumer will not be able to make progress. 
> To fix this, we should advance the next fetch offset to the last offset from 
> the batch once we have consumed that batch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KAFKA-5443) Consumer should use last offset from batch to set next fetch offset

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16048688#comment-16048688
 ] 

ASF GitHub Bot commented on KAFKA-5443:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/3331

KAFKA-5443: Consumer should use last offset from batch to set next fetch 
offset



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-5443

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3331.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3331


commit b991c8185c5bce1cedb39dc5158afaed1b8db2f4
Author: Jason Gustafson 
Date:   2017-06-14T04:17:33Z

KAFKA-5443: Consumer should use last offset from batch to set next fetch 
offset




> Consumer should use last offset from batch to set next fetch offset
> ---
>
> Key: KAFKA-5443
> URL: https://issues.apache.org/jira/browse/KAFKA-5443
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.11.0.0
>
>
> With message format v2, the log cleaner preserves the last offset in each 
> batch even if the last record is removed. Currently when the batch is 
> consumed by the consumer, we use the last record in the batch to determine 
> the next offset to fetch. So if the last record in the batch was removed 
> through compaction, the next fetch offset will still point to an offset in 
> the current batch and it will be refetched. In the worst case, if the fetch 
> size has room for that batch, the consumer will not be able to make progress. 
> To fix this, we should advance the next fetch offset to the last offset from 
> the batch once we have consumed that batch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)