[
https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719723#comment-13719723
]
Jun Rao commented on KAFKA-987:
-------------------------------
1. The issue on startup is the following. If a consumer starts up from the end
of the log and there is no new message coming in, no offset will be
checkpointed to ZK. This will affect tools like ConsumerOffsetChecker.
2. During rebalance, a consumer may pick up offsets committed by other consumer
instances. If we don't update the offset cache in addPartitionTopicInfo(), we
will do an extra unnecessary offset update to ZK.
It seems to me that the impact for #1 is bigger than the slight performance
impact in #2. Another way to do that is to always force the very first offset
(per partition) write to ZK. However, I am not sure if it's worth the
complexity.
> Avoid checkpointing offsets in Kafka consumer that have not changed since the
> last commit
> -----------------------------------------------------------------------------------------
>
> Key: KAFKA-987
> URL: https://issues.apache.org/jira/browse/KAFKA-987
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Swapnil Ghike
> Assignee: Swapnil Ghike
> Labels: improvement
> Fix For: 0.8
>
> Attachments: kafka-987.patch, kafka-987-v2.patch
>
>
> We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets
> that have not changed since the last offset commit. This will help reduce the
> write load on zookeeper.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
