[
https://issues.apache.org/jira/browse/KAFKA-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14662845#comment-14662845
]
Guozhang Wang commented on KAFKA-1006:
--------------------------------------
Hi [~enn], the new Java consumer is going to be released to replace the old
Scala consumer soon, in which a rebalance callback is provided in the API such
that users can manually set the starting offset to be any value upon newly
added topic / partitions. This should resolve this problem. Would you be
willing to wait for the Java consumer and try it out? We can try to patch the
old Consumer implementation but at this stage I think people are trying to
minimize the changes unless it is a severe production issue.
> Consumer loses messages of a new topic with auto.offset.reset = largest
> -----------------------------------------------------------------------
>
> Key: KAFKA-1006
> URL: https://issues.apache.org/jira/browse/KAFKA-1006
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Swapnil Ghike
> Assignee: Guozhang Wang
> Labels: usability
>
> Consumer currently uses auto.offset.reset = largest by default. If a new
> topic is created, consumer's topic watcher is fired. The consumer will first
> finish partition reassignment as part of rebalance and then start consuming
> from the tail of each partition. Until the partition reassignment is over,
> the server may have appended new messages to the new topic, consumer won't
> consume these messages. Thus, multiple batches of messages may be lost when a
> topic is newly created.
> The fix is to start consuming from the earliest offset for newly created
> topics.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
