Not sure if related, but this line in AbstractFetcherThread (Kafka 0.8.2 branch) looks suspicious https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L122
This logic seems to have been changed significantly via https://github.com/apache/kafka/commit/79f7cca85e9ed6511ad50cb9412bfa5c8e5b9ddb#diff-2d03a5d4349a23e56c09b4e773055249 for https://issues.apache.org/jira/browse/KAFKA-1461 so in Kafka 0.9 branch it looks like this https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/server/AbstractFetcherThread.scala#L132 Have dirty workaround for the moment, not trusting nextOffset info from MessageAndOffset, transforming response to a new one with custom logic calculating nextOffset (= offset used in fetch request + index of message in message set). Will recheck at latest once 0.9 is fully released. Unwanted behavior seems would get triggered only when consumer lag is of appropriate size - I guess async producer made this issue more likely, because of batch sending multiple messages to same partition. On Wed, Nov 11, 2015 at 9:40 AM, Stevo Slavić <ssla...@gmail.com> wrote: > Hello Apache Kafka community, > > > I'm using simple consumer with Kafka 0.8.2.2 and noticed that under some > conditions fetch response message set for a partition can contain at least > one (if not all) MessageAndOffset with nextOffset being equal to current > (committed) offset, offset used in fetch request. Not sure how it's related > but I could notice this behavior especially often when I was using new > async producer, and when fetch request was able to fetch several messages > all the way to the end of the partition. > > Is this a feature or a bug? > > > Kind regards, > > Stevo Slavic. >
