Hello Damian, Yes, there's a +1 difference. See related discussion http://mail-archives.apache.org/mod_mbox/kafka-users/201507.mbox/%3CCAOeJiJh2SMzVn23JsoWiNk3sfsw82Jr_-kRLcNRd-oZ7pR1yWg%40mail.gmail.com%3E
Kind regards, Stevo Slavic. On Tue, Sep 15, 2015 at 3:56 PM, Damian Guy <[email protected]> wrote: > I turned off compression and still get duplicates, but only 1 from each > topic. > Should the initial fetch offset for a partition be committed offset +1 ? > > Thanks, > Damian > > On 15 September 2015 at 14:07, Damian Guy <[email protected]> wrote: > > > Hi, > > > > I've been trying out the new consumer and have noticed that i get > > duplicate messages when i stop the consumer and then restart (different > > processes, same consumer group). > > > > I consume all of the messages on the topic and commit the offsets for > each > > partition and stop the consumer. On the next run i expect to get 0 > > messages, however i get a batch of records from each partition - in this > > case works out 1020 messages. Run it again and i get the same batch of > > records. > > > > My logging shows that i've received messages with offsets lower than were > > previously committed. > > > > committed:{damian_test_one-2=137669} > > committed:{damian_test_one-0=139161} > > committed:{damian_test_one-1=137663} > > > > min offsets received: {damian_test_one-0=138824, > damian_test_one-1=137321, > > damian_test_one-2=137331} > > > > I've debugged the initial fetch requests for offsets and the offsets > match > > up with what has been committed. Is this expected behaviour? Something to > > do with batching of compression of message sets? > > > > TIA, > > Damian > > >
