autooffset.reset="largest" in conjunction with deleting the corresponding zookeeper offsets solved the problem. Thanks to both for sharing this excellent advice.
On Mon, Jan 23, 2012 at 8:12 PM, Joel Koshy <jjko...@gmail.com> wrote: > Largest works - however, autooffset only applies if the consumer group > is new (non-existent in ZK) or if the stored offset is out of range. > If you restart your consumer at a later point it will resume from the > last check-pointed offset. So from the scenario you describe (down for > a period of time...) it appears that it wouldn't be enough for you. > Depending on the details of your set up, one thing you could do is to > clean up the zookeeper entries for your consumer group before bringing > up your consumer service again. > > Joel > > On Mon, Jan 23, 2012 at 4:49 PM, Evan Chan <e...@ooyala.com> wrote: > > Michael, > > > > It seems there is a property "autooffset.reset" here: > > http://incubator.apache.org/kafka/configuration.html > > > > which you can set to "largest". I haven't used it though so I can't be > > sure. > > > > -Evan > > > > > > On Mon, Jan 23, 2012 at 4:12 PM, Michael Luban <michael.lu...@gmail.com > >wrote: > > > >> I'm interested in having my consumer service start consuming the most > >> recent message rather than the message corresponding to the last offset > >> stored. For instance, if the kafka consumer service is down for a > period > >> of time, I would like to skip over all messages since the last recorded > >> offset and begin consuming from the most recent message. > >> > >> According to the following, it seems I have to use SimpleConsumer to > >> achieve this? Seems like this would be difficult to manage in a "load > >> balanced" consumer scenario (federation of consumers in the same group > >> consuming the same topic). > >> > >> > https://groups.google.com/group/kafka-dev/browse_thread/thread/ee7fe5f8eef49a4d/a4469233880a7f76?lnk=gst&q=offset#a4469233880a7f76 > >> > >> Any additional insight appreciated. > >> > > > > > > > > -- > > -- > > *Evan Chan* > > Senior Software Engineer | > > e...@ooyala.com | (650) 996-4600 > > www.ooyala.com | blog <http://www.ooyala.com/blog> | > > @ooyala<http://www.twitter.com/ooyala> >
