Taylor, When you start a consumer, it always tries to get the last checkpointed offset from ZK. If no offset can be found in ZK, the consumer starts from either the smallest or the largest available offset in the broker.
Thanks, Jun On Thu, Nov 17, 2011 at 9:20 PM, Taylor Gautier <tgaut...@tagged.com> wrote: > hmmm - and if you turn off zookeeper? > > On Thu, Nov 17, 2011 at 9:15 PM, Inder Pall <inder.p...@gmail.com> wrote: > > > The consumer offsets are stored in ZooKeeper by topic and partition. > > That's how in a consumer fail over scenario you don't get duplicate > > messages > > > > - Inder > > > > On Fri, Nov 18, 2011 at 10:33 AM, Taylor Gautier <tgaut...@tagged.com > > >wrote: > > > > > Hi, > > > > > > We've noticed that the cleaner script in Kafka removes empty log > segments > > > but not the directories themselves. I am actually wondering something > - > > I > > > always assumed that Kafka could restore the latest offset for existing > > > topics by scanning the log directory for all directories and scanning > the > > > directories for log segment files to restore the latest offset. > > > > > > Now this conclusion I have made simply by observation - so it could be > > > entirely wrong. > > > > > > My question is however - if I am right, and the cleaner removes all the > > log > > > segments for a given topic so that a given topic directory is empty, > how > > > does Kafka behave when restarted? How does it know what the next > offset > > > should be? > > > > > > > > > > > -- > > -- Inder > > >
