This is true for the high-level ZK-based consumer. Jun
On Thu, Nov 17, 2011 at 10:59 PM, Inder Pall <inder.p...@gmail.com> wrote: > Jun & Taylor, > would it be right to say that consumers without ZK won't be a viable option > if you can't handle replay of old messages in your application. > > - inder > > On Fri, Nov 18, 2011 at 12:27 PM, Jun Rao <jun...@gmail.com> wrote: > > > Taylor, > > > > When you start a consumer, it always tries to get the last checkpointed > > offset from ZK. If no offset can be found in ZK, the consumer starts from > > either the smallest or the largest available offset in the broker. > > > > Thanks, > > > > Jun > > > > On Thu, Nov 17, 2011 at 9:20 PM, Taylor Gautier <tgaut...@tagged.com> > > wrote: > > > > > hmmm - and if you turn off zookeeper? > > > > > > On Thu, Nov 17, 2011 at 9:15 PM, Inder Pall <inder.p...@gmail.com> > > wrote: > > > > > > > The consumer offsets are stored in ZooKeeper by topic and partition. > > > > That's how in a consumer fail over scenario you don't get duplicate > > > > messages > > > > > > > > - Inder > > > > > > > > On Fri, Nov 18, 2011 at 10:33 AM, Taylor Gautier < > tgaut...@tagged.com > > > > >wrote: > > > > > > > > > Hi, > > > > > > > > > > We've noticed that the cleaner script in Kafka removes empty log > > > segments > > > > > but not the directories themselves. I am actually wondering > > something > > > - > > > > I > > > > > always assumed that Kafka could restore the latest offset for > > existing > > > > > topics by scanning the log directory for all directories and > scanning > > > the > > > > > directories for log segment files to restore the latest offset. > > > > > > > > > > Now this conclusion I have made simply by observation - so it could > > be > > > > > entirely wrong. > > > > > > > > > > My question is however - if I am right, and the cleaner removes all > > the > > > > log > > > > > segments for a given topic so that a given topic directory is > empty, > > > how > > > > > does Kafka behave when restarted? How does it know what the next > > > offset > > > > > should be? > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Inder > > > > > > > > > > > > > -- > -- Inder >
