Jun & Taylor, would it be right to say that consumers without ZK won't be a viable option if you can't handle replay of old messages in your application.
- inder On Fri, Nov 18, 2011 at 12:27 PM, Jun Rao <jun...@gmail.com> wrote: > Taylor, > > When you start a consumer, it always tries to get the last checkpointed > offset from ZK. If no offset can be found in ZK, the consumer starts from > either the smallest or the largest available offset in the broker. > > Thanks, > > Jun > > On Thu, Nov 17, 2011 at 9:20 PM, Taylor Gautier <tgaut...@tagged.com> > wrote: > > > hmmm - and if you turn off zookeeper? > > > > On Thu, Nov 17, 2011 at 9:15 PM, Inder Pall <inder.p...@gmail.com> > wrote: > > > > > The consumer offsets are stored in ZooKeeper by topic and partition. > > > That's how in a consumer fail over scenario you don't get duplicate > > > messages > > > > > > - Inder > > > > > > On Fri, Nov 18, 2011 at 10:33 AM, Taylor Gautier <tgaut...@tagged.com > > > >wrote: > > > > > > > Hi, > > > > > > > > We've noticed that the cleaner script in Kafka removes empty log > > segments > > > > but not the directories themselves. I am actually wondering > something > > - > > > I > > > > always assumed that Kafka could restore the latest offset for > existing > > > > topics by scanning the log directory for all directories and scanning > > the > > > > directories for log segment files to restore the latest offset. > > > > > > > > Now this conclusion I have made simply by observation - so it could > be > > > > entirely wrong. > > > > > > > > My question is however - if I am right, and the cleaner removes all > the > > > log > > > > segments for a given topic so that a given topic directory is empty, > > how > > > > does Kafka behave when restarted? How does it know what the next > > offset > > > > should be? > > > > > > > > > > > > > > > > -- > > > -- Inder > > > > > > -- -- Inder
