The consumer offsets are stored in ZooKeeper by topic and partition. That's how in a consumer fail over scenario you don't get duplicate messages
- Inder On Fri, Nov 18, 2011 at 10:33 AM, Taylor Gautier <tgaut...@tagged.com>wrote: > Hi, > > We've noticed that the cleaner script in Kafka removes empty log segments > but not the directories themselves. I am actually wondering something - I > always assumed that Kafka could restore the latest offset for existing > topics by scanning the log directory for all directories and scanning the > directories for log segment files to restore the latest offset. > > Now this conclusion I have made simply by observation - so it could be > entirely wrong. > > My question is however - if I am right, and the cleaner removes all the log > segments for a given topic so that a given topic directory is empty, how > does Kafka behave when restarted? How does it know what the next offset > should be? > -- -- Inder
