Navneet, In the design document, this is described fairly well. Basically, it is up to the consumers to keep track of where they are in the consumption of each topic/partition. The supplied consumers use zookeeper for this. Check out the 2nd paragraph named "Consumer State" in http://incubator.apache.org/kafka/design.html
Ed On Tue, May 8, 2012 at 11:38 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > Navneet, Zookeeper is a persistent storage system. If you delete its > data and are not running in distributed mode it will start-over, > right? I think the answer is not to delete the Zookeeper data, right? > > -Jay > > On Tue, May 8, 2012 at 1:35 AM, navneet sharma > <navneetsharma0...@gmail.com> wrote: >> Hi Neha, >> >> if i delete the zookeper folder, then i can replicate this problem. >> Actually, that folder was getting created in "/tmp" folder and it was >> getting deleted each day. Now i changed the location and i am seeing that >> the exception is gone. >> >> Basically, i was just checking if i restart the consumer, say after 2-3 >> days, where it will start consuming from. My assumption was that this >> information should be part of broker/server but looks like its part of >> zookeeper and thats why i observed these exceptions. >> >> Attaching logs for your reference. >> >> Thanks, >> Navneet Sharma >> >> >> On Tue, May 8, 2012 at 12:06 AM, Neha Narkhede <neha.narkh...@gmail.com> >> wrote: >>> >>> >> Is it an expected behaviour? >>> >>> Not really. That exception means that the Zookeeper server your client is >>> connected to, is trying to send a response to the client's "ping", but >>> failed, since probably the client disconnected from the server already. >>> I'm not sure I understand what led to this. Could you attach the entire >>> kafka + zookeeper log here ? >>> >>> Thanks, >>> Neha >>> >>> >>> On Mon, May 7, 2012 at 8:18 AM, Jun Rao <jun...@gmail.com> wrote: >>> >>> > ZK stores critical data such as topics/partitions on each broker, in >>> > addition to consumer offsets. So ZK data needs to be protected. >>> > >>> > Thanks, >>> > >>> > Jun >>> > >>> > On Mon, May 7, 2012 at 4:21 AM, navneet sharma >>> > <navneetsharma0...@gmail.com>wrote: >>> > >>> > > Hi, >>> > > >>> > > I observed that if zookeeper folder is deleted, consumer is trying to >>> > read >>> > > the messages it read in previous runs. And there are lots of >>> > > exceptions >>> > > seen on zookeeper console as well:: >>> > > >>> > > java.nio.channels.CancelledKeyException >>> > > at >>> > > sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55) >>> > > at >>> > > sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59) >>> > > at >>> > > >>> > > >>> > >>> > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418) >>> > > at >>> > > >>> > > >>> > >>> > org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509) >>> > > at >>> > > >>> > > >>> > >>> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171) >>> > > at >>> > > >>> > > >>> > >>> > org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:135) >>> > > [2012-05-07 15:57:35,569] ERROR Unexpected Exception: >>> > > (org.apache.zookeeper.server.NIOServerCnxn) >>> > > >>> > > >>> > > Is it an expected behaviour? Is zookeeper directory need to be >>> > > protected >>> > to >>> > > avoid consumer falling off on offsets? >>> > > >>> > > Thanks, >>> > > Navneet Sharma >>> > > >>> > >> >>
