Re: Not performing work in the zookeeper even thread.
On Mon, Jan 5, 2009 at 2:16 PM, Benjamin Reed wrote: > I'm not sure what your Watcher:ZooKeeper notation means, Sorry... just that each ZooKeeper instance has an associate watcher. > but just to be clear: each ZooKeeper object has an event thread with a > corresponding event queue. Whenever zookeeper needs to do a callback it > queues the callback to the event thread to handle the invocation. > Thanks. -- Founder/CEO Spinn3r.com Location: San Francisco, CA AIM/YIM: sfburtonator Skype: burtonator Work: http://spinn3r.com
RE: Not performing work in the zookeeper even thread.
I'm not sure what your Watcher:ZooKeeper notation means, but just to be clear: each ZooKeeper object has an event thread with a corresponding event queue. Whenever zookeeper needs to do a callback it queues the callback to the event thread to handle the invocation. ben From: burtona...@gmail.com [burtona...@gmail.com] On Behalf Of Kevin Burton [bur...@spinn3r.com] Sent: Monday, January 05, 2009 2:08 PM To: zookeeper-user@hadoop.apache.org Subject: Re: Not performing work in the zookeeper even thread. On Mon, Jan 5, 2009 at 1:23 PM, Benjamin Reed wrote: > the event thread is the client's own thread. zookeeper queues events to the > event thread so that it doesn't have to worry about a backlog. > So each Watcher:ZooKeeper shares one thread (I was thinking you might be doing this)? Wouldn't this be problematic when pooling ZooKeeper objects across threads? I implemented a new API which I think is a bit easier to use than the ZK API (I'll publish the source in a bit). I had it implement a watchNode() method which registers an event. Then your thread calls poll() which handles all events from ZK... the API then blocks and waits for ZK to add events to its own internal queue. This way you can have a fully event driven application but you don't have to worry about object corruption or threading issues. > > even if an application is slow in processing callbacks from the event > thread, the rest of zookeeper (the heartbeats, the synchronous calls, etc.) > will not be adversely affected. > OK... so even in multithreaded applications if you block you can't kill ZK (which would be bad). Kevin -- Founder/CEO Spinn3r.com Location: San Francisco, CA AIM/YIM: sfburtonator Skype: burtonator Work: http://spinn3r.com
Re: Not performing work in the zookeeper even thread.
On Mon, Jan 5, 2009 at 1:23 PM, Benjamin Reed wrote: > the event thread is the client's own thread. zookeeper queues events to the > event thread so that it doesn't have to worry about a backlog. > So each Watcher:ZooKeeper shares one thread (I was thinking you might be doing this)? Wouldn't this be problematic when pooling ZooKeeper objects across threads? I implemented a new API which I think is a bit easier to use than the ZK API (I'll publish the source in a bit). I had it implement a watchNode() method which registers an event. Then your thread calls poll() which handles all events from ZK... the API then blocks and waits for ZK to add events to its own internal queue. This way you can have a fully event driven application but you don't have to worry about object corruption or threading issues. > > even if an application is slow in processing callbacks from the event > thread, the rest of zookeeper (the heartbeats, the synchronous calls, etc.) > will not be adversely affected. > OK... so even in multithreaded applications if you block you can't kill ZK (which would be bad). Kevin -- Founder/CEO Spinn3r.com Location: San Francisco, CA AIM/YIM: sfburtonator Skype: burtonator Work: http://spinn3r.com
RE: Not performing work in the zookeeper even thread.
the event thread is the client's own thread. zookeeper queues events to the event thread so that it doesn't have to worry about a backlog. even if an application is slow in processing callbacks from the event thread, the rest of zookeeper (the heartbeats, the synchronous calls, etc.) will not be adversely affected. ben From: burtona...@gmail.com [burtona...@gmail.com] On Behalf Of Kevin Burton [bur...@spinn3r.com] Sent: Sunday, January 04, 2009 12:23 PM To: zookeeper-user@hadoop.apache.org Subject: Not performing work in the zookeeper even thread. It looks like events from Zookeeper are dispatches from its event thread: java.lang.Exception at foo.zookeeper.WatcherImpl.process(NodeWatcher.java:54) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:349) ... it seems like a good idea to have implementations perform work in their own thread to avoid zookeeper from hitting backlog if one client decides to perform some complex operation in one event when there are hundreds of events pending delivery. Kevin -- Founder/CEO Spinn3r.com Location: San Francisco, CA AIM/YIM: sfburtonator Skype: burtonator Work: http://spinn3r.com
Re: Not performing work in the zookeeper even thread.
HI Kevin, We have a single threaded c client in which the application needs to call the zookeeper_process to process the events. What we noticed is that all the users who are using it, have problems using it and would like to get rid of it (only if they we not running on bsd). So, the events being dispacthed by the event thread makes it easeir for the users to use zookeeper. We do assume that they do minimum work in the process method. We sould probably document this (if its not yet)... The huge backlog is a problem for the clients and they should be aware that doing any complex operations in the process method is asking for trouble. Hope this helps. mahadev On 1/4/09 12:23 PM, "Kevin Burton" wrote: > It looks like events from Zookeeper are dispatches from its event thread: > java.lang.Exception > at foo.zookeeper.WatcherImpl.process(NodeWatcher.java:54) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:349) > > ... it seems like a good idea to have implementations perform work in their > own thread to avoid zookeeper from hitting backlog if one client decides to > perform some complex operation in one event when there are hundreds of > events pending delivery. > > Kevin