Re: Using Zookeeper to distribute tasks
The general way to do this is either a) have lots of watchers who all try to create a single file when a watched file changes. This is very simple to code, but leads to a lot of notifications when you have thousands of watchers. b) arrange the watchers in a chain. This is similar to the recommended queue implementation. See http://hadoop.apache.org/zookeeper/docs/r3.2.1/recipes.html#sc_recipes_Queues This requires careful implementation to deal correctly with session expiration. c) do as in (a), but spread the load by picking one of several files to watch. This still has herds jumping on a single file, but you can adjust the number of files being used to keep this at a reasonable level. On Tue, Apr 27, 2010 at 12:23 AM, David Rouchy drou...@gmail.com wrote: My idea would be to use multiple ephemeral watcher connected to one znode. A process will send data to this node when a new task has to be launched. But here is my issue, I would like only one watcher to be triggered of this change (a random watcher in the list). Is there way to do such thing in ZooKeeper?
Re: Using Zookeeper to distribute tasks
David Rouchy: Hi all, We are studying using ZooKeeper to manage configuration across multiple processes server. What would be also interesting, as ZooKeeper know the list of process running, would be to use it to distribute tasks. We have some long running tasks, so we used multiple servers to process multiple tasks in the same time. My idea would be to use multiple ephemeral watcher connected to one znode. A process will send data to this node when a new task has to be launched. But here is my issue, I would like only one watcher to be triggered of this change (a random watcher in the list). Is there way to do such thing in ZooKeeper? Regards, David Rouchy Hi David, there's gearman[1], a client-server system to distribute tasks. I've mentioned it already two times on this list, so sorry for the repetition. Gearman servers and clients are availabe in different implementations that use the same protocol and are interchangeable AFAIK. If somebody would build a gearman server using Zookeeper, there would be a whole ecosystem for ZK for free to conquer! [1] http://gearman.org Regards, Thomas Koch, http://www.koch.ro
Re: Using Zookeeper to distribute tasks
Thank you for your quick answers. I'll have a look. David On Tue, Apr 27, 2010 at 10:14 AM, Thomas Koch tho...@koch.ro wrote: David Rouchy: Hi all, We are studying using ZooKeeper to manage configuration across multiple processes server. What would be also interesting, as ZooKeeper know the list of process running, would be to use it to distribute tasks. We have some long running tasks, so we used multiple servers to process multiple tasks in the same time. My idea would be to use multiple ephemeral watcher connected to one znode. A process will send data to this node when a new task has to be launched. But here is my issue, I would like only one watcher to be triggered of this change (a random watcher in the list). Is there way to do such thing in ZooKeeper? Regards, David Rouchy Hi David, there's gearman[1], a client-server system to distribute tasks. I've mentioned it already two times on this list, so sorry for the repetition. Gearman servers and clients are availabe in different implementations that use the same protocol and are interchangeable AFAIK. If somebody would build a gearman server using Zookeeper, there would be a whole ecosystem for ZK for free to conquer! [1] http://gearman.org Regards, Thomas Koch, http://www.koch.ro
