Re: Using Zookeeper to distribute tasks
That's the beauty of zk, it's provides primitives that allow you much flexibility in what/how you implement something. Perhaps you could sketch out a bit more detail on what you had in mind, requirements/goals and such? What I mean is, from a high level coordination perspective, then we could see how that might map down to zk operations. Patrick On 05/19/2010 03:29 AM, David Rouchy wrote: Hi all, I'm studying ZooKeeper to see how we can use it in a project. I'm still a little big puzzled by the usage of zookeeper for distributing tasks. The usage of queue does not fully fit the bill. I was just wondering if it would be interesting / realistic to use a kind of "election" to decide on which znode you want to send the task? David Rouchy On Wed, Apr 28, 2010 at 7:39 AM, David Rouchy wrote: Thank you for your quick answers. I'll have a look. David On Tue, Apr 27, 2010 at 10:14 AM, Thomas Koch wrote: David Rouchy: Hi all, We are studying using ZooKeeper to manage configuration across multiple processes& server. What would be also interesting, as ZooKeeper know the list of process running, would be to use it to distribute tasks. We have some long running tasks, so we used multiple servers to process multiple tasks in the same time. My idea would be to use multiple ephemeral watcher connected to one znode. A process will send data to this node when a new task has to be launched. But here is my issue, I would like only one watcher to be triggered of this change (a random watcher in the list). Is there way to do such thing in ZooKeeper? Regards, David Rouchy Hi David, there's gearman[1], a client-server system to distribute tasks. I've mentioned it already two times on this list, so sorry for the repetition. Gearman servers and clients are availabe in different implementations that use the same protocol and are interchangeable AFAIK. If somebody would build a gearman server using Zookeeper, there would be a whole ecosystem for ZK for free to conquer! [1] http://gearman.org Regards, Thomas Koch, http://www.koch.ro
Re: Using Zookeeper to distribute tasks
Hi all, I'm studying ZooKeeper to see how we can use it in a project. I'm still a little big puzzled by the usage of zookeeper for distributing tasks. The usage of queue does not fully fit the bill. I was just wondering if it would be interesting / realistic to use a kind of "election" to decide on which znode you want to send the task? David Rouchy On Wed, Apr 28, 2010 at 7:39 AM, David Rouchy wrote: > Thank you for your quick answers. I'll have a look. > > David > > > On Tue, Apr 27, 2010 at 10:14 AM, Thomas Koch wrote: > >> David Rouchy: >> > Hi all, >> > >> > We are studying using ZooKeeper to manage configuration across multiple >> > processes & server. What would be also interesting, as ZooKeeper know >> the >> > list of process running, would be to use it to distribute tasks. >> > >> > We have some long running tasks, so we used multiple servers to process >> > multiple tasks in the same time. My idea would be to use multiple >> ephemeral >> > watcher connected to one znode. A process will send data to this node >> when >> > a new task has to be launched. But here is my issue, I would like only >> one >> > watcher to be triggered of this change (a random watcher in the list). >> Is >> > there way to do such thing in ZooKeeper? >> > >> > Regards, >> > >> > David Rouchy >> >> Hi David, >> >> there's gearman[1], a client-server system to distribute tasks. I've >> mentioned >> it already two times on this list, so sorry for the repetition. Gearman >> servers and clients are availabe in different implementations that use the >> same protocol and are interchangeable AFAIK. >> If somebody would build a gearman server using Zookeeper, there would be a >> whole ecosystem for ZK for free to conquer! >> >> [1] http://gearman.org >> >> Regards, >> >> Thomas Koch, http://www.koch.ro >> > >
Re: Using Zookeeper to distribute tasks
Thank you for your quick answers. I'll have a look. David On Tue, Apr 27, 2010 at 10:14 AM, Thomas Koch wrote: > David Rouchy: > > Hi all, > > > > We are studying using ZooKeeper to manage configuration across multiple > > processes & server. What would be also interesting, as ZooKeeper know the > > list of process running, would be to use it to distribute tasks. > > > > We have some long running tasks, so we used multiple servers to process > > multiple tasks in the same time. My idea would be to use multiple > ephemeral > > watcher connected to one znode. A process will send data to this node > when > > a new task has to be launched. But here is my issue, I would like only > one > > watcher to be triggered of this change (a random watcher in the list). > Is > > there way to do such thing in ZooKeeper? > > > > Regards, > > > > David Rouchy > > Hi David, > > there's gearman[1], a client-server system to distribute tasks. I've > mentioned > it already two times on this list, so sorry for the repetition. Gearman > servers and clients are availabe in different implementations that use the > same protocol and are interchangeable AFAIK. > If somebody would build a gearman server using Zookeeper, there would be a > whole ecosystem for ZK for free to conquer! > > [1] http://gearman.org > > Regards, > > Thomas Koch, http://www.koch.ro >
Re: Using Zookeeper to distribute tasks
David Rouchy: > Hi all, > > We are studying using ZooKeeper to manage configuration across multiple > processes & server. What would be also interesting, as ZooKeeper know the > list of process running, would be to use it to distribute tasks. > > We have some long running tasks, so we used multiple servers to process > multiple tasks in the same time. My idea would be to use multiple ephemeral > watcher connected to one znode. A process will send data to this node when > a new task has to be launched. But here is my issue, I would like only one > watcher to be triggered of this change (a random watcher in the list). Is > there way to do such thing in ZooKeeper? > > Regards, > > David Rouchy Hi David, there's gearman[1], a client-server system to distribute tasks. I've mentioned it already two times on this list, so sorry for the repetition. Gearman servers and clients are availabe in different implementations that use the same protocol and are interchangeable AFAIK. If somebody would build a gearman server using Zookeeper, there would be a whole ecosystem for ZK for free to conquer! [1] http://gearman.org Regards, Thomas Koch, http://www.koch.ro
Re: Using Zookeeper to distribute tasks
The general way to do this is either a) have lots of watchers who all try to create a single file when a watched file changes. This is very simple to code, but leads to a lot of notifications when you have thousands of watchers. b) arrange the watchers in a chain. This is similar to the recommended queue implementation. See http://hadoop.apache.org/zookeeper/docs/r3.2.1/recipes.html#sc_recipes_Queues This requires careful implementation to deal correctly with session expiration. c) do as in (a), but spread the load by picking one of several files to watch. This still has herds jumping on a single file, but you can adjust the number of files being used to keep this at a reasonable level. On Tue, Apr 27, 2010 at 12:23 AM, David Rouchy wrote: > My idea would be to use multiple ephemeral > watcher connected to one znode. A process will send data to this node when > a > new task has to be launched. But here is my issue, I would like only one > watcher to be triggered of this change (a random watcher in the list). Is > there way to do such thing in ZooKeeper? >
Using Zookeeper to distribute tasks
Hi all, We are studying using ZooKeeper to manage configuration across multiple processes & server. What would be also interesting, as ZooKeeper know the list of process running, would be to use it to distribute tasks. We have some long running tasks, so we used multiple servers to process multiple tasks in the same time. My idea would be to use multiple ephemeral watcher connected to one znode. A process will send data to this node when a new task has to be launched. But here is my issue, I would like only one watcher to be triggered of this change (a random watcher in the list). Is there way to do such thing in ZooKeeper? Regards, David Rouchy
