Using Zookeeper to distribute tasks

2010-04-27 Thread David Rouchy
Hi all, We are studying using ZooKeeper to manage configuration across multiple processes server. What would be also interesting, as ZooKeeper know the list of process running, would be to use it to distribute tasks. We have some long running tasks, so we used multiple servers to process

Re: Using Zookeeper to distribute tasks

2010-04-27 Thread Ted Dunning
The general way to do this is either a) have lots of watchers who all try to create a single file when a watched file changes. This is very simple to code, but leads to a lot of notifications when you have thousands of watchers. b) arrange the watchers in a chain. This is similar to the

Re: zkCli.sh missing from zookeeper package in Cloudera CDH contrib repo?

2010-04-27 Thread Henry Robinson
Hi David - As far as I can tell this was not a deliberate omission - earlier versions of the package definitely had zkCli bundled with. Apologies for the oversight. Can you fix by copying zkCli.sh from a 3.2.* tarball from the Apache site? However, these packages are rather old, and we haven't

Re: Using Zookeeper to distribute tasks

2010-04-27 Thread Thomas Koch
David Rouchy: Hi all, We are studying using ZooKeeper to manage configuration across multiple processes server. What would be also interesting, as ZooKeeper know the list of process running, would be to use it to distribute tasks. We have some long running tasks, so we used multiple

Re: zkCli.sh missing from zookeeper package in Cloudera CDH contrib repo?

2010-04-27 Thread David Rosenstrauch
On 04/27/2010 04:10 AM, Thomas Koch wrote: Zookeeper is also in the Debian unstable archive: http://packages.debian.org/sid/zookeeper I'm planning to push an update as soon as ZK 3.3.1 is out. Regards, Thomas Koch, http://www.koch.ro Thanks for the heads up. Unfortunately doesn't help me,

Re: Zookeeper client

2010-04-27 Thread Mahadev Konar
HI Avinash, The zk client does itself maintain liveness information and also randomizes the list of servers to balance the number of clients connected to a single ZooKeeper server. Hope that helps. Thanks mahadev On 4/27/10 10:56 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Let's

Re: Zookeeper client

2010-04-27 Thread Mahadev Konar
Hi Avinash, No, the randomization only happens on the list of servers that are passed to the client. If you just pass A to the client, the client will only be able to connect to A and will not know about the other servers. Does that help? Thanks mahadev On 4/27/10 11:40 AM, Avinash Lakshman

Re: Bizarre ZooKeeper Client Behaviour

2010-04-27 Thread Lei Zhang
Sorry for hijacking this thread. By default the server will only allow 10 sessions to be active from a particular client (ip), perhaps you hit this and it triggered the storm? I am very surprise to hear about this limit. Our system has a number of daemons using zookeeper service, each daemon

Re: Bizarre ZooKeeper Client Behaviour

2010-04-27 Thread Patrick Hunt
Hi Lei, that's a good question -- DOS attack. We had a few cases where clients were creating infinite numbers of sessions and never closing them due to programming bugs. This results in the ZK service maxing out resources (typically file descriptors) and failing. This is very bad from an

Re: Bizarre ZooKeeper Client Behaviour

2010-04-27 Thread Ted Dunning
Lei, A contrary question for you is why you don't just share zk sessions within a single process. On Tue, Apr 27, 2010 at 5:17 PM, Lei Zhang lzvoya...@gmail.com wrote: I am in the process of changing to each thread of each daemon maintaining a zk session. That means we will hit this 10

Re: Bizarre ZooKeeper Client Behaviour

2010-04-27 Thread Lei Zhang
Ted - You can think of this as a problem with using singleton in a multi-threaded program. The solution that provides better code readability at affordable cost should win. Specifically the problem I am trying to solve is this: We have a multi-threaded webapp based on a framework (means I am

Re: Using Zookeeper to distribute tasks

2010-04-27 Thread David Rouchy
Thank you for your quick answers. I'll have a look. David On Tue, Apr 27, 2010 at 10:14 AM, Thomas Koch tho...@koch.ro wrote: David Rouchy: Hi all, We are studying using ZooKeeper to manage configuration across multiple processes server. What would be also interesting, as ZooKeeper