Re: Using Zookeeper to distribute tasks
Thank you for your quick answers. I'll have a look. David On Tue, Apr 27, 2010 at 10:14 AM, Thomas Koch wrote: > David Rouchy: > > Hi all, > > > > We are studying using ZooKeeper to manage configuration across multiple > > processes & server. What would be also interesting, as ZooKeeper know the > > list of process running, would be to use it to distribute tasks. > > > > We have some long running tasks, so we used multiple servers to process > > multiple tasks in the same time. My idea would be to use multiple > ephemeral > > watcher connected to one znode. A process will send data to this node > when > > a new task has to be launched. But here is my issue, I would like only > one > > watcher to be triggered of this change (a random watcher in the list). > Is > > there way to do such thing in ZooKeeper? > > > > Regards, > > > > David Rouchy > > Hi David, > > there's gearman[1], a client-server system to distribute tasks. I've > mentioned > it already two times on this list, so sorry for the repetition. Gearman > servers and clients are availabe in different implementations that use the > same protocol and are interchangeable AFAIK. > If somebody would build a gearman server using Zookeeper, there would be a > whole ecosystem for ZK for free to conquer! > > [1] http://gearman.org > > Regards, > > Thomas Koch, http://www.koch.ro >
Re: Bizarre ZooKeeper Client Behaviour
Ted - You can think of this as a problem with using singleton in a multi-threaded program. The solution that provides better code readability at affordable cost should win. Specifically the problem I am trying to solve is this: We have a multi-threaded webapp based on a framework (means I am not yet able to find out how the framework spawns threads, cleaning up the threads). Each thread has local states that must be cleaned up on zookeeper fatal error; and the desired zookeeper error recovery is to reopen the session. The choices on the table are: 1. 1 zk session for the whole webapp 2. 1 zk session for each thread Option 1 proves to be tricky to implement - since more than 1 threads can catch zk fatal error, I would have to implement proper synchronization mechanism to prevent same session being closed/reopened rapidly back to back; besides, I need to implement a way to request the threads that did not catch zk fatal error to clean up their local states. Option 2 I haven't prototyped out completely. It is likely the webapp may become laden with zk threads. I suspect we'll run into zk threads race condition. I'd like to hear from people who actually struggled with similar design decision. Lei On Tue, Apr 27, 2010 at 7:12 PM, Ted Dunning wrote: > Lei, > > A contrary question for you is why you don't just share zk sessions within > a > single process. > >
Re: Bizarre ZooKeeper Client Behaviour
Lei, A contrary question for you is why you don't just share zk sessions within a single process. On Tue, Apr 27, 2010 at 5:17 PM, Lei Zhang wrote: > I am > in the process of changing to each thread of each daemon maintaining a zk > session. That means we will hit this 10 session limit very easily. > > Patrick - what is the rationale for this 10 session limit? >
Re: Bizarre ZooKeeper Client Behaviour
Hi Lei, that's a good question -- DOS attack. We had a few cases where clients were creating infinite numbers of sessions and never closing them due to programming bugs. This results in the ZK service maxing out resources (typically file descriptors) and failing. This is very bad from an availability perspective so to protect against this we added a limit. The good thing is that this limit was made configurable, so you can set it however you like: See "maxClientCnxns" here: http://bit.ly/9nAowJ Patrick On 04/27/2010 05:17 PM, Lei Zhang wrote: Sorry for hijacking this thread. By default the server will only allow 10 sessions to be active from a particular client (ip), perhaps you hit this and it triggered the storm? I am very surprise to hear about this limit. Our system has a number of daemons using zookeeper service, each daemon maintaining a zk session. I am in the process of changing to each thread of each daemon maintaining a zk session. That means we will hit this 10 session limit very easily. Patrick - what is the rationale for this 10 session limit? Thanks, Lei
Re: Bizarre ZooKeeper Client Behaviour
Sorry for hijacking this thread. By default the server will only allow 10 sessions to be active from a > particular client (ip), perhaps you hit this and it triggered the storm? > > I am very surprise to hear about this limit. Our system has a number of daemons using zookeeper service, each daemon maintaining a zk session. I am in the process of changing to each thread of each daemon maintaining a zk session. That means we will hit this 10 session limit very easily. Patrick - what is the rationale for this 10 session limit? Thanks, Lei
Re: Bizarre ZooKeeper Client Behaviour
Are you sure you're closing the ZooKeeper object before creating a new one? I see names like: "task-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread-EventThread" this most likely means you are creating ZK sessions in your watcher. This is fine, but you just need to make sure you close the old one. Perhaps some corner case in your code got triggered and it started creating ZK sessions recursively w/o bound? By default the server will only allow 10 sessions to be active from a particular client (ip), perhaps you hit this and it triggered the storm? Patrick On 04/27/2010 03:52 PM, Satish Bhatti wrote: One of our processes that uses ZooKeeper was behaving oddly so I ran a jstack on it. It is attached to this email. Note the "EventThread" lines. Over 6000 of them. Has it _really_ spun up 6000 event threads? Satish
Re: Zookeeper client
Hi Avinash, No, the randomization only happens on the list of servers that are passed to the client. If you just pass A to the client, the client will only be able to connect to A and will not know about the other servers. Does that help? Thanks mahadev On 4/27/10 11:40 AM, "Avinash Lakshman" wrote: > Thanks Mahadev. Does this happen irrespective of what I provide in the > connect string? Let's say I have servers A, B, C, D and E in the ZK cluster. > But all my clients instantiate ZooKeeper instances by providing information > only about A. Does the randomization occur even in this case? > > Thanks > Avinash > > On Tue, Apr 27, 2010 at 11:00 AM, Mahadev Konar wrote: > >> HI Avinash, >> The zk client does itself maintain liveness information and also >> randomizes the list of servers to balance the number of clients connected >> to >> a single ZooKeeper server. >> >> Hope that helps. >> >> Thanks >> mahadev >> >> >> On 4/27/10 10:56 AM, "Avinash Lakshman" >> wrote: >> >>> Let's assume I have 100 clients connecting to a cluster of 5 Zookeeper >>> servers over time. On the client side I instantiate a ZooKeeper instance >> and >>> use it whenever I need to read/write into ZK. Now I know I can pass in a >>> connect string with the list of all servers that make up the ZK cluster. >>> Does the ZK client automatically maintain liveness information and load >>> balance my connections across the machines? How can I do this >> effectively? I >>> basically want to spread the connections from the 100 clients to the 5 ZK >>> instances effectively. >>> >>> Thanks >>> Avinash >> >>
Re: Zookeeper client
Thanks Mahadev. Does this happen irrespective of what I provide in the connect string? Let's say I have servers A, B, C, D and E in the ZK cluster. But all my clients instantiate ZooKeeper instances by providing information only about A. Does the randomization occur even in this case? Thanks Avinash On Tue, Apr 27, 2010 at 11:00 AM, Mahadev Konar wrote: > HI Avinash, > The zk client does itself maintain liveness information and also > randomizes the list of servers to balance the number of clients connected > to > a single ZooKeeper server. > > Hope that helps. > > Thanks > mahadev > > > On 4/27/10 10:56 AM, "Avinash Lakshman" > wrote: > > > Let's assume I have 100 clients connecting to a cluster of 5 Zookeeper > > servers over time. On the client side I instantiate a ZooKeeper instance > and > > use it whenever I need to read/write into ZK. Now I know I can pass in a > > connect string with the list of all servers that make up the ZK cluster. > > Does the ZK client automatically maintain liveness information and load > > balance my connections across the machines? How can I do this > effectively? I > > basically want to spread the connections from the 100 clients to the 5 ZK > > instances effectively. > > > > Thanks > > Avinash > >
Re: Zookeeper client
HI Avinash, The zk client does itself maintain liveness information and also randomizes the list of servers to balance the number of clients connected to a single ZooKeeper server. Hope that helps. Thanks mahadev On 4/27/10 10:56 AM, "Avinash Lakshman" wrote: > Let's assume I have 100 clients connecting to a cluster of 5 Zookeeper > servers over time. On the client side I instantiate a ZooKeeper instance and > use it whenever I need to read/write into ZK. Now I know I can pass in a > connect string with the list of all servers that make up the ZK cluster. > Does the ZK client automatically maintain liveness information and load > balance my connections across the machines? How can I do this effectively? I > basically want to spread the connections from the 100 clients to the 5 ZK > instances effectively. > > Thanks > Avinash
Zookeeper client
Let's assume I have 100 clients connecting to a cluster of 5 Zookeeper servers over time. On the client side I instantiate a ZooKeeper instance and use it whenever I need to read/write into ZK. Now I know I can pass in a connect string with the list of all servers that make up the ZK cluster. Does the ZK client automatically maintain liveness information and load balance my connections across the machines? How can I do this effectively? I basically want to spread the connections from the 100 clients to the 5 ZK instances effectively. Thanks Avinash
Re: zkCli.sh missing from zookeeper package in Cloudera CDH contrib repo?
Hmmm ... contrib repo definitely is still available. I just used it the other day. (See: http://archive.cloudera.com/redhat/cdh/ ) Bummer about the missing zkCli script. I'll have to figure some way around it. Perhaps it might even make sense to downgrade to an earlier version of the package in the repo. I'd rather not install it from a tarball, though. I much prefer using a package manager. As far as adding ZK as a first-class package in the CDH, yes I've heard it's coming. But it's not here yet, and unfortunately I need to get a ZK instance up and running ASAP. Is there an anticipated release date for ZK in the CDH? I can't speak for anyone else, but I'd be happy if you guys released something ASAP - even if it was explicitly an alpha or beta quality release - rather than having to wait. Thanks much for all the hard work you guys put into the CDH! Best, DR On 04/27/2010 03:39 AM, Henry Robinson wrote: Hi David - As far as I can tell this was not a deliberate omission - earlier versions of the package definitely had zkCli bundled with. Apologies for the oversight. Can you fix by copying zkCli.sh from a 3.2.* tarball from the Apache site? However, these packages are rather old, and we haven't updated them in a while. In fact, I believe the contrib repo is no longer available. We at Cloudera will be releasing first-class packages for ZooKeeper with the next beta release of our distribution, CDH3 Beta 2. For more details see http://www.cloudera.com/blog/2010/03/cdh3-beta1-now-available/. cheers, Henry On 26 April 2010 14:05, David Rosenstrauch wrote: I installed the zookeeper package (hbase-0.20-zookeeper) from the Cloudera CDH contrib repo. But neither it nor the hbase-0.20 package that it's dependent on seem to supply the zkCli command line utility. Anyone know if this is intentional and/or how to fix? Thanks, DR
Re: zkCli.sh missing from zookeeper package in Cloudera CDH contrib repo?
On 04/27/2010 04:10 AM, Thomas Koch wrote: Zookeeper is also in the Debian unstable archive: http://packages.debian.org/sid/zookeeper I'm planning to push an update as soon as ZK 3.3.1 is out. Regards, Thomas Koch, http://www.koch.ro Thanks for the heads up. Unfortunately doesn't help me, though, as we're running CentOS. Thanks, DR
Re: ZooKeeper gets three Google Summer of Code students
3 GSOC projects approved, that's great! Thank you students for applying for a project, should be some interesting new functionality/features developed over the summer. Thanks to all the mentors for stepping up and volunteering their time and Henry for shepherding the gsoc process (as well as being a mentor). I encourage our user base to engage with the students if they have interest in a particular project. If you have ideas, problems, or suggestions for features please discuss with the student (on the list is great). Exposing the mentees to how ZK based software is used in real life production environments will allow them to build a better product. Regards, Patrick On Mon, Apr 26, 2010 at 4:55 PM, Henry Robinson wrote: > Hi - > > Just wanted to announce to the community that we are lucky to have three > talented students working on Google's Summer of Code projects directly > related to ZooKeeper. > > Andrei Savu will be working with Patrick Hunt on a Web-based > Administrative > Interface, extending and improving Patrick's Django-based front end. > Abmar Barros will be working with Flavio Junqueira on improving ZooKeeper's > failure detector module - making the code cleaner and easier to try out new > implementations, as well as implementing a few failure detection algorithms > himself! > Finally, Sergey Doroshenko will be working with me on a Read-Only Mode for > ZooKeeper, which will help bolster ZK's availability in certain > circumstances when a network partition is detected, as well as potentially > optimising the read-path. > > (The full list of 450 GSoC students is here: > http://socghop.appspot.com/gsoc/program/list_projects/google/gsoc2010) > > Congratulations to all three - we look forward to seeing what you produce > over the summer. Thanks to everyone who applied, suggested projects and > offered to mentor students; this program will have a big effect on > ZooKeeper's visibility and community, as well as hopefully producing some > great code! > > cheers, > Henry > > -- > Henry Robinson > Software Engineer > Cloudera > 415-994-6679 >
Re: Using Zookeeper to distribute tasks
David Rouchy: > Hi all, > > We are studying using ZooKeeper to manage configuration across multiple > processes & server. What would be also interesting, as ZooKeeper know the > list of process running, would be to use it to distribute tasks. > > We have some long running tasks, so we used multiple servers to process > multiple tasks in the same time. My idea would be to use multiple ephemeral > watcher connected to one znode. A process will send data to this node when > a new task has to be launched. But here is my issue, I would like only one > watcher to be triggered of this change (a random watcher in the list). Is > there way to do such thing in ZooKeeper? > > Regards, > > David Rouchy Hi David, there's gearman[1], a client-server system to distribute tasks. I've mentioned it already two times on this list, so sorry for the repetition. Gearman servers and clients are availabe in different implementations that use the same protocol and are interchangeable AFAIK. If somebody would build a gearman server using Zookeeper, there would be a whole ecosystem for ZK for free to conquer! [1] http://gearman.org Regards, Thomas Koch, http://www.koch.ro
Re: zkCli.sh missing from zookeeper package in Cloudera CDH contrib repo?
Zookeeper is also in the Debian unstable archive: http://packages.debian.org/sid/zookeeper I'm planning to push an update as soon as ZK 3.3.1 is out. Regards, Thomas Koch, http://www.koch.ro
Re: zkCli.sh missing from zookeeper package in Cloudera CDH contrib repo?
Hi David - As far as I can tell this was not a deliberate omission - earlier versions of the package definitely had zkCli bundled with. Apologies for the oversight. Can you fix by copying zkCli.sh from a 3.2.* tarball from the Apache site? However, these packages are rather old, and we haven't updated them in a while. In fact, I believe the contrib repo is no longer available. We at Cloudera will be releasing first-class packages for ZooKeeper with the next beta release of our distribution, CDH3 Beta 2. For more details see http://www.cloudera.com/blog/2010/03/cdh3-beta1-now-available/. cheers, Henry On 26 April 2010 14:05, David Rosenstrauch wrote: > I installed the zookeeper package (hbase-0.20-zookeeper) from the Cloudera > CDH contrib repo. But neither it nor the hbase-0.20 package that it's > dependent on seem to supply the zkCli command line utility. > > Anyone know if this is intentional and/or how to fix? > > Thanks, > > DR > -- Henry Robinson Software Engineer Cloudera 415-994-6679
Re: Using Zookeeper to distribute tasks
The general way to do this is either a) have lots of watchers who all try to create a single file when a watched file changes. This is very simple to code, but leads to a lot of notifications when you have thousands of watchers. b) arrange the watchers in a chain. This is similar to the recommended queue implementation. See http://hadoop.apache.org/zookeeper/docs/r3.2.1/recipes.html#sc_recipes_Queues This requires careful implementation to deal correctly with session expiration. c) do as in (a), but spread the load by picking one of several files to watch. This still has herds jumping on a single file, but you can adjust the number of files being used to keep this at a reasonable level. On Tue, Apr 27, 2010 at 12:23 AM, David Rouchy wrote: > My idea would be to use multiple ephemeral > watcher connected to one znode. A process will send data to this node when > a > new task has to be launched. But here is my issue, I would like only one > watcher to be triggered of this change (a random watcher in the list). Is > there way to do such thing in ZooKeeper? >
Using Zookeeper to distribute tasks
Hi all, We are studying using ZooKeeper to manage configuration across multiple processes & server. What would be also interesting, as ZooKeeper know the list of process running, would be to use it to distribute tasks. We have some long running tasks, so we used multiple servers to process multiple tasks in the same time. My idea would be to use multiple ephemeral watcher connected to one znode. A process will send data to this node when a new task has to be launched. But here is my issue, I would like only one watcher to be triggered of this change (a random watcher in the list). Is there way to do such thing in ZooKeeper? Regards, David Rouchy
