On Thu, Nov 18, 2010 at 3:46 PM, Jeremy Hanna <jeremy.hanna1...@gmail.com> wrote: > Unless I misunderstand, active watches aren't open sessions. If that's the > case, I don't think we'll hit the 10K-20K number of open sessions at a given > time. However, that's a good boundary to keep in mind as we put the system > together.
Right. A session is represented by a "ZooKeeper" object. One session per object. So if you have 10 client hosts each creating it's own ZooKeeper instance you'll have 10 sessions. This is regardless of the number of znodes, watches, etc... Watches were designed to be lightweight and you can maintain a large number of them. (25million spread across 500 sessions in my example) Patrick > > On 11/18/10 2:06 PM, "Fournier, Camille F. [Tech]" <camille.fourn...@gs.com> > wrote: > >> We tested up to the ulimit (~16K) of connections against a single server and >> performance was ok, but I would definitely try to do some serious load >> testing >> before I put a system into production that I knew was going to have that load >> from the get-go. >> The system degrades VERY ungracefully when you hit the ulimit for the >> process, >> so be sure to have enough ensemble nodes to spread those connections across >> that this won't happen. I think maybe there's a JIRA out to deal with this >> issue, not sure what the status is. >> >> C >> >> -----Original Message----- >> From: Patrick Hunt [mailto:ph...@apache.org] >> Sent: Thursday, November 18, 2010 2:57 PM >> To: zookeeper-user@hadoop.apache.org >> Subject: Re: number of clients/watchers >> >> fyi: I haven't heard of anyone running over 10k sessions. I've tried >> 20k before and had issues, you may want to look at this sooner rather >> than later. >> >> * Server gc tuning will be an issue (be sure to use cms/incremental). >> * Be sure to disable clients accessing the leader (server configuration >> param). >> * You may need to use the Observers feature to scale out this large. >> >> Patrick >> >> On Thu, Nov 18, 2010 at 10:31 AM, Jeremy Hanna >> <jeremy.hanna1...@gmail.com> wrote: >>>>> Can you clarify what you mean when you say 10-100K watchers? Do you mean >>>>> 10-100K clients with 1 active watch, or some lesser number of clients with >>>>> more watches, or a few clients doing a lot of watches and other clients >>>>> doing other things? >>> >>> Probably 10-100K clients each with 1 or 2 active watches. The clients will >>> respond to watch events and sometimes initiate actions of their own. >>> >>>> here's a similar test setup I used: >>> >>> Thanks Patrick - it's really nice to have those numbers and test harness >>> basis. >>> >>> We're still in architecture mode so some of the details are still in flux, >>> but I think this gives us an idea. >>> >>> Thanks very much. >>> >>> On Nov 18, 2010, at 11:51 AM, Patrick Hunt wrote: >>> >>>> Camille, that's a very good question. Largest cluster I've heard about >>>> is 10k sessions. >>>> >>>> Jeremy - largest I've ever tested was a 3 server cluster with ~500 >>>> sessions. Each session created 10k znodes (100bytes each znode) and >>>> set 5 watches on each. So 5 million znodes and 25million watches. I >>>> then had the sessions delete the znodes and looked for the >>>> notifications. They were processed by the clients quite quickly (order >>>> of seconds) iirc. Note: this required some GC tuning on the servers to >>>> operate correctly (in particular cms and incremental gc was turned on >>>> and sufficient memory was allocated for the heaps). >>>> >>>> here's a similar test setup I used: >>>> http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview >>>> this is the latency tester tool >>>> https://github.com/phunt/zk-smoketest >>>> >>>> Patrick >>>> >>>> On Thu, Nov 18, 2010 at 9:44 AM, Fournier, Camille F. [Tech] >>>> <camille.fourn...@gs.com> wrote: >>>>> Can you clarify what you mean when you say 10-100K watchers? Do you mean >>>>> 10-100K clients with 1 active watch, or some lesser number of clients with >>>>> more watches, or a few clients doing a lot of watches and other clients >>>>> doing other things? >>>>> >>>>> -----Original Message----- >>>>> From: Jeremy Hanna [mailto:jeremy.hanna1...@gmail.com] >>>>> Sent: Thursday, November 18, 2010 12:15 PM >>>>> To: zookeeper-user@hadoop.apache.org >>>>> Subject: number of clients/watchers >>>>> >>>>> I had a question about number of clients against a zookeeper cluster. I >>>>> was looking at having between 10,000 and 100,000 (towards 100,000) >>>>> watchers >>>>> within a single datacenter at a given time. Assuming that some fraction >>>>> of >>>>> that number are active clients and the r/w ratio is well within the >>>>> zookeeper norms, is that number within the realm of possibility for >>>>> zookeeper? We're going to do testing and benchmarking and things, but I >>>>> didn't want to go down a rabbit hole if this is simply too much for a >>>>> single zookeeper cluster to handle. The numbers I've seen in blog posts >>>>> vary and I saw that the observers feature may be useful in this kind of >>>>> setting. >>>>> >>>>> Maybe I'm underestimating zookeeper or maybe I don't have enough >>>>> information to tell. I'm just trying to see if zookeeper is a good fit >>>>> for >>>>> our use case. >>>>> >>>>> Thanks. >>>>> >>> >>> > > >
