Camille, that's a very good question. Largest cluster I've heard about is 10k sessions.
Jeremy - largest I've ever tested was a 3 server cluster with ~500 sessions. Each session created 10k znodes (100bytes each znode) and set 5 watches on each. So 5 million znodes and 25million watches. I then had the sessions delete the znodes and looked for the notifications. They were processed by the clients quite quickly (order of seconds) iirc. Note: this required some GC tuning on the servers to operate correctly (in particular cms and incremental gc was turned on and sufficient memory was allocated for the heaps). here's a similar test setup I used: http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview this is the latency tester tool https://github.com/phunt/zk-smoketest Patrick On Thu, Nov 18, 2010 at 9:44 AM, Fournier, Camille F. [Tech] <camille.fourn...@gs.com> wrote: > Can you clarify what you mean when you say 10-100K watchers? Do you mean > 10-100K clients with 1 active watch, or some lesser number of clients with > more watches, or a few clients doing a lot of watches and other clients doing > other things? > > -----Original Message----- > From: Jeremy Hanna [mailto:jeremy.hanna1...@gmail.com] > Sent: Thursday, November 18, 2010 12:15 PM > To: zookeeper-user@hadoop.apache.org > Subject: number of clients/watchers > > I had a question about number of clients against a zookeeper cluster. I was > looking at having between 10,000 and 100,000 (towards 100,000) watchers > within a single datacenter at a given time. Assuming that some fraction of > that number are active clients and the r/w ratio is well within the zookeeper > norms, is that number within the realm of possibility for zookeeper? We're > going to do testing and benchmarking and things, but I didn't want to go down > a rabbit hole if this is simply too much for a single zookeeper cluster to > handle. The numbers I've seen in blog posts vary and I saw that the > observers feature may be useful in this kind of setting. > > Maybe I'm underestimating zookeeper or maybe I don't have enough information > to tell. I'm just trying to see if zookeeper is a good fit for our use case. > > Thanks. >
