Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "ZooKeeper/ServiceLatencyOverview" page has been changed by PatrickHunt. http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview?action=diff&rev1=16&rev2=17 -------------------------------------------------- * zk-latencies.py from http://github.com/phunt/zk-smoketest was used as the test client. * python 2.4.3 + == Result Summary == + + In the following two graphs you can see the results of this experiment. The X axis shows the number of operations per second that were processed by the server for 1, 2 & 4 cores, where 1, 10 & 20 clients were testing the cluster. + + ||'''Standalone Server'''||'''Ensemble with 3 Serving Nodes'''|| + ||{{attachment:standalone_perf.png|Standalone Server}}||{{attachment:ensemble_3_perf.png|Ensemble with 3 serving nodes}}|| + + Each client is performing 110,000 asynchronous operations against the service, where the workload is: + * create 10k permanent znodes of size 100 bytes + * set 10k znodes of size 100 bytes + * get 10k znodes (100 bytes returned) + * delete all 10k znodes + * create 10k ephemeral znodes of size 100 bytes + * watch each of the 10k znodes 5 times (50k watches) + * delete 10k znodes and measure time for notifications to be received by the client (from the 50k watches) + + Of the 110k operations 50k are write operations, the remaining 60k are read operations. Read operations being local to the connected server while writes must be propagated through the leader & atomic broadcast protocol (ie copied to all servers in an ensemble) + + === Standalone Result === + + From the graph and the data below we can see that moving from a workload of 1 to 10 clients shows an increase in overall operations performed by the server. However moving from 10 to 20 clients we see no further increase (rather some decrease, potentially due to the overhead of selector processing and connection management in the server networking code). Increasing the number of cores provides modest increase in the overall output of the server. This makes sense as there are a limited number of threads running in a standalone server, it is unable to take advantage of the additional cores. + + === Ensemble with 3 Serving Nodes Result === + + Here we see distinct increase in the output of the cluster as we move the workload from 1 to 10 to 20 clients. There is some additional headroom that we could see if we were to test with higher workloads. Additionally as the core count increases we see output gains there as well. Ensemble servers are running additional threads to process the quorum communication, these threads are taking advantage of the additional parallelism. + + === Conclusion === + + I hope this sheds more light on the tradeoffs that can be made when setting up a ZK cluster. In particular we can see the performance of a small cluster (reliable even in the face of a single server going down) is quite solid even under a heavy (30k+ operations per second on dual core with a high write to read ratio, much higher than you should typically see in a production deployment) client workload. Additionally the session stability was rock solid - no timeouts or session expirations were seen during any of the tests. + + === Notes, Futures === + + I used the out of the box zkServer.sh for this test. I did look at using the incremental garbage collector however as far as I could tell there was no benefit seen for these tests. It might be interesting to look at this more, is there some benefit to be had? + + I also did not use the --server flag, again, I did some limited testing with this and afaics there is no benefit to using this option with a ZK cluster. + + It would be interesting to test other workloads. Here I've weighted things more to a balanced write/read workload. In production deployments we typically see a heavy read dominant workload. In this case the service performance should be even better than what we are seeing here. + - == Results == + == Result Details == the results are as follows for standalone and 3 node ensemble
