Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "ZooKeeper/ServiceLatencyOverview" page has been changed by PatrickHunt. http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview?action=diff&rev1=4&rev2=5 -------------------------------------------------- == ZooKeeper service latencies under various loads & configurations == - The intent of this document is to test the performance of various loads on a ZooKeeper service. + The intent of this document is to examine the performance a ZooKeeper service under various loads. This is not an exhaustive test but meant to shed some light on the how much load can be supported for various hardware configurations. In particular I was interested in seeing how non-dedicated servers performed while varying the number of cpus and clients. === Hardware === + All systems were : + - All systems were dual quad core Intel(R) Xeon(R) CPUs running at 2.50GHz. Eight cores were available however as noted below Linux's processor affinity feature was used to limit CPU availability to the JVM. + dual quad core Intel(R) Xeon(R) CPUs running at 2.50GHz. Eight cores were available however as noted below Linux's processor affinity feature was used to limit CPU availability to the JVM. All systems had 16gig memory available, however unless specifically noted the JVM's -Xmx option was used to limit the size of the JVM heap to 512m. + + All systems had 7200RPM SATA drives. === Operating System === Linux version 2.6.18-53.1.13.el5 compiled using gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) + === Software === + + * ZooKeeper 3.2.1 + * java version "1.6.0_05" (build 1.6.0_05-b13) + * Java HotSpot(TM) Server VM (build 10.0-b19, mixed mode) + + === ZooKeeper Configuration === The following basic configuration was used on all serving nodes: + {{{ tickTime=2000 initLimit=10 syncLimit=5 dataDir=<data directory> clientPort=2181 + }}} + A dedicated log device was not used, the data directory is store on the same device as the operating system, the ZK snapshots and log4j logs. The standard zkServer.sh script that ships with the release was used to start the service. No modification were made other than to control processor affinity and heap size as noted in this document. === Test Client === - zk-latencies.py from http://github.com/phunt/zk-smoketest was used as the test client. + * zk-latencies.py from http://github.com/phunt/zk-smoketest was used as the test client. + * python 2.4.3 - - The client was run as follows unless otherwise noted: - - {{{ - zk-latencies.py --servers \"host60:2181\" --root_znode=/`hostname` --znode_size=100 --znode_count=10000 --timeout=5000 --watch_multiple=5 - }}} - - This means each client: - * has a 5 second timeout - * creates 10k znodes of size 100 bytes - * sets 5 watches on each of the znodes it creates, for 50k znodes by each client == Results == - the results are as follows for standalone and 3 node ensemble - - == Results for Standalone Server == - - === 1 Core - 1 Client === - - 110k operations against the server with elapsed time of 18 seconds - - {{{ - host40: Testing latencies on server host60:2181 using asynchronous calls - host40: created 10000 permanent znodes in 1219 ms (0.121911 ms/op 8202.684317/sec) - host40: set 10000 znodes in 1600 ms (0.160094 ms/op 6246.310789/sec) - host40: get 10000 znodes in 1744 ms (0.174442 ms/op 5732.564236/sec) - host40: deleted 10000 permanent znodes in 1386 ms (0.138648 ms/op 7212.503931/sec) - host40: created 10000 ephemeral znodes in 1608 ms (0.160836 ms/op 6217.525272/sec) - host40: watched 50000 znodes in 7509 ms (0.150180 ms/op 6658.670068/sec) - host40: deleted 10000 ephemeral znodes in 2139 ms (0.213949 ms/op 4674.011338/sec) - host40: notif 50000 watches in 0 ms (included in prior) - host40: Latency test complete - host40: 15.20user 7.06system 0:17.74elapsed 125%CPU (0avgtext+0avgdata 0maxresident)k - host40: 0inputs+0outputs (0major+71244minor)pagefaults 0swaps - }}} - - === 1 Core - 10 Client === - - 1.1 million operations against the server with elapsed time of 58 seconds - - {{{ - host46: Testing latencies on server host60:2181 using asynchronous calls - host46: created 10000 permanent znodes in 6493 ms (0.649358 ms/op 1539.982995/sec) - host46: set 10000 znodes in 4538 ms (0.453877 ms/op 2203.239641/sec) - host46: get 10000 znodes in 13229 ms (1.322971 ms/op 755.874244/sec) - host46: deleted 10000 permanent znodes in 4967 ms (0.496763 ms/op 2013.030379/sec) - host46: created 10000 ephemeral znodes in 5196 ms (0.519624 ms/op 1924.468834/sec) - host46: watched 50000 znodes in 17421 ms (0.348424 ms/op 2870.067528/sec) - host46: deleted 10000 ephemeral znodes in 4901 ms (0.490112 ms/op 2040.351571/sec) - host46: notif 50000 watches in 0 ms (included in prior) - host46: Latency test complete - host46: 15.94user 6.40system 0:57.40elapsed 38%CPU (0avgtext+0avgdata 0maxresident)k - host46: 0inputs+0outputs (2major+69981minor)pagefaults 0swaps - }}} - - === 1 Core - 20 Client === - - 2.2 million operations against the server with elapsed time of 150 seconds - - {{{ - host67: Testing latencies on server host60:2181 using asynchronous calls - host67: created 10000 permanent znodes in 12881 ms (1.288103 ms/op 776.335485/sec) - host67: set 10000 znodes in 10448 ms (1.044807 ms/op 957.114388/sec) - host67: get 10000 znodes in 14859 ms (1.485911 ms/op 672.988006/sec) - host67: deleted 10000 permanent znodes in 17420 ms (1.742049 ms/op 574.036736/sec) - host67: created 10000 ephemeral znodes in 13450 ms (1.345097 ms/op 743.440587/sec) - host67: watched 50000 znodes in 53507 ms (1.070143 ms/op 934.454518/sec) - host67: deleted 10000 ephemeral znodes in 26196 ms (2.619615 ms/op 381.735542/sec) - host67: notif 50000 watches in 0 ms (included in prior) - host67: Latency test complete - host67: 16.16user 6.43system 2:29.38elapsed 15%CPU (0avgtext+0avgdata 0maxresident)k - host67: 0inputs+0outputs (2major+73149minor)pagefaults 0swaps - }}} - - === 2 Core - 1 Client === -
