I run PE for (almost*) all the releases. I can add YCSB if you want. It's a dedicated cluster where I have only Hadoop 2.2.0 running, ZK (on the master) and HBase. Nothing else. Only 3RS+1MASTER for now. Just send me the YCSB workload (as long as it doesn't require to run for more than 24h).
JM *: I sometime don't hve the bandwidth to test 2 releases at a time but try and wil continue to try to test all the releases. 2014-05-11 3:27 GMT-04:00 Andrew Purtell <[email protected]>: > For cost and time reasons, and for regression checking operation latencies > not drag racing peak throughput. I wanted YCSB because many try HBase out > with it, I find that YCSB is difficult to use to say the least in a > coordinated way with multiple instances. > > For benchmarking throughput I would use a completely different methodology > involving many test clients. > > > On May 11, 2014, at 12:29 PM, Konstantin Boudnik <[email protected]> wrote: > > > > I think the point wasn't a benchmarking but merely to make sure there's > no > > regressions. > > > >> On Wed, May 07, 2014 at 02:16PM, Vladimir Rodionov wrote: > >> *7x EC2 c3.8xlarge: 1 master, 5 slaves, 1 test client* > >> > >> Andrew, I think these numbers are far from maximum you can get from this > >> set up. Why only 1 test client? > >> > >> -Vladimir Rodionov > >> > >> > >>> On Tue, May 6, 2014 at 6:58 PM, Andrew Purtell <[email protected]> > wrote: > >>> > >>> Comparing the relative performance of 0.98.2 RC0 and 0.98.0 on Hadoop > 2.2.0 > >>> using YCSB. > >>> > >>> The hardware used is different than for the previous report comparing > >>> 0.98.1 to 0.98.0. However the results are very similar, both in terms > of > >>> 0.98.2 RC0 numbers with respect to those measured for 0.98.0, and the > >>> workload specific deltas observed when testing 0.98.1. > >>> > >>> *Hardware and Versions* > >>> > >>> Hadoop 2.2.0 > >>> HBase 0.98.2-hadoop2 RC0 > >>> > >>> 7x EC2 c3.8xlarge: 1 master, 5 slaves, 1 test client > >>> > >>> 32 cores > >>> > >>> 60 GB RAM > >>> > >>> 2 x 320 GB directly attached SSD > >>> > >>> NameNode: 4 GB heap > >>> > >>> DataNode: 1 GB heap > >>> > >>> Master: 1 GB heap > >>> > >>> RegionServer: 8 GB heap, 24 GB bucket cache offheap engine > >>> > >>> > >>> *Methodology* > >>> > >>> > >>> Setup: > >>> > >>> 0. Start cluster > >>> 1. shell: create "seed", { NAME=>"u", COMPRESSION=>"snappy" } > >>> 2. YCSB: Preload 100 million rows into table "seed" > >>> 3. shell: flush "seed" ; compact "seed" > >>> 4. Wait for compaction to complete > >>> 5. shell: create_snapshot "seed", "seed_snap" > >>> 6. shell: disable "seed" > >>> > >>> > >>> For each test: > >>> > >>> 7. shell: clone_snapshot "seed_snap", "test" > >>> 8. YCSB: Run test -p operationcount=10000000 -threads 32 -target > >>> 50000 (clamp at ~10k ops/server/sec) > >>> 9. shell: disable "test" > >>> 10. shell: drop "test" > >>> > >>> > >>> *Workload A* > >>> > >>> *0.98.0* > >>> > >>> [OVERALL], RunTime(ms), 2097825 > >>> [OVERALL], Throughput(ops/sec), 4767 > >>> [UPDATE], Operations, 4999049 > >>> [UPDATE], AverageLatency(us), 1.107036384 > >>> [UPDATE], MinLatency(us), 0 > >>> [UPDATE], MaxLatency(us), 97865 > >>> [UPDATE], 95thPercentileLatency(ms), 0 > >>> [UPDATE], 99thPercentileLatency(ms), 0 > >>> [READ], Operations, 5000952 > >>> [READ], AverageLatency(us), 413.9172277 > >>> [READ], MinLatency(us), 295 > >>> [READ], MaxLatency(us), 927729 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 0 > >>> > >>> > >>> *0.98.2* > >>> > >>> [OVERALL], RunTime(ms), 2082682 > >>> [OVERALL], Throughput(ops/sec), 4802 > >>> [UPDATE], Operations, 5001208 > >>> [UPDATE], AverageLatency(us), 1.227632714 > >>> [UPDATE], MinLatency(us), 0 > >>> [UPDATE], MaxLatency(us), 720423 > >>> [UPDATE], 95thPercentileLatency(ms), 0 > >>> [UPDATE], 99thPercentileLatency(ms), 0 > >>> [READ], Operations, 4998792.667 > >>> [READ], AverageLatency(us), 411.0522393 > >>> [READ], MinLatency(us), 288 > >>> [READ], MaxLatency(us), 977500 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 0 > >>> > >>> > >>> > >>> *Workload B* > >>> > >>> *0.98.0* > >>> > >>> [OVERALL], RunTime(ms), 3678408 > >>> [OVERALL], Throughput(ops/sec), 2719 > >>> [UPDATE], Operations, 500239 > >>> [UPDATE], AverageLatency(us), 2.218397098 > >>> [UPDATE], MinLatency(us), 0 > >>> [UPDATE], MaxLatency(us), 101523 > >>> [UPDATE], 95thPercentileLatency(ms), 0 > >>> [UPDATE], 99thPercentileLatency(ms), 0 > >>> [READ], Operations, 9499762.333 > >>> [READ], AverageLatency(us), 384.8231468 > >>> [READ], MinLatency(us), 283 > >>> [READ], MaxLatency(us), 922395 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 0 > >>> > >>> *0.98.2* > >>> > >>> [OVERALL], RunTime(ms), 3643856 > >>> [OVERALL], Throughput(ops/sec), 2744 > >>> [UPDATE], Operations, 499256 > >>> [UPDATE], AverageLatency(us), 2.561636579 > >>> [UPDATE], MinLatency(us), 0 > >>> [UPDATE], MaxLatency(us), 713811 > >>> [UPDATE], 95thPercentileLatency(ms), 0 > >>> [UPDATE], 99thPercentileLatency(ms), 0 > >>> [READ], Operations, 9500745 > >>> [READ], AverageLatency(us), 381.1349225 > >>> [READ], MinLatency(us), 284 > >>> [READ], MaxLatency(us), 921680 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 0 > >>> > >>> > >>> *Workload C* > >>> > >>> *0.98.0* > >>> > >>> [OVERALL], RunTime(ms), 3258845 > >>> [OVERALL], Throughput(ops/sec), 3069 > >>> [READ], Operations, 10000000 > >>> [READ], AverageLatency(us), 323.7287128 > >>> [READ], MinLatency(us), 276 > >>> [READ], MaxLatency(us), 928472 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 0 > >>> > >>> *0.98.2* > >>> > >>> [OVERALL], RunTime(ms), 3288822 > >>> [OVERALL], Throughput(ops/sec), 3041 > >>> [READ], Operations, 10000000 > >>> [READ], AverageLatency(us), 326.6214268 > >>> [READ], MinLatency(us), 284 > >>> [READ], MaxLatency(us), 924632 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 0 > >>> > >>> *Workload D* > >>> > >>> *0.98.0* > >>> > >>> [OVERALL], RunTime(ms), 3707601 > >>> [OVERALL], Throughput(ops/sec), 2700 > >>> [INSERT], Operations, 500774 > >>> [INSERT], AverageLatency(us), 6.432826519 > >>> [INSERT], MinLatency(us), 4 > >>> [INSERT], MaxLatency(us), 40274 > >>> [INSERT], 95thPercentileLatency(ms), 0 > >>> [INSERT], 99thPercentileLatency(ms), 0 > >>> [READ], Operations, 9499225.667 > >>> [READ], AverageLatency(us), 387.7104498 > >>> [READ], MinLatency(us), 283 > >>> [READ], MaxLatency(us), 927377 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 1 > >>> > >>> *0.98.2* > >>> > >>> [OVERALL], RunTime(ms), 3650724 > >>> [OVERALL], Throughput(ops/sec), 2740 > >>> [INSERT], Operations, 499872 > >>> [INSERT], AverageLatency(us), 6.46417158 > >>> [INSERT], MinLatency(us), 4 > >>> [INSERT], MaxLatency(us), 47732 > >>> [INSERT], 95thPercentileLatency(ms), 0 > >>> [INSERT], 99thPercentileLatency(ms), 0 > >>> [READ], Operations, 9500128 > >>> [READ], AverageLatency(us), 381.6517188 > >>> [READ], MinLatency(us), 278 > >>> [READ], MaxLatency(us), 922107 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 1 > >>> > >>> > >>> *Workload E* > >>> > >>> *0.98.0* > >>> > >>> [OVERALL], RunTime(ms), 15717450.00 > >>> [OVERALL], Throughput(ops/sec), 636.2355217 > >>> [INSERT], Operations, 499943 > >>> [INSERT], AverageLatency(us), 12.57311534 > >>> [INSERT], MinLatency(us), 6 > >>> [INSERT], MaxLatency(us), 39539 > >>> [INSERT], 95thPercentileLatency(ms), 0 > >>> [INSERT], 99thPercentileLatency(ms), 0 > >>> [SCAN], Operations, 9500057 > >>> [SCAN], AverageLatency(us), 1648.836612 > >>> [SCAN], MinLatency(us), 768 > >>> [SCAN], MaxLatency(us), 1001461 > >>> [SCAN], 95thPercentileLatency(ms), 3 > >>> [SCAN], 99thPercentileLatency(ms), 5 > >>> > >>> *0.98.2* > >>> > >>> [OVERALL], RunTime(ms), 15624378 > >>> [OVERALL], Throughput(ops/sec), 640 > >>> [INSERT], Operations, 499679 > >>> [INSERT], AverageLatency(us), 11.62045822 > >>> [INSERT], MinLatency(us), 5 > >>> [INSERT], MaxLatency(us), 40475 > >>> [INSERT], 95thPercentileLatency(ms), 0 > >>> [INSERT], 99thPercentileLatency(ms), 0 > >>> [SCAN], Operations, 9500321 > >>> [SCAN], AverageLatency(us), 1639.114033 > >>> [SCAN], MinLatency(us), 753 > >>> [SCAN], MaxLatency(us), 942908 > >>> [SCAN], 95thPercentileLatency(ms), 3 > >>> [SCAN], 99thPercentileLatency(ms), 5 > >>> > >>> > >>> *Workload F* > >>> > >>> *0.98.0* > >>> > >>> [OVERALL], RunTime(ms), 4144245 > >>> [OVERALL], Throughput(ops/sec), 2413 > >>> [UPDATE], Operations, 4999220 > >>> [UPDATE], AverageLatency(us), 1.520920945 > >>> [UPDATE], MinLatency(us), 0 > >>> [UPDATE], MaxLatency(us), 94874 > >>> [UPDATE], 95thPercentileLatency(ms), 0 > >>> [UPDATE], 99thPercentileLatency(ms), 0 > >>> [READ-MODIFY-WRITE], Operations, 4999219 > >>> [READ-MODIFY-WRITE], AverageLatency(us), 413.7597696 > >>> [READ-MODIFY-WRITE], MinLatency(us), 0 > >>> [READ-MODIFY-WRITE], MaxLatency(us), 107201 > >>> [READ-MODIFY-WRITE], 95thPercentileLatency(ms), 0 > >>> [READ-MODIFY-WRITE], 99thPercentileLatency(ms), 0 > >>> [READ], Operations, 10000000 > >>> [READ], AverageLatency(us), 410.3120381 > >>> [READ], MinLatency(us), 1 > >>> [READ], MaxLatency(us), 921156 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 0 > >>> > >>> *0.98.2* > >>> > >>> [OVERALL], RunTime(ms), 4169551 > >>> [OVERALL], Throughput(ops/sec), 2398 > >>> [UPDATE], Operations, 4999046 > >>> [UPDATE], AverageLatency(us), 1.524197967 > >>> [UPDATE], MinLatency(us), 0 > >>> [UPDATE], MaxLatency(us), 103630 > >>> [UPDATE], 95thPercentileLatency(ms), 0 > >>> [UPDATE], 99thPercentileLatency(ms), 0 > >>> [READ-MODIFY-WRITE], Operations, 4999045 > >>> [READ-MODIFY-WRITE], AverageLatency(us), 416.3519861 > >>> [READ-MODIFY-WRITE], MinLatency(us), 0 > >>> [READ-MODIFY-WRITE], MaxLatency(us), 926671 > >>> [READ-MODIFY-WRITE], 95thPercentileLatency(ms), 0 > >>> [READ-MODIFY-WRITE], 99thPercentileLatency(ms), 0 > >>> [READ], Operations, 10000000 > >>> [READ], AverageLatency(us), 412.8674851 > >>> [READ], MinLatency(us), 1 > >>> [READ], MaxLatency(us), 928123 > >>> [READ], 95thPercentileLatency(ms), 0 > >>> [READ], 99thPercentileLatency(ms), 0 > >>> > >>> > >>> > >>> -- > >>> Best regards, > >>> > >>> - Andy > >>> > >>> Problems worthy of attack prove their worth by hitting back. - Piet > Hein > >>> (via Tom White) > >>> >
