--autoFlush=true is ignored in PE. Make sure you run your own tests with autoFlush= false in HTable.
-Vlad On Thu, Aug 13, 2015 at 1:22 AM, Serega Sheypak <[email protected]> wrote: > Hi. I used PerformanceEvaluation randomWrite for perf measurement. > > Here are my metrics: > > -- Timers > ---------------------------------------------------------------------- > > .putTimer > > count = 3944591 > > mean rate = 12389.71 calls/second > > 1-minute rate = 8853.79 calls/second > > 5-minute rate = 9800.49 calls/second > > 15-minute rate = 7816.68 calls/second > > min = 1.85 milliseconds > > max = 2524.64 milliseconds > > mean = 17.36 milliseconds > > stddev = 135.73 milliseconds > > median = 6.18 milliseconds > > 75% <= 10.41 milliseconds > > 95% <= 24.38 milliseconds > > 98% <= 32.82 milliseconds > > 99% <= 40.77 milliseconds > > 99.9% <= 2524.64 milliseconds > > > And here are metrics from randomWrite: > > > ## sudo -u hdfs hbase org.apache.hadoop.hbase.PerformanceEvaluation > --nomapred --flushCommits=true --autoFlush=true --oneCon=true --latency > randomWrite 10 > > 15/08/13 00:30:49 INFO hbase.PerformanceEvaluation: Finished TestClient-4 > in 461794ms over 1048576 rows > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: > 9437184/10485700/10485760, latency mean=440.17, min=1.00, max=5525520.00, > stdDev=20817.84, 95th=2.00, 99th=3.00 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest latency > log (microseconds), on 1048576 measures > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Min > = 1.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Avg > = 440.1488666534424 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest StdDev > = 20817.257168459604 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 50th > = 2.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 75th > = 2.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 95th > = 2.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99th > = 3.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99.9th > = 3884.876000001328 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99.99th > = 922522.8667998798 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest > 99.999th = 2311970.1423621834 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Max > = 5525520.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest > valueSize after 0 measures > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Min > = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Avg > = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest StdDev > = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 50th > = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 75th > = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 95th > = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99th > = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99.9th > = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest 99.99th > = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest > 99.999th = 0.0 > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: RandomWriteTest Max > = 0.0 > > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: [RandomWriteTest] > Summary of timings (ms): [458553, 459646, 461004, 458014, 461794, 455945, > 458489, 457333, 461017, 463250] > > 15/08/13 00:30:51 INFO hbase.PerformanceEvaluation: [RandomWriteTest] Min: > 455945ms Max: 463250ms Avg: 459504ms > > So my 99th percentile is 40MS, randowWrite shows 4ms > > What could be the reason for 10 times diff? > My code does random put (Random string as a key), the size of put is > hundreds of bytes. >
