-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32802/
-----------------------------------------------------------

(Updated April 3, 2015, 2:19 a.m.)


Review request for Aurora and Maxim Khutornenko.


Repository: aurora


Description
-------

The first of several benchmarks to exercise the thrift API.  The real 
motivation here is to have performance data to assess the impact of migrating 
the task store to H2.  I'm targeting the `getRoleSummary` RPC specifically here 
as i consider it the likely candidate to suffer.

One 'trick' i employed here was JSON for input parameters.  I took this 
approach since i really wanted to explore specific axes of growth without the 
full outer product of each individual parameters.

Some changes along the way:
- upgraded to jmh 1.7.1
- moved some JMH run configuration parameters out of the gradle config and into 
benchmark classes.  This allows us to define run configurations for each 
microbenchmark
- changed the benchmark mode for existing and new tests to throughput.  We had 
several benchmarks whose iteration time was measuring in hundreds of 
microseconds, which is too high resolution to capture performance variation.  
By running iterations for a set amount of time and counting iterations 
accomplished, we can capture more reliable data.


Diffs
-----

  build.gradle 66dbdfde830c81c9b2291d2b8391f1ccac94b485 
  src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
5309e8140fff411da30ee87c1b3b1a55d6fdaeeb 
  src/jmh/java/org/apache/aurora/benchmark/Tasks.java 
491c6871ba7a4c76a4eb084afb2627a3f93db8df 
  src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
PRE-CREATION 
  src/main/java/org/apache/aurora/scheduler/thrift/ThriftModule.java 
c65e0218211b797ad50e2ac62e71136f967e5f5d 

Diff: https://reviews.apache.org/r/32802/diff/


Testing (updated)
-------

FYI numbers and stddev are not useful in the data captured below, as my machine 
was doing other intensive things while running this.  Output is more useful as 
an example of how the data and run configurations are presented.

```
# Run complete. Total time: 00:18:51

Benchmark                                                                      
(testConfiguration)   Mode  Cnt       Score       Error  Units
SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark               
               N/A  thrpt   10  499824.589 ± 86328.152  ops/s
SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark      
               N/A  thrpt   10   40482.762 ±  5394.306  ops/s
SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark    
               N/A  thrpt   10    2933.958 ±   577.571  ops/s
SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                  
               N/A  thrpt   10     220.079 ±    28.943  ops/s
SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark    
               N/A  thrpt   10   41948.230 ±  1906.106  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
      {"roles": 1}  thrpt    5    1978.614 ±   736.704  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
     {"roles": 10}  thrpt    5     143.465 ±    85.579  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
    {"roles": 100}  thrpt    5       4.003 ±     2.742  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
    {"roles": 500}  thrpt    5       1.964 ±     0.569  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
       {"jobs": 1}  thrpt    5    4073.564 ±  3343.602  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
      {"jobs": 10}  thrpt    5    4401.662 ±  3943.372  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
     {"jobs": 100}  thrpt    5    4327.726 ±   726.900  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
     {"jobs": 500}  thrpt    5    2473.799 ±  5396.188  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
  {"instances": 1}  thrpt    5    7351.543 ± 13368.279  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
 {"instances": 10}  thrpt    5    9496.478 ±  3066.478  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                 
{"instances": 100}  thrpt    5    4640.480 ±  3307.831  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                                
{"instances": 1000}  thrpt    5     429.841 ±   191.521  ops/s
ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
{"instances": 10000}  thrpt    5      23.327 ±     7.011  ops/s
```


Thanks,

Bill Farner

Reply via email to