-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32802/#review78800
-----------------------------------------------------------

Ship it!



src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java
<https://reviews.apache.org/r/32802/#comment127824>

    Nice! I could not make this work through annotations in earlier versions of 
jmh/gradle plugin.



src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java
<https://reviews.apache.org/r/32802/#comment127844>

    Perhaps just hardcode this in Tasks.Builder() instead and drop the builder 
method? I don't see why we would ever want to change this value.


- Maxim Khutornenko


On April 3, 2015, 2:19 a.m., Bill Farner wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/32802/
> -----------------------------------------------------------
> 
> (Updated April 3, 2015, 2:19 a.m.)
> 
> 
> Review request for Aurora and Maxim Khutornenko.
> 
> 
> Repository: aurora
> 
> 
> Description
> -------
> 
> The first of several benchmarks to exercise the thrift API.  The real 
> motivation here is to have performance data to assess the impact of migrating 
> the task store to H2.  I'm targeting the `getRoleSummary` RPC specifically 
> here as i consider it the likely candidate to suffer.
> 
> One 'trick' i employed here was JSON for input parameters.  I took this 
> approach since i really wanted to explore specific axes of growth without the 
> full outer product of each individual parameters.
> 
> Some changes along the way:
> - upgraded to jmh 1.7.1
> - moved some JMH run configuration parameters out of the gradle config and 
> into benchmark classes.  This allows us to define run configurations for each 
> microbenchmark
> - changed the benchmark mode for existing and new tests to throughput.  We 
> had several benchmarks whose iteration time was measuring in hundreds of 
> microseconds, which is too high resolution to capture performance variation.  
> By running iterations for a set amount of time and counting iterations 
> accomplished, we can capture more reliable data.
> 
> 
> Diffs
> -----
> 
>   build.gradle 66dbdfde830c81c9b2291d2b8391f1ccac94b485 
>   src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java 
> 5309e8140fff411da30ee87c1b3b1a55d6fdaeeb 
>   src/jmh/java/org/apache/aurora/benchmark/Tasks.java 
> 491c6871ba7a4c76a4eb084afb2627a3f93db8df 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> PRE-CREATION 
>   src/main/java/org/apache/aurora/scheduler/thrift/ThriftModule.java 
> c65e0218211b797ad50e2ac62e71136f967e5f5d 
> 
> Diff: https://reviews.apache.org/r/32802/diff/
> 
> 
> Testing
> -------
> 
> FYI numbers and stddev are not useful in the data captured below, as my 
> machine was doing other intensive things while running this.  Output is more 
> useful as an example of how the data and run configurations are presented.
> 
> ```
> # Run complete. Total time: 00:18:51
> 
> Benchmark                                                                     
>  (testConfiguration)   Mode  Cnt       Score       Error  Units
> SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark             
>                  N/A  thrpt   10  499824.589 ± 86328.152  ops/s
> SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark    
>                  N/A  thrpt   10   40482.762 ±  5394.306  ops/s
> SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark  
>                  N/A  thrpt   10    2933.958 ±   577.571  ops/s
> SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark                
>                  N/A  thrpt   10     220.079 ±    28.943  ops/s
> SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark  
>                  N/A  thrpt   10   41948.230 ±  1906.106  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>         {"roles": 1}  thrpt    5    1978.614 ±   736.704  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>        {"roles": 10}  thrpt    5     143.465 ±    85.579  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>       {"roles": 100}  thrpt    5       4.003 ±     2.742  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>       {"roles": 500}  thrpt    5       1.964 ±     0.569  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>          {"jobs": 1}  thrpt    5    4073.564 ±  3343.602  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>         {"jobs": 10}  thrpt    5    4401.662 ±  3943.372  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>        {"jobs": 100}  thrpt    5    4327.726 ±   726.900  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>        {"jobs": 500}  thrpt    5    2473.799 ±  5396.188  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>     {"instances": 1}  thrpt    5    7351.543 ± 13368.279  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>    {"instances": 10}  thrpt    5    9496.478 ±  3066.478  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>   {"instances": 100}  thrpt    5    4640.480 ±  3307.831  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
>  {"instances": 1000}  thrpt    5     429.841 ±   191.521  ops/s
> ThriftApiBenchmarks.GetRoleSummaryBenchmark.run                               
> {"instances": 10000}  thrpt    5      23.327 ±     7.011  ops/s
> ```
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Reply via email to