----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55284/ -----------------------------------------------------------
Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, and Santhosh Kumar Shanmugham. Repository: aurora Description ------- # Adding SLA benchmarks benchmarks This RB is related to [AURORA-1854](https://issues.apache.org/jira/browse/AURORA-1854). The goal is to add JMH benchmarks for SlaCalculation and then other metric calculators. I am not sure if the results are very resonable. For instance see that the ops/sec is 54.701±19.743, which seems to have a high variance. Diffs ----- src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 5d8d5bd8f705770979f284d26d2e932aabe707e5 Diff: https://reviews.apache.org/r/55284/diff/ Testing ------- Following is the benchmarks collectcted: # rmotamedi@tw-mbp-rmotamedi:~/workspace/aurora on git:jmh-sla ?? [14:39:56] ? ./gradlew jmh -Pbenchmarks='SlaAlgorithmBenchmark' the content of <./aurora/dist/reports/jmh/human.txt> ``` # JMH 1.15 (released 98 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA # Parameters: (numTasks = 100) # Run progress: 0.00% complete, ETA 00:05:15 # Fork: 1 of 1 # Warmup Iteration 1: 127519.626 ops/s Iteration 1: 152258.072 ops/s Iteration 2: 170404.745 ops/s Iteration 3: 138346.324 ops/s Iteration 4: 150597.403 ops/s Iteration 5: 169640.531 ops/s Result "measureMTTA": 156249.415 ±(99.9%) 52660.004 ops/s [Average] (min, avg, max) = (138346.324, 156249.415, 170404.745), stdev = 13675.636 CI (99.9%): [103589.411, 208909.419] (assumes normal distribution) # JMH 1.15 (released 98 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA # Parameters: (numTasks = 1000) # Run progress: 11.11% complete, ETA 00:04:45 # Fork: 1 of 1 # Warmup Iteration 1: 4329.734 ops/s Iteration 1: 4115.153 ops/s Iteration 2: 3873.048 ops/s Iteration 3: 3479.972 ops/s Iteration 4: 3225.995 ops/s Iteration 5: 3002.072 ops/s Result "measureMTTA": 3539.248 ±(99.9%) 1757.523 ops/s [Average] (min, avg, max) = (3002.072, 3539.248, 4115.153), stdev = 456.423 CI (99.9%): [1781.725, 5296.771] (assumes normal distribution) # JMH 1.15 (released 98 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA # Parameters: (numTasks = 10000) # Run progress: 22.22% complete, ETA 00:04:10 # Fork: 1 of 1 # Warmup Iteration 1: 44.147 ops/s Iteration 1: 46.116 ops/s Iteration 2: 54.255 ops/s Iteration 3: 57.967 ops/s Iteration 4: 56.174 ops/s Iteration 5: 58.992 ops/s Result "measureMTTA": 54.701 ±(99.9%) 19.743 ops/s [Average] (min, avg, max) = (46.116, 54.701, 58.992), stdev = 5.127 CI (99.9%): [34.957, 74.444] (assumes normal distribution) # JMH 1.15 (released 98 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR # Parameters: (numTasks = 100) # Run progress: 33.33% complete, ETA 00:03:38 # Fork: 1 of 1 # Warmup Iteration 1: 106633.191 ops/s Iteration 1: 136861.727 ops/s Iteration 2: 120490.231 ops/s Iteration 3: 106383.718 ops/s Iteration 4: 126247.774 ops/s Iteration 5: 131648.970 ops/s Result "measureMTTR": 124326.484 ±(99.9%) 45198.021 ops/s [Average] (min, avg, max) = (106383.718, 124326.484, 136861.727), stdev = 11737.783 CI (99.9%): [79128.463, 169524.505] (assumes normal distribution) # JMH 1.15 (released 98 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR # Parameters: (numTasks = 1000) # Run progress: 44.44% complete, ETA 00:03:01 # Fork: 1 of 1 # Warmup Iteration 1: 4012.815 ops/s Iteration 1: 4396.495 ops/s Iteration 2: 4086.905 ops/s Iteration 3: 4024.187 ops/s Iteration 4: 3809.202 ops/s Iteration 5: 3551.905 ops/s Result "measureMTTR": 3973.739 ±(99.9%) 1216.296 ops/s [Average] (min, avg, max) = (3551.905, 3973.739, 4396.495), stdev = 315.868 CI (99.9%): [2757.443, 5190.034] (assumes normal distribution) # JMH 1.15 (released 98 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR # Parameters: (numTasks = 10000) # Run progress: 55.56% complete, ETA 00:02:25 # Fork: 1 of 1 # Warmup Iteration 1: 50.933 ops/s Iteration 1: 57.153 ops/s Iteration 2: 58.741 ops/s Iteration 3: 56.398 ops/s Iteration 4: 55.078 ops/s Iteration 5: 58.142 ops/s Result "measureMTTR": 57.102 ±(99.9%) 5.567 ops/s [Average] (min, avg, max) = (55.078, 57.102, 58.741), stdev = 1.446 CI (99.9%): [51.535, 62.670] (assumes normal distribution) # JMH 1.15 (released 98 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS # Parameters: (numTasks = 100) # Run progress: 66.67% complete, ETA 00:01:49 # Fork: 1 of 1 # Warmup Iteration 1: 107724.308 ops/s Iteration 1: 121787.913 ops/s Iteration 2: 119594.939 ops/s Iteration 3: 128441.840 ops/s Iteration 4: 134176.538 ops/s Iteration 5: 144275.262 ops/s Result "measureMTTS": 129655.298 ±(99.9%) 38446.901 ops/s [Average] (min, avg, max) = (119594.939, 129655.298, 144275.262), stdev = 9984.538 CI (99.9%): [91208.397, 168102.199] (assumes normal distribution) # JMH 1.15 (released 98 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS # Parameters: (numTasks = 1000) # Run progress: 77.78% complete, ETA 00:01:12 # Fork: 1 of 1 # Warmup Iteration 1: 3738.823 ops/s Iteration 1: 4026.955 ops/s Iteration 2: 3070.155 ops/s Iteration 3: 2706.577 ops/s Iteration 4: 3510.103 ops/s Iteration 5: 4063.217 ops/s Result "measureMTTS": 3475.401 ±(99.9%) 2283.158 ops/s [Average] (min, avg, max) = (2706.577, 3475.401, 4063.217), stdev = 592.929 CI (99.9%): [1192.244, 5758.559] (assumes normal distribution) # JMH 1.15 (released 98 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS # Parameters: (numTasks = 10000) # Run progress: 88.89% complete, ETA 00:00:36 # Fork: 1 of 1 # Warmup Iteration 1: 56.489 ops/s Iteration 1: 59.288 ops/s Iteration 2: 59.160 ops/s Iteration 3: 56.142 ops/s Iteration 4: 53.336 ops/s Iteration 5: 53.147 ops/s Result "measureMTTS": 56.215 ±(99.9%) 11.524 ops/s [Average] (min, avg, max) = (53.147, 56.215, 59.288), stdev = 2.993 CI (99.9%): [44.691, 67.738] (assumes normal distribution) # Run complete. Total time: 00:05:27 Benchmark (numTasks) Mode Cnt Score Error Units SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 100 thrpt 5 156249.415 ± 52660.004 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 1000 thrpt 5 3539.248 ± 1757.523 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 10000 thrpt 5 54.701 ± 19.743 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 100 thrpt 5 124326.484 ± 45198.021 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 1000 thrpt 5 3973.739 ± 1216.296 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 10000 thrpt 5 57.102 ± 5.567 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 100 thrpt 5 129655.298 ± 38446.901 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 1000 thrpt 5 3475.401 ± 2283.158 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 10000 thrpt 5 56.215 ± 11.524 ops/s Benchmark result is saved to dist/reports/jmh/results.txt ``` Thanks, Reza Motamedi
