----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55284/ -----------------------------------------------------------
(Updated Jan. 10, 2017, 9 a.m.) Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, and Santhosh Kumar Shanmugham. Changes ------- - partially refactored Task Builder to accept a list of TaskEvents. - updated the number of tasks to show a mathc a more reasonable load. ``` Benchmark (numTasks) Mode Cnt Score Error Units SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 1000 thrpt 5 5540.905 ± 244.059 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 10000 thrpt 5 63.778 ± 5.053 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 100000 thrpt 5 0.546 ± 0.159 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 200000 thrpt 5 0.137 ± 0.049 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 1000 thrpt 5 3556.555 ± 2569.449 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 10000 thrpt 5 54.058 ± 18.486 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 100000 thrpt 5 0.460 ± 0.238 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 200000 thrpt 5 0.127 ± 0.025 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 1000 thrpt 5 3243.565 ± 6417.109 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 10000 thrpt 5 55.796 ± 13.886 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 100000 thrpt 5 0.528 ± 0.071 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 200000 thrpt 5 0.115 ± 0.039 ops/s ``` Repository: aurora Description (updated) ------- # Adding SLA benchmarks benchmarks This RB is related to [AURORA-1854](https://issues.apache.org/jira/browse/AURORA-1854). The goal is to add JMH benchmarks for SlaCalculation and then other metric calculators. Diffs (updated) ----- src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Tasks.java 3aabc5ef222a97d8ebde693c52030a66bca340d2 src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 5d8d5bd8f705770979f284d26d2e932aabe707e5 Diff: https://reviews.apache.org/r/55284/diff/ Testing (updated) ------- the content of <./aurora/dist/reports/jmh/human.txt> ``` # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA # Parameters: (numTasks = 1000) # Run progress: 0.00% complete, ETA 00:07:00 # Fork: 1 of 1 # Warmup Iteration 1: 5141.403 ops/s Iteration 1: 5514.591 ops/s Iteration 2: 5508.875 ops/s Iteration 3: 5594.224 ops/s Iteration 4: 5467.605 ops/s Iteration 5: 5619.229 ops/s Result "measureMTTA": 5540.905 ±(99.9%) 244.059 ops/s [Average] (min, avg, max) = (5467.605, 5540.905, 5619.229), stdev = 63.381 CI (99.9%): [5296.846, 5784.963] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA # Parameters: (numTasks = 10000) # Run progress: 8.33% complete, ETA 00:06:34 # Fork: 1 of 1 # Warmup Iteration 1: 64.993 ops/s Iteration 1: 61.762 ops/s Iteration 2: 64.468 ops/s Iteration 3: 65.047 ops/s Iteration 4: 63.199 ops/s Iteration 5: 64.412 ops/s Result "measureMTTA": 63.778 ±(99.9%) 5.053 ops/s [Average] (min, avg, max) = (61.762, 63.778, 65.047), stdev = 1.312 CI (99.9%): [58.725, 68.830] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA # Parameters: (numTasks = 100000) # Run progress: 16.67% complete, ETA 00:06:00 # Fork: 1 of 1 # Warmup Iteration 1: 0.490 ops/s Iteration 1: 0.556 ops/s Iteration 2: 0.503 ops/s Iteration 3: 0.502 ops/s Iteration 4: 0.576 ops/s Iteration 5: 0.591 ops/s Result "measureMTTA": 0.546 ±(99.9%) 0.159 ops/s [Average] (min, avg, max) = (0.502, 0.546, 0.591), stdev = 0.041 CI (99.9%): [0.387, 0.705] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA # Parameters: (numTasks = 200000) # Run progress: 25.00% complete, ETA 00:05:40 # Fork: 1 of 1 # Warmup Iteration 1: 0.099 ops/s Iteration 1: 0.130 ops/s Iteration 2: 0.146 ops/s Iteration 3: 0.117 ops/s Iteration 4: 0.145 ops/s Iteration 5: 0.144 ops/s Result "measureMTTA": 0.137 ±(99.9%) 0.049 ops/s [Average] (min, avg, max) = (0.117, 0.137, 0.146), stdev = 0.013 CI (99.9%): [0.088, 0.185] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR # Parameters: (numTasks = 1000) # Run progress: 33.33% complete, ETA 00:05:28 # Fork: 1 of 1 # Warmup Iteration 1: 4184.816 ops/s Iteration 1: 4274.550 ops/s Iteration 2: 2630.657 ops/s Iteration 3: 3168.746 ops/s Iteration 4: 4054.913 ops/s Iteration 5: 3653.909 ops/s Result "measureMTTR": 3556.555 ±(99.9%) 2569.449 ops/s [Average] (min, avg, max) = (2630.657, 3556.555, 4274.550), stdev = 667.278 CI (99.9%): [987.106, 6126.004] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR # Parameters: (numTasks = 10000) # Run progress: 41.67% complete, ETA 00:04:40 # Fork: 1 of 1 # Warmup Iteration 1: 39.150 ops/s Iteration 1: 56.293 ops/s Iteration 2: 59.521 ops/s Iteration 3: 46.747 ops/s Iteration 4: 55.249 ops/s Iteration 5: 52.481 ops/s Result "measureMTTR": 54.058 ±(99.9%) 18.486 ops/s [Average] (min, avg, max) = (46.747, 54.058, 59.521), stdev = 4.801 CI (99.9%): [35.572, 72.544] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR # Parameters: (numTasks = 100000) # Run progress: 50.00% complete, ETA 00:03:57 # Fork: 1 of 1 # Warmup Iteration 1: 0.448 ops/s Iteration 1: 0.482 ops/s Iteration 2: 0.475 ops/s Iteration 3: 0.353 ops/s Iteration 4: 0.478 ops/s Iteration 5: 0.513 ops/s Result "measureMTTR": 0.460 ±(99.9%) 0.238 ops/s [Average] (min, avg, max) = (0.353, 0.460, 0.513), stdev = 0.062 CI (99.9%): [0.223, 0.698] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR # Parameters: (numTasks = 200000) # Run progress: 58.33% complete, ETA 00:03:23 # Fork: 1 of 1 # Warmup Iteration 1: 0.085 ops/s Iteration 1: 0.125 ops/s Iteration 2: 0.133 ops/s Iteration 3: 0.131 ops/s Iteration 4: 0.129 ops/s Iteration 5: 0.117 ops/s Result "measureMTTR": 0.127 ±(99.9%) 0.025 ops/s [Average] (min, avg, max) = (0.117, 0.127, 0.133), stdev = 0.006 CI (99.9%): [0.102, 0.152] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS # Parameters: (numTasks = 1000) # Run progress: 66.67% complete, ETA 00:02:50 # Fork: 1 of 1 # Warmup Iteration 1: 3914.893 ops/s Iteration 1: 4447.483 ops/s Iteration 2: 1089.132 ops/s Iteration 3: 1799.594 ops/s Iteration 4: 4622.600 ops/s Iteration 5: 4259.013 ops/s Result "measureMTTS": 3243.565 ±(99.9%) 6417.109 ops/s [Average] (min, avg, max) = (1089.132, 3243.565, 4622.600), stdev = 1666.503 CI (99.9%): [? 0, 9660.673] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS # Parameters: (numTasks = 10000) # Run progress: 75.00% complete, ETA 00:02:05 # Fork: 1 of 1 # Warmup Iteration 1: 55.750 ops/s Iteration 1: 56.535 ops/s Iteration 2: 57.740 ops/s Iteration 3: 59.033 ops/s Iteration 4: 55.974 ops/s Iteration 5: 49.698 ops/s Result "measureMTTS": 55.796 ±(99.9%) 13.886 ops/s [Average] (min, avg, max) = (49.698, 55.796, 59.033), stdev = 3.606 CI (99.9%): [41.910, 69.682] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS # Parameters: (numTasks = 100000) # Run progress: 83.33% complete, ETA 00:01:22 # Fork: 1 of 1 # Warmup Iteration 1: 0.484 ops/s Iteration 1: 0.523 ops/s Iteration 2: 0.544 ops/s Iteration 3: 0.550 ops/s Iteration 4: 0.518 ops/s Iteration 5: 0.506 ops/s Result "measureMTTS": 0.528 ±(99.9%) 0.071 ops/s [Average] (min, avg, max) = (0.506, 0.528, 0.550), stdev = 0.018 CI (99.9%): [0.458, 0.599] (assumes normal distribution) # JMH 1.15 (released 102 days ago) # VM version: JDK 1.8.0_111, VM 25.111-b14 # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 10 s each # Measurement: 5 iterations, 5 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS # Parameters: (numTasks = 200000) # Run progress: 91.67% complete, ETA 00:00:41 # Fork: 1 of 1 # Warmup Iteration 1: 0.099 ops/s Iteration 1: 0.131 ops/s Iteration 2: 0.116 ops/s Iteration 3: 0.114 ops/s Iteration 4: 0.107 ops/s Iteration 5: 0.105 ops/s Result "measureMTTS": 0.115 ±(99.9%) 0.039 ops/s [Average] (min, avg, max) = (0.105, 0.115, 0.131), stdev = 0.010 CI (99.9%): [0.076, 0.154] (assumes normal distribution) # Run complete. Total time: 00:08:45 Benchmark (numTasks) Mode Cnt Score Error Units SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 1000 thrpt 5 5540.905 ± 244.059 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 10000 thrpt 5 63.778 ± 5.053 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 100000 thrpt 5 0.546 ± 0.159 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 200000 thrpt 5 0.137 ± 0.049 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 1000 thrpt 5 3556.555 ± 2569.449 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 10000 thrpt 5 54.058 ± 18.486 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 100000 thrpt 5 0.460 ± 0.238 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 200000 thrpt 5 0.127 ± 0.025 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 1000 thrpt 5 3243.565 ± 6417.109 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 10000 thrpt 5 55.796 ± 13.886 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 100000 thrpt 5 0.528 ± 0.071 ops/s SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 200000 thrpt 5 0.115 ± 0.039 ops/s Benchmark result is saved to dist/reports/jmh/results.txt ``` Thanks, Reza Motamedi
