----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55284/#review161084 -----------------------------------------------------------
src/jmh/java/org/apache/aurora/benchmark/Tasks.java (lines 98 - 105) <https://reviews.apache.org/r/55284/#comment232326> I see what you are trying to keep task status and events consistant. But if I set the task events first and then set the status then I'll lose the events. Also setting status based on event timestamps is rather shaky (all those events are likely to have the same timestamp). So, I would suggest moving the consistency checking logic to `build()` method and only verifying that the last event (last element of the `events` list) matches the status. - Mehrdad Nurolahzade On Jan. 10, 2017, 1 a.m., Reza Motamedi wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/55284/ > ----------------------------------------------------------- > > (Updated Jan. 10, 2017, 1 a.m.) > > > Review request for Aurora, Joshua Cohen, Mehrdad Nurolahzade, and Santhosh > Kumar Shanmugham. > > > Repository: aurora > > > Description > ------- > > # Adding SLA benchmarks benchmarks > This RB is related to > [AURORA-1854](https://issues.apache.org/jira/browse/AURORA-1854). > The goal is to add JMH benchmarks for SlaCalculation and then other metric > calculators. > > > Diffs > ----- > > src/jmh/java/org/apache/aurora/benchmark/SlaAlgorithmBenchmark.java > PRE-CREATION > src/jmh/java/org/apache/aurora/benchmark/Tasks.java > 3aabc5ef222a97d8ebde693c52030a66bca340d2 > src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java > 5d8d5bd8f705770979f284d26d2e932aabe707e5 > > Diff: https://reviews.apache.org/r/55284/diff/ > > > Testing > ------- > > the content of <./aurora/dist/reports/jmh/human.txt> > > ``` > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA > # Parameters: (numTasks = 1000) > > # Run progress: 0.00% complete, ETA 00:07:00 > # Fork: 1 of 1 > # Warmup Iteration 1: 5141.403 ops/s > Iteration 1: 5514.591 ops/s > Iteration 2: 5508.875 ops/s > Iteration 3: 5594.224 ops/s > Iteration 4: 5467.605 ops/s > Iteration 5: 5619.229 ops/s > > > Result "measureMTTA": > 5540.905 ±(99.9%) 244.059 ops/s [Average] > (min, avg, max) = (5467.605, 5540.905, 5619.229), stdev = 63.381 > CI (99.9%): [5296.846, 5784.963] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA > # Parameters: (numTasks = 10000) > > # Run progress: 8.33% complete, ETA 00:06:34 > # Fork: 1 of 1 > # Warmup Iteration 1: 64.993 ops/s > Iteration 1: 61.762 ops/s > Iteration 2: 64.468 ops/s > Iteration 3: 65.047 ops/s > Iteration 4: 63.199 ops/s > Iteration 5: 64.412 ops/s > > > Result "measureMTTA": > 63.778 ±(99.9%) 5.053 ops/s [Average] > (min, avg, max) = (61.762, 63.778, 65.047), stdev = 1.312 > CI (99.9%): [58.725, 68.830] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA > # Parameters: (numTasks = 100000) > > # Run progress: 16.67% complete, ETA 00:06:00 > # Fork: 1 of 1 > # Warmup Iteration 1: 0.490 ops/s > Iteration 1: 0.556 ops/s > Iteration 2: 0.503 ops/s > Iteration 3: 0.502 ops/s > Iteration 4: 0.576 ops/s > Iteration 5: 0.591 ops/s > > > Result "measureMTTA": > 0.546 ±(99.9%) 0.159 ops/s [Average] > (min, avg, max) = (0.502, 0.546, 0.591), stdev = 0.041 > CI (99.9%): [0.387, 0.705] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA > # Parameters: (numTasks = 200000) > > # Run progress: 25.00% complete, ETA 00:05:40 > # Fork: 1 of 1 > # Warmup Iteration 1: 0.099 ops/s > Iteration 1: 0.130 ops/s > Iteration 2: 0.146 ops/s > Iteration 3: 0.117 ops/s > Iteration 4: 0.145 ops/s > Iteration 5: 0.144 ops/s > > > Result "measureMTTA": > 0.137 ±(99.9%) 0.049 ops/s [Average] > (min, avg, max) = (0.117, 0.137, 0.146), stdev = 0.013 > CI (99.9%): [0.088, 0.185] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR > # Parameters: (numTasks = 1000) > > # Run progress: 33.33% complete, ETA 00:05:28 > # Fork: 1 of 1 > # Warmup Iteration 1: 4184.816 ops/s > Iteration 1: 4274.550 ops/s > Iteration 2: 2630.657 ops/s > Iteration 3: 3168.746 ops/s > Iteration 4: 4054.913 ops/s > Iteration 5: 3653.909 ops/s > > > Result "measureMTTR": > 3556.555 ±(99.9%) 2569.449 ops/s [Average] > (min, avg, max) = (2630.657, 3556.555, 4274.550), stdev = 667.278 > CI (99.9%): [987.106, 6126.004] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR > # Parameters: (numTasks = 10000) > > # Run progress: 41.67% complete, ETA 00:04:40 > # Fork: 1 of 1 > # Warmup Iteration 1: 39.150 ops/s > Iteration 1: 56.293 ops/s > Iteration 2: 59.521 ops/s > Iteration 3: 46.747 ops/s > Iteration 4: 55.249 ops/s > Iteration 5: 52.481 ops/s > > > Result "measureMTTR": > 54.058 ±(99.9%) 18.486 ops/s [Average] > (min, avg, max) = (46.747, 54.058, 59.521), stdev = 4.801 > CI (99.9%): [35.572, 72.544] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR > # Parameters: (numTasks = 100000) > > # Run progress: 50.00% complete, ETA 00:03:57 > # Fork: 1 of 1 > # Warmup Iteration 1: 0.448 ops/s > Iteration 1: 0.482 ops/s > Iteration 2: 0.475 ops/s > Iteration 3: 0.353 ops/s > Iteration 4: 0.478 ops/s > Iteration 5: 0.513 ops/s > > > Result "measureMTTR": > 0.460 ±(99.9%) 0.238 ops/s [Average] > (min, avg, max) = (0.353, 0.460, 0.513), stdev = 0.062 > CI (99.9%): [0.223, 0.698] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR > # Parameters: (numTasks = 200000) > > # Run progress: 58.33% complete, ETA 00:03:23 > # Fork: 1 of 1 > # Warmup Iteration 1: 0.085 ops/s > Iteration 1: 0.125 ops/s > Iteration 2: 0.133 ops/s > Iteration 3: 0.131 ops/s > Iteration 4: 0.129 ops/s > Iteration 5: 0.117 ops/s > > > Result "measureMTTR": > 0.127 ±(99.9%) 0.025 ops/s [Average] > (min, avg, max) = (0.117, 0.127, 0.133), stdev = 0.006 > CI (99.9%): [0.102, 0.152] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS > # Parameters: (numTasks = 1000) > > # Run progress: 66.67% complete, ETA 00:02:50 > # Fork: 1 of 1 > # Warmup Iteration 1: 3914.893 ops/s > Iteration 1: 4447.483 ops/s > Iteration 2: 1089.132 ops/s > Iteration 3: 1799.594 ops/s > Iteration 4: 4622.600 ops/s > Iteration 5: 4259.013 ops/s > > > Result "measureMTTS": > 3243.565 ±(99.9%) 6417.109 ops/s [Average] > (min, avg, max) = (1089.132, 3243.565, 4622.600), stdev = 1666.503 > CI (99.9%): [? 0, 9660.673] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS > # Parameters: (numTasks = 10000) > > # Run progress: 75.00% complete, ETA 00:02:05 > # Fork: 1 of 1 > # Warmup Iteration 1: 55.750 ops/s > Iteration 1: 56.535 ops/s > Iteration 2: 57.740 ops/s > Iteration 3: 59.033 ops/s > Iteration 4: 55.974 ops/s > Iteration 5: 49.698 ops/s > > > Result "measureMTTS": > 55.796 ±(99.9%) 13.886 ops/s [Average] > (min, avg, max) = (49.698, 55.796, 59.033), stdev = 3.606 > CI (99.9%): [41.910, 69.682] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS > # Parameters: (numTasks = 100000) > > # Run progress: 83.33% complete, ETA 00:01:22 > # Fork: 1 of 1 > # Warmup Iteration 1: 0.484 ops/s > Iteration 1: 0.523 ops/s > Iteration 2: 0.544 ops/s > Iteration 3: 0.550 ops/s > Iteration 4: 0.518 ops/s > Iteration 5: 0.506 ops/s > > > Result "measureMTTS": > 0.528 ±(99.9%) 0.071 ops/s [Average] > (min, avg, max) = (0.506, 0.528, 0.550), stdev = 0.018 > CI (99.9%): [0.458, 0.599] (assumes normal distribution) > > > # JMH 1.15 (released 102 days ago) > # VM version: JDK 1.8.0_111, VM 25.111-b14 > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home/jre/bin/java > # VM options: -Xmx3g -Dfile.encoding=UTF-8 -Duser.country=US > -Duser.language=en -Duser.variant > # Warmup: 1 iterations, 10 s each > # Measurement: 5 iterations, 5 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Throughput, ops/time > # Benchmark: > org.apache.aurora.benchmark.SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS > # Parameters: (numTasks = 200000) > > # Run progress: 91.67% complete, ETA 00:00:41 > # Fork: 1 of 1 > # Warmup Iteration 1: 0.099 ops/s > Iteration 1: 0.131 ops/s > Iteration 2: 0.116 ops/s > Iteration 3: 0.114 ops/s > Iteration 4: 0.107 ops/s > Iteration 5: 0.105 ops/s > > > Result "measureMTTS": > 0.115 ±(99.9%) 0.039 ops/s [Average] > (min, avg, max) = (0.105, 0.115, 0.131), stdev = 0.010 > CI (99.9%): [0.076, 0.154] (assumes normal distribution) > > > # Run complete. Total time: 00:08:45 > > Benchmark (numTasks) Mode Cnt > Score Error Units > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 1000 thrpt 5 > 5540.905 ± 244.059 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 10000 thrpt 5 > 63.778 ± 5.053 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 100000 thrpt 5 > 0.546 ± 0.159 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTA 200000 thrpt 5 > 0.137 ± 0.049 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 1000 thrpt 5 > 3556.555 ± 2569.449 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 10000 thrpt 5 > 54.058 ± 18.486 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 100000 thrpt 5 > 0.460 ± 0.238 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTR 200000 thrpt 5 > 0.127 ± 0.025 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 1000 thrpt 5 > 3243.565 ± 6417.109 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 10000 thrpt 5 > 55.796 ± 13.886 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 100000 thrpt 5 > 0.528 ± 0.071 ops/s > SlaAlgorithmBenchmark.MTTXBenchmark.measureMTTS 200000 thrpt 5 > 0.115 ± 0.039 ops/s > > Benchmark result is saved to dist/reports/jmh/results.txt > > > ``` > > > Thanks, > > Reza Motamedi > >
