----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/#review66714 -----------------------------------------------------------
src/jmh/java/org/apache/aurora/benchmark/Hosts.java <https://reviews.apache.org/r/28731/#comment110299> It would be nice to supply the minimal attributes we expect - host and rack designations. this will at least cause diversity computation to be realistic. Seems like all you need to accomplish that is a parameter for the number of hosts per rack. src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java <https://reviews.apache.org/r/28731/#comment110313> Did you consider using a fake for `StateManager`? I consider its behavior out of scope for this benchmark, and cutting it out might remove the need for a bunch of additional binding glue. The same can be said (to a lesster extent) about `TaskAssigner`. src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java <https://reviews.apache.org/r/28731/#comment110307> Is this only to get the tasks out of PENDING? If so, would it make sense to instead initialize the storage with tasks in the state you want? Seems like that would reduce the setup complexity. src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java <https://reviews.apache.org/r/28731/#comment110300> `getClusterUtilization`? src/jmh/java/org/apache/aurora/benchmark/Tasks.java <https://reviews.apache.org/r/28731/#comment110301> This might be worth omitting, i fear its prominent inclusion here could be misleading about how priority is used in scheduling. src/jmh/java/org/apache/aurora/benchmark/fakes/FakeClusterState.java <https://reviews.apache.org/r/28731/#comment110309> Did you consider using `CachedClusterState` instead? That might cause some other things to fall into place, and alleviate the manual maintenance of this mapping. AFAICT there's a small amount of wiring necessary to make this work (including a more real `EventSink`), but that might be a worthwhile investment if it simplifies the benchmark implementations. - Bill Farner On Dec. 15, 2014, 10:15 p.m., Maxim Khutornenko wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/28731/ > ----------------------------------------------------------- > > (Updated Dec. 15, 2014, 10:15 p.m.) > > > Review request for Aurora, Kevin Sweeney and Bill Farner. > > > Repository: aurora > > > Description > ------- > > Added baseline benchmarks for a few static veto cases. > > > Diffs > ----- > > build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 > src/jmh/java/org/apache/aurora/benchmark/Hosts.java PRE-CREATION > src/jmh/java/org/apache/aurora/benchmark/Offers.java PRE-CREATION > src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java > 5cecada93e4e04b689e826af49f691ed7e94ae49 > src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java > PRE-CREATION > src/jmh/java/org/apache/aurora/benchmark/Tasks.java PRE-CREATION > src/jmh/java/org/apache/aurora/benchmark/fakes/FakeClusterState.java > PRE-CREATION > src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java PRE-CREATION > > src/jmh/java/org/apache/aurora/benchmark/fakes/FakeRescheduleCalculator.java > PRE-CREATION > src/jmh/java/org/apache/aurora/benchmark/fakes/FakeStatsProvider.java > PRE-CREATION > src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java > b6402ae42e3c7e4dca1c120fa6ef82d2d69e69d5 > src/main/java/org/apache/aurora/scheduler/async/preemptor/ClusterState.java > f7e157c890b5627411acd4bd5c2559ef4829147c > > src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java > 0204d14b19ae412236f19ca274d81decb4eba12d > > Diff: https://reviews.apache.org/r/28731/diff/ > > > Testing > ------- > > Sample run on a local box: > ``` > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java > # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en > -Duser.variant > # Warmup: 10 iterations, 1 s each > # Measurement: 100 iterations, 1 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Average time, time/op > # Benchmark: > org.apache.aurora.benchmark.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark > > # Run progress: 0.00% complete, ETA 00:05:30 > # Fork: 1 of 1 > # Warmup Iteration 1: 278284250.000 ns/op > # Warmup Iteration 2: 70294312.500 ns/op > # Warmup Iteration 3: 19293379.310 ns/op > # Warmup Iteration 4: 11945387.097 ns/op > # Warmup Iteration 5: 10725388.350 ns/op > # Warmup Iteration 6: 13043282.353 ns/op > # Warmup Iteration 7: 9233083.333 ns/op > # Warmup Iteration 8: 9521051.724 ns/op > # Warmup Iteration 9: 10750854.369 ns/op > # Warmup Iteration 10: 7460243.243 ns/op > Iteration 1: 7885364.286 ns/op > Iteration 2: 7735139.860 ns/op > Iteration 3: 7660208.333 ns/op > Iteration 4: 7761204.225 ns/op > Iteration 5: 7868907.143 ns/op > Iteration 6: 7567404.110 ns/op > Iteration 7: 7611000.000 ns/op > Iteration 8: 7766154.930 ns/op > Iteration 9: 7669344.828 ns/op > Iteration 10: 7707783.217 ns/op > Iteration 11: 7435651.007 ns/op > Iteration 12: 7697631.944 ns/op > Iteration 13: 7712531.469 ns/op > Iteration 14: 7899407.143 ns/op > Iteration 15: 7448472.973 ns/op > Iteration 16: 7791521.127 ns/op > Iteration 17: 7612213.793 ns/op > Iteration 18: 7710867.133 ns/op > Iteration 19: 7649296.552 ns/op > Iteration 20: 7768309.859 ns/op > Iteration 21: 7688666.667 ns/op > Iteration 22: 7531557.823 ns/op > Iteration 23: 7381193.333 ns/op > Iteration 24: 7726006.993 ns/op > Iteration 25: 7603358.621 ns/op > Iteration 26: 7653631.944 ns/op > Iteration 27: 7442275.168 ns/op > Iteration 28: 7613186.207 ns/op > Iteration 29: 7765823.944 ns/op > Iteration 30: 7489687.075 ns/op > Iteration 31: 7811443.662 ns/op > Iteration 32: 8015007.246 ns/op > Iteration 33: 8192392.593 ns/op > Iteration 34: 8040335.766 ns/op > Iteration 35: 7584212.329 ns/op > Iteration 36: 8001934.783 ns/op > Iteration 37: 9744815.789 ns/op > Iteration 38: 11688284.211 ns/op > Iteration 39: 8661406.250 ns/op > Iteration 40: 7678413.793 ns/op > Iteration 41: 8502223.077 ns/op > Iteration 42: 7640820.690 ns/op > Iteration 43: 7875624.113 ns/op > Iteration 44: 7506809.524 ns/op > Iteration 45: 8005431.655 ns/op > Iteration 46: 8081664.234 ns/op > Iteration 47: 7579438.356 ns/op > Iteration 48: 7993405.797 ns/op > Iteration 49: 7571958.904 ns/op > Iteration 50: 8116463.235 ns/op > Iteration 51: 7941330.935 ns/op > Iteration 52: 7687145.833 ns/op > Iteration 53: 8082554.745 ns/op > Iteration 54: 7597889.655 ns/op > Iteration 55: 7299907.285 ns/op > Iteration 56: 7992789.855 ns/op > Iteration 57: 7648268.966 ns/op > Iteration 58: 7570863.014 ns/op > Iteration 59: 7885078.571 ns/op > Iteration 60: 7647158.621 ns/op > Iteration 61: 7830858.156 ns/op > Iteration 62: 7773690.141 ns/op > Iteration 63: 7905850.000 ns/op > Iteration 64: 7653800.000 ns/op > Iteration 65: 7408248.322 ns/op > Iteration 66: 7961352.518 ns/op > Iteration 67: 7879785.714 ns/op > Iteration 68: 8308616.541 ns/op > Iteration 69: 7396852.349 ns/op > Iteration 70: 8203214.815 ns/op > Iteration 71: 7548821.918 ns/op > Iteration 72: 7834631.206 ns/op > Iteration 73: 7451594.595 ns/op > Iteration 74: 7633262.069 ns/op > Iteration 75: 8182348.148 ns/op > Iteration 76: 7412646.667 ns/op > Iteration 77: 7662375.000 ns/op > Iteration 78: 7714825.175 ns/op > Iteration 79: 7726195.804 ns/op > Iteration 80: 7552568.493 ns/op > Iteration 81: 7642655.172 ns/op > Iteration 82: 7632737.931 ns/op > Iteration 83: 7567616.438 ns/op > Iteration 84: 7641517.241 ns/op > Iteration 85: 7664152.778 ns/op > Iteration 86: 8082532.847 ns/op > Iteration 87: 8027427.536 ns/op > Iteration 88: 7837191.489 ns/op > Iteration 89: 7587938.356 ns/op > Iteration 90: 7631262.069 ns/op > Iteration 91: 7672423.611 ns/op > Iteration 92: 7505258.503 ns/op > Iteration 93: 7410449.664 ns/op > Iteration 94: 8164419.118 ns/op > Iteration 95: 7712930.070 ns/op > Iteration 96: 7934287.770 ns/op > Iteration 97: 8007239.130 ns/op > Iteration 98: 8250343.284 ns/op > Iteration 99: 8482853.846 ns/op > Iteration 100: 8138595.588 ns/op > > > Result: 7831342.453 ±(99.9%) 172179.897 ns/op [Average] > Statistics: (min, avg, max) = (7299907.285, 7831342.453, 11688284.211), > stdev = 507676.348 > Confidence interval (99.9%): [7659162.555, 8003522.350] > > > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java > # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en > -Duser.variant > # Warmup: 10 iterations, 1 s each > # Measurement: 100 iterations, 1 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Average time, time/op > # Benchmark: > org.apache.aurora.benchmark.SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark > > # Run progress: 33.33% complete, ETA 00:04:43 > # Fork: 1 of 1 > # Warmup Iteration 1: 130891625.000 ns/op > # Warmup Iteration 2: 18382600.000 ns/op > # Warmup Iteration 3: 8235328.358 ns/op > # Warmup Iteration 4: 8724039.370 ns/op > # Warmup Iteration 5: 5823163.158 ns/op > # Warmup Iteration 6: 5425686.275 ns/op > # Warmup Iteration 7: 5722766.839 ns/op > # Warmup Iteration 8: 5716419.689 ns/op > # Warmup Iteration 9: 6278869.318 ns/op > # Warmup Iteration 10: 5738276.042 ns/op > Iteration 1: 5761427.083 ns/op > Iteration 2: 5946838.710 ns/op > Iteration 3: 6091066.298 ns/op > Iteration 4: 5721787.565 ns/op > Iteration 5: 6785852.761 ns/op > Iteration 6: 5813373.684 ns/op > Iteration 7: 5866824.468 ns/op > Iteration 8: 6948911.950 ns/op > Iteration 9: 6042060.109 ns/op > Iteration 10: 5633122.449 ns/op > Iteration 11: 5753838.542 ns/op > Iteration 12: 5853433.862 ns/op > Iteration 13: 5752322.917 ns/op > Iteration 14: 5332053.140 ns/op > Iteration 15: 5318278.846 ns/op > Iteration 16: 5260276.190 ns/op > Iteration 17: 5700340.206 ns/op > Iteration 18: 5767635.417 ns/op > Iteration 19: 5367514.563 ns/op > Iteration 20: 5721994.819 ns/op > Iteration 21: 5711243.523 ns/op > Iteration 22: 6276647.727 ns/op > Iteration 23: 5934473.118 ns/op > Iteration 24: 5821626.316 ns/op > Iteration 25: 5533788.945 ns/op > Iteration 26: 7226183.007 ns/op > Iteration 27: 5448920.792 ns/op > Iteration 28: 5749453.125 ns/op > Iteration 29: 5906727.273 ns/op > Iteration 30: 5702489.691 ns/op > Iteration 31: 5355660.194 ns/op > Iteration 32: 7413523.490 ns/op > Iteration 33: 7467831.081 ns/op > Iteration 34: 8952487.805 ns/op > Iteration 35: 8635742.188 ns/op > Iteration 36: 7981898.551 ns/op > Iteration 37: 7761732.394 ns/op > Iteration 38: 7130438.710 ns/op > Iteration 39: 8349406.015 ns/op > Iteration 40: 7980985.612 ns/op > Iteration 41: 8037847.826 ns/op > Iteration 42: 6315828.571 ns/op > Iteration 43: 5375892.683 ns/op > Iteration 44: 5897005.348 ns/op > Iteration 45: 5568793.970 ns/op > Iteration 46: 5273685.714 ns/op > Iteration 47: 8879392.000 ns/op > Iteration 48: 8641343.750 ns/op > Iteration 49: 10007136.364 ns/op > Iteration 50: 8484908.397 ns/op > Iteration 51: 7659333.333 ns/op > Iteration 52: 8661968.750 ns/op > Iteration 53: 8923153.226 ns/op > Iteration 54: 8746527.559 ns/op > Iteration 55: 8981138.211 ns/op > Iteration 56: 5631193.878 ns/op > Iteration 57: 5945069.892 ns/op > Iteration 58: 5610161.616 ns/op > Iteration 59: 5348665.049 ns/op > Iteration 60: 5429064.039 ns/op > Iteration 61: 5432418.719 ns/op > Iteration 62: 5621515.306 ns/op > Iteration 63: 5268630.332 ns/op > Iteration 64: 5849894.180 ns/op > Iteration 65: 5418126.829 ns/op > Iteration 66: 6133177.778 ns/op > Iteration 67: 6195095.506 ns/op > Iteration 68: 6111276.243 ns/op > Iteration 69: 5886244.681 ns/op > Iteration 70: 6430941.860 ns/op > Iteration 71: 5783329.843 ns/op > Iteration 72: 6507141.176 ns/op > Iteration 73: 10092783.784 ns/op > Iteration 74: 11873978.495 ns/op > Iteration 75: 8876424.000 ns/op > Iteration 76: 5878223.404 ns/op > Iteration 77: 7130651.613 ns/op > Iteration 78: 5508910.000 ns/op > Iteration 79: 5481338.308 ns/op > Iteration 80: 5999592.391 ns/op > Iteration 81: 6065917.582 ns/op > Iteration 82: 6087812.155 ns/op > Iteration 83: 5653512.821 ns/op > Iteration 84: 5528065.000 ns/op > Iteration 85: 5599314.721 ns/op > Iteration 86: 5300822.115 ns/op > Iteration 87: 6341833.333 ns/op > Iteration 88: 5511190.000 ns/op > Iteration 89: 5813026.316 ns/op > Iteration 90: 5914112.299 ns/op > Iteration 91: 5663928.205 ns/op > Iteration 92: 5723891.192 ns/op > Iteration 93: 5580843.434 ns/op > Iteration 94: 5990407.609 ns/op > Iteration 95: 5656717.949 ns/op > Iteration 96: 5432532.020 ns/op > Iteration 97: 5797827.225 ns/op > Iteration 98: 5647410.256 ns/op > Iteration 99: 5250814.286 ns/op > Iteration 100: 5602177.665 ns/op > > > Result: 6418401.739 ±(99.9%) 447052.433 ns/op [Average] > Statistics: (min, avg, max) = (5250814.286, 6418401.739, 11873978.495), > stdev = 1318144.279 > Confidence interval (99.9%): [5971349.307, 6865454.172] > > > # VM invoker: > /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java > # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en > -Duser.variant > # Warmup: 10 iterations, 1 s each > # Measurement: 100 iterations, 1 s each > # Timeout: 10 min per iteration > # Threads: 1 thread, will synchronize iterations > # Benchmark mode: Average time, time/op > # Benchmark: > org.apache.aurora.benchmark.SchedulingBenchmarks.PreemptorFallbackForLargeClusterBenchmark.runBenchmark > > # Run progress: 66.67% complete, ETA 00:02:19 > # Fork: 1 of 1 > # Warmup Iteration 1: 7780760000.000 ns/op > # Warmup Iteration 2: 165927428.571 ns/op > # Warmup Iteration 3: 94242076.923 ns/op > # Warmup Iteration 4: 66061000.000 ns/op > # Warmup Iteration 5: 70444500.000 ns/op > # Warmup Iteration 6: 62350000.000 ns/op > # Warmup Iteration 7: 63746555.556 ns/op > # Warmup Iteration 8: 62712111.111 ns/op > # Warmup Iteration 9: 62776388.889 ns/op > # Warmup Iteration 10: 62455277.778 ns/op > Iteration 1: 65696764.706 ns/op > Iteration 2: 66255823.529 ns/op > Iteration 3: 68231647.059 ns/op > Iteration 4: 71606750.000 ns/op > Iteration 5: 66780235.294 ns/op > Iteration 6: 69914750.000 ns/op > Iteration 7: 74681333.333 ns/op > Iteration 8: 64133944.444 ns/op > Iteration 9: 60831736.842 ns/op > Iteration 10: 63261444.444 ns/op > Iteration 11: 60814105.263 ns/op > Iteration 12: 67882941.176 ns/op > Iteration 13: 62512888.889 ns/op > Iteration 14: 65551823.529 ns/op > Iteration 15: 65574294.118 ns/op > Iteration 16: 67928352.941 ns/op > Iteration 17: 65510588.235 ns/op > Iteration 18: 62637055.556 ns/op > Iteration 19: 62900222.222 ns/op > Iteration 20: 63715944.444 ns/op > Iteration 21: 62284611.111 ns/op > Iteration 22: 68174235.294 ns/op > Iteration 23: 69029187.500 ns/op > Iteration 24: 68584764.706 ns/op > Iteration 25: 63332777.778 ns/op > Iteration 26: 64316000.000 ns/op > Iteration 27: 64219444.444 ns/op > Iteration 28: 64219888.889 ns/op > Iteration 29: 63113888.889 ns/op > Iteration 30: 65802117.647 ns/op > Iteration 31: 63661277.778 ns/op > Iteration 32: 63322000.000 ns/op > Iteration 33: 63770000.000 ns/op > Iteration 34: 63920333.333 ns/op > Iteration 35: 62403500.000 ns/op > Iteration 36: 64137166.667 ns/op > Iteration 37: 64725444.444 ns/op > Iteration 38: 63145166.667 ns/op > Iteration 39: 64183944.444 ns/op > Iteration 40: 65873666.667 ns/op > Iteration 41: 65648529.412 ns/op > Iteration 42: 66487176.471 ns/op > Iteration 43: 61702888.889 ns/op > Iteration 44: 63267666.667 ns/op > Iteration 45: 69614125.000 ns/op > Iteration 46: 61574222.222 ns/op > Iteration 47: 62678611.111 ns/op > Iteration 48: 62792277.778 ns/op > Iteration 49: 65355352.941 ns/op > Iteration 50: 65206823.529 ns/op > Iteration 51: 64246722.222 ns/op > Iteration 52: 68074823.529 ns/op > Iteration 53: 67314823.529 ns/op > Iteration 54: 65511882.353 ns/op > Iteration 55: 63607722.222 ns/op > Iteration 56: 66646529.412 ns/op > Iteration 57: 67675764.706 ns/op > Iteration 58: 62502388.889 ns/op > Iteration 59: 70476625.000 ns/op > Iteration 60: 97138500.000 ns/op > Iteration 61: 93276461.538 ns/op > Iteration 62: 103508818.182 ns/op > Iteration 63: 96725916.667 ns/op > Iteration 64: 97539500.000 ns/op > Iteration 65: 93591461.538 ns/op > Iteration 66: 95847083.333 ns/op > Iteration 67: 101147666.667 ns/op > Iteration 68: 91144230.769 ns/op > Iteration 69: 69291375.000 ns/op > Iteration 70: 67021941.176 ns/op > Iteration 71: 64101166.667 ns/op > Iteration 72: 66573823.529 ns/op > Iteration 73: 66116235.294 ns/op > Iteration 74: 62875611.111 ns/op > Iteration 75: 72765812.500 ns/op > Iteration 76: 65663176.471 ns/op > Iteration 77: 64187000.000 ns/op > Iteration 78: 64425444.444 ns/op > Iteration 79: 63835111.111 ns/op > Iteration 80: 62942166.667 ns/op > Iteration 81: 69006500.000 ns/op > Iteration 82: 64798176.471 ns/op > Iteration 83: 67823117.647 ns/op > Iteration 84: 77090866.667 ns/op > Iteration 85: 71548625.000 ns/op > Iteration 86: 63718333.333 ns/op > Iteration 87: 67034764.706 ns/op > Iteration 88: 61701833.333 ns/op > Iteration 89: 65456529.412 ns/op > Iteration 90: 63462388.889 ns/op > Iteration 91: 61797555.556 ns/op > Iteration 92: 66772823.529 ns/op > Iteration 93: 66240117.647 ns/op > Iteration 94: 63701444.444 ns/op > Iteration 95: 64047888.889 ns/op > Iteration 96: 62162777.778 ns/op > Iteration 97: 64846588.235 ns/op > Iteration 98: 61865666.667 ns/op > Iteration 99: 61965222.222 ns/op > Iteration 100: 64089333.333 ns/op > > > Result: 68178760.746 ±(99.9%) 3225821.640 ns/op [Average] > Statistics: (min, avg, max) = (60814105.263, 68178760.746, 103508818.182), > stdev = 9511408.567 > Confidence interval (99.9%): [64952939.106, 71404582.386] > > > # Run complete. Total time: 00:07:18 > > Benchmark > Mode Samples Score Error Units > o.a.a.b.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark > avgt 100 7831342.453 ± 172179.897 ns/op > o.a.a.b.SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark > avgt 100 6418401.739 ± 447052.433 ns/op > o.a.a.b.SchedulingBenchmarks.PreemptorFallbackForLargeClusterBenchmark.runBenchmark > avgt 100 68178760.746 ± 3225821.640 ns/op > ``` > > > Thanks, > > Maxim Khutornenko > >
