Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64578 --- Ship it! Ship It! - Bill Farner On Dec. 10, 2014, 1:22 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 10, 2014, 1:22 a.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
On Dec. 5, 2014, 8:06 p.m., Bill Farner wrote: Not to bikeshed, but do either of you have a sense for what it would take to put the benchmarking code into a gradle subproject? My sense is that would avoid further complicating the root build.gradle, but i'm not familiar enough with gradle to know what, if any, hurdles that would present. Maxim Khutornenko wrote: There is not much to hide in a subproject as all you see here is configuring the plugin to run with our gradle. I don't see any sane way of hiding it in a subproject short of reimplementing the plugin ourselves. Bill Farner wrote: Kevin - do you agree with this? You're more versed in subprojects than i am, so i'll defer to your judgement. Ping - Kevin? - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64066 --- On Dec. 6, 2014, 12:33 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 6, 2014, 12:33 a.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64487 --- src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java https://reviews.apache.org/r/28710/#comment107200 What does the main get us here? Isn't the gradle plugin wiring this up? - Kevin Sweeney On Dec. 5, 2014, 4:33 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 5, 2014, 4:33 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
On Dec. 10, 2014, 1:09 a.m., Kevin Sweeney wrote: src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java, lines 29-32 https://reviews.apache.org/r/28710/diff/4/?file=784251#file784251line29 What does the main get us here? Isn't the gradle plugin wiring this up? Good point. I used it to run benchmarks from command line but given the command line execution is rather tedious with all jmh flags passed into it, it's cleaner to support only one way via gradle. Dropped. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64487 --- On Dec. 6, 2014, 12:33 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 6, 2014, 12:33 a.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 10, 2014, 1:22 a.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Changes --- CR comments. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs (updated) - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
On Dec. 5, 2014, 12:06 p.m., Bill Farner wrote: Not to bikeshed, but do either of you have a sense for what it would take to put the benchmarking code into a gradle subproject? My sense is that would avoid further complicating the root build.gradle, but i'm not familiar enough with gradle to know what, if any, hurdles that would present. Maxim Khutornenko wrote: There is not much to hide in a subproject as all you see here is configuring the plugin to run with our gradle. I don't see any sane way of hiding it in a subproject short of reimplementing the plugin ourselves. Bill Farner wrote: Kevin - do you agree with this? You're more versed in subprojects than i am, so i'll defer to your judgement. Bill Farner wrote: Ping - Kevin? This plugin doesn't seem geared toward use in a subproject so I think leaving src/jmh in place is (if not ideal) acceptable here. - Kevin --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64066 --- On Dec. 9, 2014, 5:22 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 9, 2014, 5:22 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64492 --- Ship it! Master (2aac148) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Dec. 10, 2014, 1:22 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 10, 2014, 1:22 a.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 5, 2014, 6:32 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Changes --- Rebased. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs (updated) - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64056 --- Ship it! Master (40d2718) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Dec. 5, 2014, 6:32 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 5, 2014, 6:32 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28710/#review63897
---
build.gradle
https://reviews.apache.org/r/28710/#comment106210
This should be unneeded.
build.gradle
https://reviews.apache.org/r/28710/#comment106452
im not convinced you want to apply this to the api subproject.
build.gradle
https://reviews.apache.org/r/28710/#comment106449
Do we want to apply jmh to the 'api' project? I'd think we'd want to add
the root. That will give you the benefit of not having to disable this task on
the api project at all.
build.gradle
https://reviews.apache.org/r/28710/#comment106451
should this be a test scope? also should this be in the root project
instead of the api project?
build.gradle
https://reviews.apache.org/r/28710/#comment106454
Does
```groovy
tasks('jmh') {
//...
}
```
work?
- Kevin Sweeney
On Dec. 5, 2014, 10:32 a.m., Maxim Khutornenko wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28710/
---
(Updated Dec. 5, 2014, 10:32 a.m.)
Review request for Aurora, Kevin Sweeney and Bill Farner.
Repository: aurora
Description
---
This RB is superseding the https://reviews.apache.org/r/28474/.
I have spent some time researching the available microbenchmark frameworks
and JMH [1] came as a clear winner:
- Active development trail [2]
- Advanced featureset and built-in optimizations improving accuracy and
consistency [3]
- Well documented set of examples [4]
- Large community experience and collective wisdom.
This RB adds gradle support for running JMH benchmarks and is relying on JMH
gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command.
[1] - http://openjdk.java.net/projects/code-tools/jmh/
[2] - http://hg.openjdk.java.net/code-tools/jmh/
[3] -
https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ
[4] -
http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples
[5] - https://github.com/melix/jmh-gradle-plugin
Diffs
-
build.gradle 152ba631e2dd07f0306e58e355274e10a4128140
config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f
src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java
PRE-CREATION
Diff: https://reviews.apache.org/r/28710/diff/
Testing
---
$ ./gradlew jmh
Sample results generated:
```
# VM invoker:
/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java
# VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en
-Duser.variant
# Warmup: 1 iterations, 1 s each
# Measurement: 3 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example
# Run progress: 0.00% complete, ETA 00:00:04
# Fork: 1 of 1
# Warmup Iteration 1: 3156839103.911 ops/s
Iteration 1: 544897.411 ops/s
Iteration 2: 3357230627.218 ops/s
Iteration 3: 3461073727.560 ops/s
Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average]
Statistics: (min, avg, max) = (544897.411, 3383949750.729,
3461073727.560), stdev = 67833135.714
Confidence interval (99.9%): [2146420835.212, 4621478666.247]
# Run complete. Total time: 00:00:05
Benchmark Mode Samples Score
Error Units
o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ±
1237528915.517 ops/s
```
Thanks,
Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64066 --- Not to bikeshed, but do either of you have a sense for what it would take to put the benchmarking code into a gradle subproject? My sense is that would avoid further complicating the root build.gradle, but i'm not familiar enough with gradle to know what, if any, hurdles that would present. - Bill Farner On Dec. 5, 2014, 6:32 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 5, 2014, 6:32 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
On Dec. 5, 2014, 8:06 p.m., Bill Farner wrote: Not to bikeshed, but do either of you have a sense for what it would take to put the benchmarking code into a gradle subproject? My sense is that would avoid further complicating the root build.gradle, but i'm not familiar enough with gradle to know what, if any, hurdles that would present. There is not much to hide in a subproject as all you see here is configuring the plugin to run with our gradle. I don't see any sane way of hiding it in a subproject short of reimplementing the plugin ourselves. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64066 --- On Dec. 5, 2014, 6:32 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 5, 2014, 6:32 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
On Dec. 5, 2014, 8:06 p.m., Bill Farner wrote: Not to bikeshed, but do either of you have a sense for what it would take to put the benchmarking code into a gradle subproject? My sense is that would avoid further complicating the root build.gradle, but i'm not familiar enough with gradle to know what, if any, hurdles that would present. Maxim Khutornenko wrote: There is not much to hide in a subproject as all you see here is configuring the plugin to run with our gradle. I don't see any sane way of hiding it in a subproject short of reimplementing the plugin ourselves. Kevin - do you agree with this? You're more versed in subprojects than i am, so i'll defer to your judgement. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review64066 --- On Dec. 5, 2014, 6:32 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 5, 2014, 6:32 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 6, 2014, 12:33 a.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Changes --- CR comments. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs (updated) - build.gradle 152ba631e2dd07f0306e58e355274e10a4128140 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
On Dec. 5, 2014, 7:42 p.m., Kevin Sweeney wrote:
build.gradle, line 53
https://reviews.apache.org/r/28710/diff/1/?file=782698#file782698line53
This should be unneeded.
Dropped.
On Dec. 5, 2014, 7:42 p.m., Kevin Sweeney wrote:
build.gradle, line 103
https://reviews.apache.org/r/28710/diff/3/?file=783833#file783833line103
im not convinced you want to apply this to the api subproject.
This was only needed to make :api recognize .jmh configuration. However, played
a bit more and found a way to reference parent configuration. Dropped.
On Dec. 5, 2014, 7:42 p.m., Kevin Sweeney wrote:
build.gradle, line 156
https://reviews.apache.org/r/28710/diff/3/?file=783833#file783833line156
Do we want to apply jmh to the 'api' project? I'd think we'd want to
add the root. That will give you the benefit of not having to disable this
task on the api project at all.
Gone.
On Dec. 5, 2014, 7:42 p.m., Kevin Sweeney wrote:
build.gradle, line 165
https://reviews.apache.org/r/28710/diff/3/?file=783833#file783833line165
should this be a test scope? also should this be in the root project
instead of the api project?
It's only needed to resolve /jmh deps. Test scope is irrelevant here.
On Dec. 5, 2014, 7:42 p.m., Kevin Sweeney wrote:
build.gradle, line 458
https://reviews.apache.org/r/28710/diff/3/?file=783833#file783833line458
Does
```groovy
tasks('jmh') {
//...
}
```
work?
It does not: ` Could not find method tasks() for arguments [jmh] on root
project 'aurora'.`
- Maxim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28710/#review63897
---
On Dec. 5, 2014, 6:32 p.m., Maxim Khutornenko wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28710/
---
(Updated Dec. 5, 2014, 6:32 p.m.)
Review request for Aurora, Kevin Sweeney and Bill Farner.
Repository: aurora
Description
---
This RB is superseding the https://reviews.apache.org/r/28474/.
I have spent some time researching the available microbenchmark frameworks
and JMH [1] came as a clear winner:
- Active development trail [2]
- Advanced featureset and built-in optimizations improving accuracy and
consistency [3]
- Well documented set of examples [4]
- Large community experience and collective wisdom.
This RB adds gradle support for running JMH benchmarks and is relying on JMH
gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command.
[1] - http://openjdk.java.net/projects/code-tools/jmh/
[2] - http://hg.openjdk.java.net/code-tools/jmh/
[3] -
https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ
[4] -
http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples
[5] - https://github.com/melix/jmh-gradle-plugin
Diffs
-
build.gradle 152ba631e2dd07f0306e58e355274e10a4128140
config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f
src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java
PRE-CREATION
Diff: https://reviews.apache.org/r/28710/diff/
Testing
---
$ ./gradlew jmh
Sample results generated:
```
# VM invoker:
/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java
# VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en
-Duser.variant
# Warmup: 1 iterations, 1 s each
# Measurement: 3 iterations, 1 s each
# Timeout: 10 min per iteration
# Threads: 1 thread, will synchronize iterations
# Benchmark mode: Throughput, ops/time
# Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example
# Run progress: 0.00% complete, ETA 00:00:04
# Fork: 1 of 1
# Warmup Iteration 1: 3156839103.911 ops/s
Iteration 1: 544897.411 ops/s
Iteration 2: 3357230627.218 ops/s
Iteration 3: 3461073727.560 ops/s
Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average]
Statistics: (min, avg, max) = (544897.411, 3383949750.729,
3461073727.560), stdev = 67833135.714
Confidence interval (99.9%): [2146420835.212, 4621478666.247]
# Run complete. Total time: 00:00:05
Benchmark Mode Samples Score
Error Units
o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ±
1237528915.517 ops/s
```
Thanks,
Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review63889 --- Ship it! Master (11da34c) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Dec. 4, 2014, 7:36 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 4, 2014, 7:36 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle fb729c5096108c535229e266fa9649f997e6da37 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review63896 --- build.gradle https://reviews.apache.org/r/28710/#comment106209 Looks like you want the new-style block here instead: https://plugins.gradle.org/plugin/me.champeau.gradle.jmh - Kevin Sweeney On Dec. 4, 2014, 11:36 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 4, 2014, 11:36 a.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle fb729c5096108c535229e266fa9649f997e6da37 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
On Dec. 4, 2014, 8:12 p.m., Kevin Sweeney wrote: build.gradle, line 23 https://reviews.apache.org/r/28710/diff/1/?file=782698#file782698line23 Looks like you want the new-style block here instead: https://plugins.gradle.org/plugin/me.champeau.gradle.jmh Great suggestion. Done. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review63896 --- On Dec. 4, 2014, 7:36 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 4, 2014, 7:36 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle fb729c5096108c535229e266fa9649f997e6da37 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
Re: Review Request 28710: Adding JMH framework support for scheduler performance analysis.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/#review63900 --- Ship it! Master (11da34c) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Dec. 4, 2014, 8:20 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28710/ --- (Updated Dec. 4, 2014, 8:20 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Repository: aurora Description --- This RB is superseding the https://reviews.apache.org/r/28474/. I have spent some time researching the available microbenchmark frameworks and JMH [1] came as a clear winner: - Active development trail [2] - Advanced featureset and built-in optimizations improving accuracy and consistency [3] - Well documented set of examples [4] - Large community experience and collective wisdom. This RB adds gradle support for running JMH benchmarks and is relying on JMH gradle plugin [5]. The benchmarks are run via `./gradlew jmh` command. [1] - http://openjdk.java.net/projects/code-tools/jmh/ [2] - http://hg.openjdk.java.net/code-tools/jmh/ [3] - https://groups.google.com/forum/#!msg/mechanical-sympathy/m4opvy4xq3U/7lY8x8SvHgwJ [4] - http://hg.openjdk.java.net/code-tools/jmh/file/adb6047266d8/jmh-samples/src/main/java/org/openjdk/jmh/samples [5] - https://github.com/melix/jmh-gradle-plugin Diffs - build.gradle fb729c5096108c535229e266fa9649f997e6da37 config/findbugs/excludeFilter.xml d6c1b1681c2d8505a088f9fb082ce11ac400126f src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java PRE-CREATION Diff: https://reviews.apache.org/r/28710/diff/ Testing --- $ ./gradlew jmh Sample results generated: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 1 iterations, 1 s each # Measurement: 3 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Throughput, ops/time # Benchmark: org.apache.aurora.benchmark.SchedulerBenchmark.example # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration 1: 3156839103.911 ops/s Iteration 1: 544897.411 ops/s Iteration 2: 3357230627.218 ops/s Iteration 3: 3461073727.560 ops/s Result: 3383949750.729 ±(99.9%) 1237528915.517 ops/s [Average] Statistics: (min, avg, max) = (544897.411, 3383949750.729, 3461073727.560), stdev = 67833135.714 Confidence interval (99.9%): [2146420835.212, 4621478666.247] # Run complete. Total time: 00:00:05 Benchmark Mode Samples Score Error Units o.a.a.b.SchedulerBenchmark.examplethrpt3 3383949750.729 ± 1237528915.517 ops/s ``` Thanks, Maxim Khutornenko
