Hi Volker, I thought I'd posted my findings back here but turns out I'd tweeted them:
https://twitter.com/chriswhocodes/status/596723775779012608 Here is my JMH Octane wrapper: https://gist.github.com/chriswhocodes/c5bf3632fb7272e994e9 The gist doesn't incorporate incorporate Aleksey's @Setup suggestion. Given the time taken to reach 00s of runs I think it might be better to benchmark each Octane test in isolation rather than wait for the whole suite to stabilise? -Chris On Thu, June 25, 2015 09:04, Marcus Lagergren wrote: > No results are very unstable - using JMH will bring significant stability > to them, but you have to realize that in some cases HotSpot needs to do > hundreds of runs before plateauing out because of LambdaForms etc. > Aleksey knows more. > > > /M > > >> On 24 Jun 2015, at 17:50, Volker Simonis <volker.simo...@gmail.com> >> wrote: >> >> >> Hi Chris, Marcus, >> >> >> thanks for your responses. >> >> Some colleagues of mine who set up Octane in our environment >> complained about the test results being extremely unstable. And even more >> so if comparing two different JDK versions (like our internal build with >> an "official" Oracle-JDK build" which should "theoretically" behave the >> same). >> >> So I wanted to look at the differences and what I found was the quote >> from the Nashorn Wiki which says: "the octane base.js harness .. has a >> very primitive and deviation prone way of measuring benchmark time." >> That's why I thought to better use JMH harness as described on that >> page before starting to look at "real" problems. >> >> >> @Marcus: do you really see stable benchmark results when using your >> internal JMH Octane harness or is this a pointless undertaking anywhere? >> >> @Chris: if it isn't too much of an effort for you, I'd appreciate to >> have a look at your harness. >> >> Regards, >> Volker >> >> >> On Wed, Jun 24, 2015 at 5:38 PM, Chris Newland >> <cnewl...@chrisnewland.com> wrote: >> >>> Hi Volker, >>> >>> >>> In the end I wrote my own JMH harness around octane but I can't get >>> the results stable even with 3 day runs so my JMH code could be buggy. >>> >>> >>> Will share later off list if you're interested. >>> >>> >>> Regards, >>> >>> >>> Chris >>> @chriswhocodes >>> >>> >>> On Wed, June 24, 2015 16:00, Marcus Lagergren wrote: >>> >>>> Not sure if itâs available publically as part of JMH, but you can >>>> get the octane benchmark code itself from >>>> >>>> svn co http://octane-benchmark.googlecode.com/svn/trunk >>>> <http://octane-benchmark.googlecode.com/svn/trunk> >>>> >>>> >>>> >>>> /M >>>> >>>> >>>> >>>>> On 24 Jun 2015, at 16:41, Volker Simonis >>>>> <volker.simo...@gmail.com> >>>>> wrote: >>>>> >>>>> >>>>> >>>>> I just had the same question and couldn't find any answer with >>>>> Google >>>>> except this mail thread. >>>>> >>>>> So I'll just as again: is the Octane benchmark as part of JMH as >>>>> described in the Nashorn Wiki [1] publicly available somewhere? >>>>> >>>>> Thanks, >>>>> Volker >>>>> >>>>> >>>>> >>>>> [1] >>>>> https://wiki.openjdk.java.net/display/Nashorn/Monitoring+Nashorn+P >>>>> erfor mance >>>>> >>>>> On Thu, Apr 30, 2015 at 9:18 AM, Marcus Lagergren >>>>> <marcus.lagerg...@oracle.com> wrote: >>>>> >>>>> >>>>>> >>>>>>> On 29 Apr 2015, at 09:08, Marcus Lagergren >>>>>>> <marcus.lagerg...@oracle.com> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> The benchmark code itself is public, but I donât know if >>>>>>> the benchmarks as part of JMH are. CC Aleksey /M >>>>>>> >>>>>>> >>>>>>>> On 28 Apr 2015, at 19:24, Chris Newland >>>>>>>> <cnewl...@chrisnewland.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> https://wiki.openjdk.java.net/display/Nashorn/Monitoring+Na >>>>>>>> shorn+ Performance >>>>>>>> indicates the JMH microbenchmarks for octane are internal to >>>>>>>> Oracle (VPN >>>>>>>> needed). >>>>>>>> >>>>>>>> I can't see them elsewhere in the OpenJDK repos, are they >>>>>>>> public? >>>>>>>> >>>>>>>> >>>>>>>> Asking because the results I'm getting from >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> nashorn/test/script/external/octane$ jjs run.js >>>>>>>> >>>>>>>> are wildly variable (as per above wiki page). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Chris >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>> >>> > >
