Le mer. 20 mars 2019 à 15:04, Alex Herbert <alex.d.herb...@gmail.com> a écrit : > > > On 20/03/2019 13:08, Gilles Sadowski wrote: > > > >> - Perhaps there is another place for them? > >> > >> I am happy to not clutter the source repo and keep them somewhere for > >> devs. But where? You tell me. Maybe in the examples-stress module they > >> are OK as that one is not officially part of the library. > > Unless a script is robust and readily usable by a newbie (i.e. anyone > > but the author), there is the risk that it becomes cruft. > > What you did is great work but I doubt that many would be interested > > in detailed tables of the failures, once one can easily compare the number > > of failures. > > If more is needed, there is no shortcut to reading the doc of the test > > suites > > themselves and other resources on the web... > > > > So, to summarize, what I think is interesting is to make it easy to rerun > > the test suites and update the site (e.g. for the "current" JDK): > > * a document that mentions the external tools requirement > > * a standalone application for running "RandomStressTester" > > * a script that collects results from the above run, formats them into the > > "quality" table ready to be pasted into the "rng.apt" file, and copies > > the > > output files to their appropriate place (*not* assuming a git > > repository) > > > > It would also be great to be able to easily start the many benchmarks > > and similarly collects the results into the user guide tables. > > I will write a script that will collect the results from the benchmark, > build the apt table and copy the output files to a directory. > > Note that the benchmark files only contain the RNG class name: > > # RNG: org.apache.commons.rng.core.source32.JDKRandom > > This means the script has to be kept up to date with the corresponding > enum name, and the desired output order for the apt table. I can add > these to a simple array as the top which any newbie should be able to > extend. > > It would be desirable for the script to search for results files in a > set of directories, identify them as dieharder (dh) or TestU01 (tu) > results and then output the files for each run N of a unique generator M > to the existing directory structure, e.g: > > > collate-benchmark-results.pl output dir1 dir2 ... > > output/(dh|tu)/run_N/(dh|tu)_M > > This should be done robustly such that the script can be pointed at the > git source tree for the site and it will figure out that the files are > already in the correct place. I think this would just involve sorting > the paths to all the results numerically not alphabetically (e.g. dh_1 < > dh_2 < dh_10) before processing them to output.
Perhaps the "RandomStressTester" should be made more flexible and, instead of a hard-coded "GeneratorsList", accept a list of "enum" identifiers. e.g. a list (ASCII file) like: MT 3 MT_64 3 SPLIT_MIX_64 3 [etc.] where the second columns indicates how many runs to perform for this type of RNG. [A (mandatory) flag specifying this file would replace the first 2 command-line arguments.] > > For the JMH benchmarks I tend to run e.g. > > > mvn test -P benchmark -Dbenchmark=NextDoublePerformance > > I then have a script to parse the JSON in > target/jmh-result.NextDoublePerformance.json to a Jira format or CSV > format table. Usually I add some relative score columns in a spreadsheet > then paste into Jira (which nicely accepts the tables). > > It should not be too hard to generate the existing apt tables for > performance. I can look into this. I am assuming that the tables are > based on the highest 'number of samples' from each generator. I also have a (Perl) script, that uses a JSON library, to parse the JMH output; but it's quite ugly and not robust (IIRC, I need to make a change in the code depending on what should serve as the "reference" to be mapped to "1", in the table). Regards, Gilles > >>>> [...] > >>>> You then have a utility for dumping output of any random source to file > >>>> in a variety of formats. > >>>> > >>>> Although long output is not needed for the test suites it is useful for > >>>> native long generators. > >>>> > >>>> WDYT? > >>> Looks good! > >> OK. I will work on the raw data dumper as a Jira ticket. It is > >> encapsulated work that does not really effect anything else. > >> > >> > >> DieHarder has finished! > >> > >> I think my stupidity is what caused previous crashes. I was running the > >> stress test within the source tree and possibly git checkout onto > >> another branch makes some of the directory paths stale killing any > >> processes linked to those paths. I'll not do that again. > > Hence, the "standalone" application is the right choice it seems. > > > >> FYI: Here are the old results with incorrect byte order: > >> > >> XorShiftSerialComposite : 24, 25, 23 : 134.1 +/- 16.1 > >> XorShiftXorComposite : 88, 105, 89 : 396.2 +/- 9.9 > >> SplitXorComposite : 0, 0, 0 : 90.8 +/- 21.9 > >> > >> Here are the new results with correct byte order: > >> > >> XorShiftSerialComposite : 13, 15, 10 : 105.5 +/- 1.8 > >> XorShiftXorComposite : 57, 57, 57 : 102.9 +/- 1.5 > >> SplitXorComposite : 0, 0, 0 : 99.9 +/- 3.2 > >> > >> So interestingly passing the correct byte order lowers the number of > >> failures. There are still lots. > >> > >> And BigCrush (with the fix for passing the correct byte order): > >> > >> XorShiftSerialComposite : 40, 39, 39 : 608.2 +/- 3.9 > >> XorShiftXorComposite : 54, 53, 53 : 646.8 +/- 10.9 > >> SplitXorComposite : 0, 0, 0 : 625.8 +/- 0.2 > > Curious to know whether it is also affected by the byte ordering. > > I'll re-run BigCrush with the wrong byte ordering when I have updated > the stress test code. I finished it yesterday but will look it over with > fresh eyes before committing it. > > Alex --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
