[
https://issues.apache.org/jira/browse/RNG-88?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809742#comment-16809742
]
Alex D Herbert commented on RNG-88:
-----------------------------------
PS.
Note that the example for a benchmark class has two {{@Benchmark}} methods to
test JMH timing overhead. These should sum to approximately the same metric as
the BASELINE implementation. In the above case for {{nextDouble()}} here are
the results:
||randomSourceName||Method||Score||Error||Median||Error/Score||
|BASELINE|nextDouble|2.57593197585943|0.003460918835402205|2.57636702779831|0.00134355987185862|
|ISAAC|nextDouble|11.492061181491472|0.03105190366683462|11.4930378375858|0.00270203083471616|
|JDK|nextDouble|19.07710097015598|0.13765564275214587|19.0508121852192|0.00721575269573155|
|KISS|nextDouble|9.205196709268291|0.006192317077079371|9.20614509815848|0.000672697963189055|
|MT|nextDouble|10.653521635100258|0.010420185228341606|10.6524402774538|0.00097809772066451|
|MT_64|nextDouble|6.8707343968410886|0.04118236194818618|6.8627989246393|0.00599388064936993|
|MWC_256|nextDouble|6.582187644902875|0.05365540468787355|6.57988293788998|0.00815160666673234|
|SPLIT_MIX_64|nextDouble|4.5286626926183935|0.008298205686561705|4.52667047056436|0.0018323744226055|
|TWO_CMRES|nextDouble|5.359469359437292|0.050123182243253586|5.34417676563963|0.00935226584605685|
|WELL_1024_A|nextDouble|13.632475941306263|2.3697998386644143|12.8807508310923|0.173834881416071|
|WELL_19937_A|nextDouble|18.45089096484812|0.03811822635555045|18.4441767343849|0.00206592876344951|
|WELL_19937_C|nextDouble|19.972261377408604|0.02239912024952394|19.9724925126603|0.0011215114716484|
|WELL_44497_A|nextDouble|19.858159613693175|0.0871231306453897|19.8435844247295|0.00438727114396412|
|WELL_44497_B|nextDouble|20.725319834240956|0.14352086213185533|20.7110885030745|0.00692490457468068|
|WELL_512_A|nextDouble|12.967886115136594|0.023823029376018285|12.9610798024776|0.00183707885498865|
|XOR_SHIFT_1024_S|nextDouble|5.3340350544353825|0.005344749168553254|5.33227602956032|0.00100200863211594|
|XOR_SHIFT_1024_S_PHI|nextDouble|5.5063780535656885|0.7355642816548434|5.34975699418824|0.133584050077805|
|
|baselineDouble|2.3360076674923396|0.31171210270144445|2.26776285101707|0.133437962143361|
|
|baselineVoid|0.2657420971315234|3.32183670742148E-4|0.265707936583524|0.00125002276390459|
So:
{noformat}
baselineDouble + baselineVoid ~ BASELINE
2.33 + 0.27 ~ 2.57
{noformat}
This shows the baseline implementation is being run.
Note that the Error output by JMH is is the 0.999 percentile of the
T-distribution with n-degrees of freedom multiplied by the standard error of
the mean. This is a conservative +/- limit on the true value of the mean.
Dividing the Error by the Score gives a quick check to see if there is a lot of
variation in the mean.
In this example WELL_1024_A has a single measurement that is 16.5, which is
above the usual 12.9 time. Hence using the median for analysis to avoid skew
from single outliers.
> Update the GenerationPerformance benchmark
> ------------------------------------------
>
> Key: RNG-88
> URL: https://issues.apache.org/jira/browse/RNG-88
> Project: Commons RNG
> Issue Type: Improvement
> Components: examples
> Affects Versions: 1.3
> Reporter: Alex D Herbert
> Assignee: Alex D Herbert
> Priority: Minor
> Attachments: baseline.jpg, next.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The current GenerationPerformance benchmark runs all the generators to
> collect timing data. However the act of running the test within JMH takes
> some time (test overhead). This overhead should be measured and subtracted
> from the timing data to create a time attributed only to the method exercised
> in the RNG.
> This can be done by creating a dummy RNG that returns a fixed value. The
> implementation must be done in a manner where the JIT compiler is not able to
> remove the dummy method from the execution path. This is achieved by
> returning state variables from the dummy RNG (not final variables).
> Testing can be done using a variable number of iterations and the run-times
> assessed for linearity. If the run time scale with the number of iterations
> then the JIT compiler has not removed it from execution. The dummy RNG then
> serves as a baseline for comparison of true implementations.
> This idea with examples is shown in
> [RNG-87|https://issues.apache.org/jira/browse/RNG-87] which tested a variant
> of the MWC_256 algorithm.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
