[
https://issues.apache.org/jira/browse/RNG-88?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16809761#comment-16809761
]
Alex D Herbert commented on RNG-88:
-----------------------------------
{quote}Just a BTW The JMH harness goes to great length in removing artifacts
from the actual measurements. You can add a baseline method here however I
would avoid trying to massage the returned results in any way. For relative
comparison it’s not needed anyway (and overhead should also be observable by
varrying request length if possible)
{quote}
Hi [~b.eckenfels],
I have studied the tutorials in JMH and tried to implement best practice.
Specifically this example on
[DeadCode|https://hg.openjdk.java.net/code-tools/jmh/file/f25ae8584db1/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_08_DeadCode.java]
elimination which is used to demonstrate that there is a measured time for
doing nothing.
The issue we have here is that we want to measure the time for generation of a
random number. So we ideally want to measure:
{noformat}
[time to generate nothing] + [time to consume the value]
[time to generate random number] + [time to consume the value]
{noformat}
Subtracting one from the other gives the [time to generate random number].
The main issue with not considering a baseline is that the random generation
time is very fast. It is so fast that the time taken by JMH to test a generated
value is much longer than the time to generate it.
Leaving the results unmassaged is fine for ranking. But is not representative
for relative speed comparisons.
The baseline subtraction is not part of the new code anyway. The new code in
the PR just eliminates the loop from within the test method which is another
JMH best practice (see [Loops
example|https://hg.openjdk.java.net/code-tools/jmh/file/f25ae8584db1/jmh-samples/src/main/java/org/openjdk/jmh/samples/JMHSample_11_Loops.java]).
I hope I have explained what I am trying to achieve. Your comments on the
results I have shown above and code in the PR are welcome.
> Update the GenerationPerformance benchmark
> ------------------------------------------
>
> Key: RNG-88
> URL: https://issues.apache.org/jira/browse/RNG-88
> Project: Commons RNG
> Issue Type: Improvement
> Components: examples
> Affects Versions: 1.3
> Reporter: Alex D Herbert
> Assignee: Alex D Herbert
> Priority: Minor
> Attachments: baseline.jpg, next.png
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The current GenerationPerformance benchmark runs all the generators to
> collect timing data. However the act of running the test within JMH takes
> some time (test overhead). This overhead should be measured and subtracted
> from the timing data to create a time attributed only to the method exercised
> in the RNG.
> This can be done by creating a dummy RNG that returns a fixed value. The
> implementation must be done in a manner where the JIT compiler is not able to
> remove the dummy method from the execution path. This is achieved by
> returning state variables from the dummy RNG (not final variables).
> Testing can be done using a variable number of iterations and the run-times
> assessed for linearity. If the run time scale with the number of iterations
> then the JIT compiler has not removed it from execution. The dummy RNG then
> serves as a baseline for comparison of true implementations.
> This idea with examples is shown in
> [RNG-87|https://issues.apache.org/jira/browse/RNG-87] which tested a variant
> of the MWC_256 algorithm.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
