Hi, Claes
On 9/19/19 1:40 PM, Claes Redestad wrote:
FYI, you can control the number of forks etc run by make test:
make test TEST="micro:java.lang.StackWalkBench"
MICRO="FORK=1;WARMUP_ITER=2;OPTIONS=-p depth=128"
(documented in doc/testing.md|html)
Neat, thanks!
Still, I agree it might be a good idea to have make test TEST=... run
fewer forks/iterations by default, since this shortcut was primarily
intended as a means for devs to quickly verify specific microbenchmarks.
OK, I've submitted:
https://bugs.openjdk.java.net/browse/JDK-8231315
However, reducing forks and iterations only cuts down one factor of the
total runtime: others being the number of methods and the number of
chosen parameters (as controlled by @Param-annotated fields).
So IMHO, what should be thought through carefully when adding micros is
whether all methods are necessary or at least useful and whether we can
reduce the number of parameter chosen.
For these two, I think picking 2 or 3 values for depth (lowest, highest,
maybe a midpoint) might be sufficient. This would cut total runtime
budget by 60% or 40%, respectively.
Mandy makes a good point that configurations would be different between
doing a quick performance verification/sanity check, versus a more
comprehensive run to measure tuning work or larger scale work.
In the interest of reducing the run time when using 'make', I'd be OK
with having fewer values in the 'depth' @Param in the source. It's easy
enough to add values back from the commandline for more thorough
testing. (In the case of StackWalker, it's good to exercise its various
batch sizes.)
Thanks,
-Brent
P.S.
As you alluded to, there is also a "quick-and-efficient versus thorough
coverage" tension WRT what @Benchmark methods get included/run. Worth
giving thought to sometime...
On 2019-09-19 20:09, Brent Christian wrote:
Hi, Mandy
Yes, that 'make' job would take ~7 hours on my machine.
I believe this is typical for running micros using 'make'. AFAICT,
the jmh defaults are used, so it forks (running all the warmups +
iterations) 5 times. Trying another (>100 loc) benchmark at random,
the java.lang.ArrayCopy one would take 25 hours.
There might be a way to pass jmh args (e.g. '-f 1') via 'make test
TEST=...', though I really only ever run benchmarks using 'java -jar
path/to/benchmark.jar'.
It may be worth looking into updating the make target to setup more
practical JMH runtimes by default. But I don't think this an issue
particular the new benchmarks.
Thanks,
-Brent
On 9/18/19 7:33 PM, Mandy Chung wrote:
Hi Brent,
$ make test TEST="micro:java.lang.StackWalkBench"
It took very long that I killed the job. Does this happen to you?
Mandy
On 9/13/19 3:07 PM, Brent Christian wrote:
Hi,
Please review these StackWalker and Throwable benchmarks for
addition into the JDK microbenchmarks.
Bug:
https://bugs.openjdk.java.net/browse/JDK-8221623
Webrev:
http://cr.openjdk.java.net/~bchristi/8221623/webrev07/
The StackWalker benchmarks use StackWalker's forEach(), walk(), and
Options to measure retrieval of various types of information from
the call stack.
The Throwable benchmarks do corresponding exercises; there are also
a couple of Logging benchmarks.
A JMH @Param is used to test a variety of call stack depths.
In the future, we might consider a benchmark for
Reflection.getCallerClass(). (It is more involved today to
benchmark that method than at the time these benchmarks were
originally written, so that one's commented out.) See JDK-8230976.
Thanks,
-Brent
1. https://bugs.openjdk.java.net/browse/JDK-8230976
