On 7/7/20 7:51 PM, Roger Riggs wrote:
Hi,
I have found it very useful to be able to run benchmarks against multiple
versions of the JDK. Build the benchmark jar once and to compare
results.
If all of the classes are built with --enable-preview, none of them
will run
against older JDKs. So an approach that only compiles those files that
need
preview would be more useful.
...
Or perhaps, separate out preview based benchmarks
into a separate jar.
$.02, Roger
Perhaps the last option is the way to go. Why? Preview features are
specific to a major OpenJDK release so while a particular preview
feature may be present as a preview feature in two consecutive releases,
you can not run a class with the preview feature compiled for OpenJDK N
on an OpenJDK N+1 or vice versa. So a single benchmarks.jar file makes
sense only for benchmarks that don't use preview features and are
compiled with (or for) lowest release possible...
The benchmarks using preview feature OTOH will need to be re-classified
once the preview feature graduates and becomes a mainline feature. So
once it is classified differently in OpenJDK N+1 as it was in OpenJDK N,
it can not be built with such classification on (or for) the OpenJDK N
release any more because, among other things, names of packages/modules
may change. So these two sets of benchmarks (using or not using preview
features) have different constraints and is therefore reasonable for
them to be built separately and packed into separate benchmarks.jar files.
Packing to separate .jar files might be a pragmatic solution for a
problem that Jorn Vernee discovered: the JMH compiler plugin generates
two files in the output directory which are included in the benchmarks.jar:
/META-INF/BenchmarkList
/META-INF/CompilerHints
While the 1st one is "updated" incrementally if it already exists, its
modification is not protected by any kind of locking mechanism and so
concurrent compilation by two or more instances of javac may produce
garbled result. The 2nd one seems to be overwritten entirely by content
of a single compile session, so its final form does not represent
compiler hints for all aggregated benchmarks and may therefore produce
skewed results for some benchmarks. I see two solutions for that problem:
- fix JMH to correctly handle concurrent incremental updates to both
above files and; or
- compile the two sets of benchmarks separately with separate output
directories and create separate benchmarks.jar files for them
I think the 2nd option is a simpler, pragmatic solution.
Peter
On 7/7/20 10:26 AM, Peter Levart wrote:
I suggest adding --enable-preview to JMH_JVM_ARGS in general now (it
doesn't hurt even if classes are not compiled with --enable-preview)
and then take time to devise an effective strategy for selectively
compiling micro benchmarks with or without --enable-preview. At least
so the benchmarks would work out-of-the-box when run via make test.
WDYT?
Regards, Peter
On 6/30/20 10:15 PM, Claes Redestad wrote:
On 2020-06-30 22:12, Magnus Ihse Bursie wrote:
Second to that a solution in the build would be preferable - if we
can
come up with something that has infinitesimal impact to build times.
Are we talking about many files? Could you consider listing those
files explicitly in the makefile? That would make it cheap to
filter them out from the normal compilation, and instead do a
secondary compilation with them.
Right now there's one micro using --enable-preview, so that'd be a very
short list.
/Claes
