On 12/11/13 07:50 PM, Jonathan Gibbons wrote:
Balchandra,
Thanks for posting the commands. That gives us something to reason about.
For my part, I have found -conc:auto to be too generous. "auto" gets
evaluated to the number of processors on the host system as seen by
the Java runtime. In my usage (on langtools) I have found a good rule
of thumb to be half that number, so on a 32-way system, I use
-conc:16, etc. It is also a function of available memory.
I played with the idea of supporting expressions using auto, such as
-conc:auto/2 or -conc:auto-1 -- but then I came to my senses. This
calculation is much better done in the environment used to run jtreg.
I notice in your commands, you do NOT limit the amount of memory for
each JVM with -Xmx. That leaves all your JVMs using default
ergonomics, which may be unduly optimistic when you have lots of JVMs
running.
As a general rule for anyone using -conc to run tests concurrently, I
strongly recommend that the first few times you do this, use your
system activity profiling tool to monitor CPU and memory usage. If
all your CPUs are redlining it at 100%, or if you start memory
swapping, then your concurrency number is too high. I would recommend
that you target an average CPU utilization just below 100%, and no
memory swapping.
It is a good information.
First, I usually run tests on 4 way and 8 way systems. I believe the
objective here is to provide repeatable test results. But, the
hardware availability, OS versions, kernel, patches, window manager,
jtreg options, and many other factors play a role to produce
consistent test results. The truth is we may not able to produce
results using all combinations, but the results posted can reasonably
be used for a comparison. That is, the result posted is not
everything/unbreakable.
Is it fair to say something like "The test results posted here used
-conc:2 -Xmx:512m ..... options. Result could vary if other options
were used. However, the community members are encouraged
to send email to quality-discuss or file bugs if any issues found". If
a testcase pass with -conc:2 but fail with -conc:4 means there
might be a bug in the testcase - such failures are good to know.
Second point is to make test execution easier. Does it make sense
to add default options (concurrency, memory etc.) to jtreg? Or does
it make sense to create a separate environment files (similar to
what is used in JCK) and read those files in jtreg or other wrapper
script? However, I also think that are such micro tuning necessary?
Testing different combinations may find bugs which is beneficial to
OpenJDK.
A couple more comments inline.
-- Jon
P.S. I'll see about converting some of these notes and guidelines
into another jtreg page on OpenJDK.
-- Jon
On 12/11/2013 11:26 AM, Balchandra Vaidya wrote:
On 11/12/2013 17:55, Jonathan Gibbons wrote:
Balchandra,
The important part of that of interest here is the part about "calls
jtreg commands
(one each for jdk, langtools and hotspot) with those recommended
options".
Is there a way you could post here the segment of the script that
does that?
Here it is
2.1 Running tests in jdk/test
$ jtreg -dir:{openjdk source top directory}/jdk/test -verbose:summary
-exclude:{openjdk source top directory}/jdk/test/ProblemList.txt
-conc:auto -a -ignore:quiet -timeoutFactor:5 -othervm
-testjdk:{location of the test jdk} `cat dir.list
<http://download.java.net/jdk8/testresults/docs/dir.list>`
2.2 Running tests in langtools/test
$ jtreg -dir:{openjdk source top directory}/langtools/test
-verbose:summary -conc:auto -a -ignore:quiet -timeoutFactor:5
-agentvm -testjdk:{location of the test jdk} com tools
2.3 Running tests in hotspot/test
$ jtreg -dir:{openjdk source top directory}/hotspot/test
-verbose:summary -conc:auto -a -ignore:quiet -timeoutFactor:5
-agentvm -testjdk:{location of the test jdk} compiler gc runtime
sanity serviceability
The instructions is at
http://download.java.net/jdk8/testresults/docs/howtoruntests.html
and is linked from
http://www.java.net/download/jdk8/testresults/testresults.html
The caveat in this approach is a human error where one (me) forget to
update the instruction
or forget to remove any additional flags used in the script
temporarily - a mismatch
of the results could occur.
From the next build (b120), I am going to update "2.1 Running tests
in jdk/test" to
$ jtreg -dir:{openjdk source top directory}/jdk/test -verbose:summary
-exclude:{openjdk source top directory}/jdk/test/ProblemList.txt
-conc:auto -a -ignore:quiet -timeoutFactor:5 -agentvm
-testjdk:{location of the test jdk} :jdk_core :jdk_svc :jdk_beans
:jdk_imageio :jdk_sound :jdk_sctp javax/accessibility
com/sun/java/swing javax/print sun/pisces com/sun/awt
Generally, I would say that if you need to put a list of groups on the
command line, you need to update the groups file to create a new group
that is composed of the individual groups you want to run.
I think it would be good to have a group such as :jdk_default or
something like that.
Yes, it is a good idea.
Alternatively, that segment of the script could be a candidate for a
target in
one or more test/Makefile files.
This is good idea, but my experience with the 'make' is that if one
target critically fail, all
subsequent targets will not run. I thought it is a restriction of 'make'.
Well, ...
a) "make -k" keeps on going
b) as a user of the Makefile, you should be looking for a single
target, such as "make test", and it is up to the implementation of the
target to run all the tests, even though some might fail. There are
various ways to do this: for example, you can prefix a make rule with
"-" to ignore the exit code from a command, or you can do a composite
command like
jtreg <opts> || echo some jtreg tests failed
I will try again. I remember "make -k" didn't work from start to end. That
was an year ago, so I will retry. Is it time to say goodbye to 'make'
and say
hello to 'ant/gradle' for jdk9 ?
Thanks
Balchandra
Thanks
Balchandra
-- Jon
