Alan Bateman wrote:
Kelly O'Hair wrote:
I need some formal reviewers on these jdk/test/Makefile changes.
More polish and changes may need to be done later, but there
is value in what I have now, and I need lots of help to improve
things (and fix some of our testcases).
Here is the bugid and webrev:
6888927: Fix jdk jtreg tests to indicate which ones need othervm,
allow for use of samevm option
http://cr.openjdk.java.net/~ohair/openjdk7/jdk7-build-samevm-6888927/webrev/
The goal was to see if we could easily run most of the regression
tests in the jdk/test directory, in minimum time, and without so
much noise (e.g. do not run unstable or problematic tests).
This could then be used as a benchmark to validate some quality measure
of the jdk7 that was built.
The primary changes include:
* New jdk/test/Makefile targets: jdk_all jdk_lang jdk_util jdk_nio ...
* Addition of a jdk/test/ProblemList file to be used by
jdk/test/Makefile
I tried to group all the tests (by directory names) into jdk_* sets, then
tried to balance them as to how long each set ran and if they could be
run
with the jtreg -samevm option. The balancing was tricky, and will need
more
polish, along with the virtual frame buffer idea. I also had to give up
on the awt, rmi, and swing tests until the Xvfb issues are figured out.
I chose to not actually modify the tests themselves if they needed
fixing,
it was just too overwhelming. So developers and teams may want to
browse the
ProblemList for their favorite tests (which aren't my favorites ;^) and
consider what they might want to do.
With this new jdk/test/Makefile, anybody can:
cd jdk/test
gnumake [PRODUCT_HOME=${YOUR_JDK7_HOME}]
[JT_HOME=${YOUR_JTREG_HOME}] jdk_all
And run about 3,000+ tests that SHOULD PASS in roughly 2 hours if your
machine isn't too slow and old. I was able to run it in 90 minutes on a
monster OpenSolaris AMD machine which had 16Gb RAM and 16 2.2Ghz cpus.
Then I added "-j 4" to the gnumake command line and it only took 30mins!
So the various jdk_* make targets can be run in parallel.
Please let me know what you think.
-kto
This looks really good. I don't see any obvious issues in the Makefile
and I grabbed the patch to try it and it worked great.
My only concern is that ProblemList will require ongoing maintenance. I
realize you don't want to get into changing tests and adding @ignore to
unstable tests but I think you'll need to send a broadcast mail to make
sure that folks know that their tests have problems in samevm mode, that
they need to remove from ProblemList when fixing tests, maybe they need
to add to ProblemList if changing/adding tests that take a long time, etc.
Yeah, but I didn't feel like I had many good solutions to this issue.
I'll maintain it to some degree, but I need the people that use this
makefile to help me out, and for everyone to start looking at why
their tests are even on this list.
For the groupings, then the only odd one I noticed is that jdk_io
includes the javax/imagio tests. It would best to separate these as they
are very different areas.
It was more of a balancing act on the test groupings with regards to timings
etc., and imageio did have "io" in it's name :^), and most of them ran safely
in samevm mode. So we can move them to any other group that is run in samevm
mode. Do you have a better match than jdk_io?
-kto
-Alan.
