Uugh uugh uugh.
Stuart, I (somewhat) disagree. The current way of running the tests is
a horrible artifact of our internal JPRT system, and the desire to split
the test load up across machines. It is not something I would
recommend sane person doing by choice.
I agree with the need to have coordination between the way the small
targets work and a bigger target, but the way to do this is to merge the
directories given on the jtreg command line, not try and merge the
output of multiple test runs.
-- Jon
On 01/15/2013 04:18 PM, Stuart Marks wrote:
Hi Balchandra,
What you've done seems reasonable given that you're just trying to get
the initial setup going.
I really think it would be preferable to use the actual makefile
targets instead of assembling a directory list. The reason is that the
makefile targets not only have lists of directories, they may also
specify various jtreg options differently depending upon the target.
For example, the jdk_rmi tests are all run in othervm mode (a run mode
of jtreg) which is different from most of the other makefile targets.
This information is only present in the makefile targets.
Would it be difficult to run all the makefile targets individually and
combine the results? Yes, there are (I think) 17 makefile targets, but
it wouldn't be too bad if the combination process could be automated.
My main concern is that if you end up running a different set of tests
from our internal nightly build and internal developer builds, the
results would be difficult to compare.
s'marks
On 1/14/13 4:02 AM, Balchandra Vaidya wrote:
I think you made all valid points. Here are some my observations:
Issue 1: As you described, jdk_all and jdk_default targets depends on
individual
targets and invoke jtreg once. The issue is, the make seems to exit
with Error
when
there is an error(or failure!) in any individual target! Effectively,
I never got
jdk_all and jdk_default targets completed start to finish.
Issue 2: jdk_all target includes awt and swing tests. There are some
instability
at the moment and difficult to get consistent results. Some awt tests
may required
to be run on OS console.
Issue 3: jdk_default target do not include targets such as jdk_bean1,
jdk_bean2,etc.
and those tests are good and runs without any issue. That is, using
jdk_default target
runs many fewer tests (See Issue 1, however) than we would like to run.
Therefore, I tried
1) running individual targets (wrapped in a shell script) and merging
test results
(jtreg -ro). I managed to get to run ~3600 tests.
2) selected dir.list and passed it to jtreg directly. I managed to
run ~4000
tests. So,
started to publish this option.
Obviously, this is not an ideal/clean solution because tests added
under a new
directory or under a new make target may not be executed. So,
centrally managed
'dir.list' (include stable directory) might help.
