Hi Balchandra,
What you've done seems reasonable given that you're just trying to get the
initial setup going.
I really think it would be preferable to use the actual makefile targets
instead of assembling a directory list. The reason is that the makefile targets
not only have lists of directories, they may also specify various jtreg options
differently depending upon the target. For example, the jdk_rmi tests are all
run in othervm mode (a run mode of jtreg) which is different from most of the
other makefile targets. This information is only present in the makefile targets.
Would it be difficult to run all the makefile targets individually and combine
the results? Yes, there are (I think) 17 makefile targets, but it wouldn't be
too bad if the combination process could be automated.
My main concern is that if you end up running a different set of tests from our
internal nightly build and internal developer builds, the results would be
difficult to compare.
s'marks
On 1/14/13 4:02 AM, Balchandra Vaidya wrote:
I think you made all valid points. Here are some my observations:
Issue 1: As you described, jdk_all and jdk_default targets depends on individual
targets and invoke jtreg once. The issue is, the make seems to exit with Error
when
there is an error(or failure!) in any individual target! Effectively, I never
got
jdk_all and jdk_default targets completed start to finish.
Issue 2: jdk_all target includes awt and swing tests. There are some
instability
at the moment and difficult to get consistent results. Some awt tests may
required
to be run on OS console.
Issue 3: jdk_default target do not include targets such as jdk_bean1,
jdk_bean2,etc.
and those tests are good and runs without any issue. That is, using
jdk_default target
runs many fewer tests (See Issue 1, however) than we would like to run.
Therefore, I tried
1) running individual targets (wrapped in a shell script) and merging test
results
(jtreg -ro). I managed to get to run ~3600 tests.
2) selected dir.list and passed it to jtreg directly. I managed to run ~4000
tests. So,
started to publish this option.
Obviously, this is not an ideal/clean solution because tests added under a new
directory or under a new make target may not be executed. So, centrally managed
'dir.list' (include stable directory) might help.
