On 01/14/13 07:36 AM, Stuart Marks wrote:
On 1/11/13 2:54 PM, Jonathan Gibbons wrote:
I suggest there should be a new test/Makefile target for "run all
recommended
tests in a single jtreg run".
I think this would be ideal. Implicitly, then, Balchandra's script
would just invoke this makefile target, as would other internal
build/test systems. This would eliminate copying of information about
these targets out of the Makefiles into external scripts, where
they'll inevitably get out of date.
I say this is ideal, but this is probably more difficult to achieve
than one might think. Not impossible, but probably somewhat tedious.
The "test sets" are defined redundantly in *two* properties files,
TOP/make/jprt.properties and TOP/jdk/make/jprt.properties. They are
mostly the same, though there are some small differences. I'm not sure
if that's intentional. Note also that these aren't plain properties
files; they use some kind of variable interpolation and string
substitution syntax I'm not familiar with.
Then, there are the makefiles. Oh, the makefiles. Mainly, they are
TOP/test/Makefile, TOP/jdk/test/Makefile, and
TOP/langtools/test/Makefile. (I don't know how the JVM tests are
invoked; probably TOP/hotspot/test/Makefile.)
It looks to me like each of the Makefiles defines several individual
test targets, each of which invokes a run of jtreg. Most of these are
in the jdk repo. There seem to be targets that invoke "all" of the
tests (but not test sets), but I think the "all tests" target just
depends on the individual targets, so it'll still invoke jtreg once
for each individual target.
Some cleaning up and rearrangement is called for here. There is some
redundancy, but it would be preferable to have all the redundancy
within files in the forest (as painful as it is) than to have
redundant information copied into external scripts.
In short, it's a mess.
I think you made all valid points. Here are some my observations:
Issue 1: As you described, jdk_all and jdk_default targets depends on
individual
targets and invoke jtreg once. The issue is, the make seems to exit with
Error when
there is an error(or failure!) in any individual target! Effectively, I
never got
jdk_all and jdk_default targets completed start to finish.
Issue 2: jdk_all target includes awt and swing tests. There are some
instability
at the moment and difficult to get consistent results. Some awt tests
may required
to be run on OS console.
Issue 3: jdk_default target do not include targets such as jdk_bean1,
jdk_bean2,etc.
and those tests are good and runs without any issue. That is, using
jdk_default target
runs many fewer tests (See Issue 1, however) than we would like to run.
Therefore, I tried
1) running individual targets (wrapped in a shell script) and merging
test results
(jtreg -ro). I managed to get to run ~3600 tests.
2) selected dir.list and passed it to jtreg directly. I managed to run
~4000 tests. So,
started to publish this option.
Obviously, this is not an ideal/clean solution because tests added under
a new
directory or under a new make target may not be executed. So, centrally
managed
'dir.list' (include stable directory) might help.
Thanks
Balchandra
s'marks
