Jon,
I (somewhat) agree! :-)
The current state of affairs is as you say, merely an artifact of our internal
build system. It would indeed be preferable for Balchandra to gather up the
right set of tests and run them in a single run of jtreg.
The problem is that there is some information encoded in the makefile targets
that would be lost if that were done. It's not just the lists of directories.
Some of the targets run the tests in othervm mode, and some in agentvm mode.
Some of the targets call a SharedLibraryPermissions macro that does some magic.
(Hm, see big comment at line 670 of jdk/test/Makefile.) The makefile targets
also invoke jtreg with a specific set of options, including using the problem
list as the exclude list.
I'd like to avoid having this information extracted from the makefiles and
copied into external scripts somewhere. They'll eventually diverge, resulting
in confusion.
Ideally there would be some makefile hackery to emit this information in a form
directly usable by outside scripts, but that will take some time.
Maybe some of the information could be extracted manually and placed into the
repo itself, in a fashion that external scripts can use. This is kind of like
the duplication between jprt.properties and the makefiles. This is unpleasant,
but at least it confines the duplication to be within the repo.
Meanwhile, in the short term, it might be possible for Balchandra to invoke the
17(!) different makefile targets and merge the results. This is also
unpleasant, but if it's not too much trouble, it might be reasonable to do. On
the other hand, if it turns into a real pain for some reason or another, maybe
we needn't bother.
s'marks
On 1/15/13 4:32 PM, Jonathan Gibbons wrote:
Uugh uugh uugh.
Stuart, I (somewhat) disagree. The current way of running the tests is a
horrible artifact of our internal JPRT system, and the desire to split the test
load up across machines. It is not something I would recommend sane person
doing by choice.
I agree with the need to have coordination between the way the small targets
work and a bigger target, but the way to do this is to merge the directories
given on the jtreg command line, not try and merge the output of multiple test
runs.
-- Jon
On 01/15/2013 04:18 PM, Stuart Marks wrote:
Hi Balchandra,
What you've done seems reasonable given that you're just trying to get the
initial setup going.
I really think it would be preferable to use the actual makefile targets
instead of assembling a directory list. The reason is that the makefile
targets not only have lists of directories, they may also specify various
jtreg options differently depending upon the target. For example, the jdk_rmi
tests are all run in othervm mode (a run mode of jtreg) which is different
from most of the other makefile targets. This information is only present in
the makefile targets.
Would it be difficult to run all the makefile targets individually and
combine the results? Yes, there are (I think) 17 makefile targets, but it
wouldn't be too bad if the combination process could be automated.
My main concern is that if you end up running a different set of tests from
our internal nightly build and internal developer builds, the results would
be difficult to compare.
s'marks
On 1/14/13 4:02 AM, Balchandra Vaidya wrote:
I think you made all valid points. Here are some my observations:
Issue 1: As you described, jdk_all and jdk_default targets depends on
individual
targets and invoke jtreg once. The issue is, the make seems to exit with Error
when
there is an error(or failure!) in any individual target! Effectively, I
never got
jdk_all and jdk_default targets completed start to finish.
Issue 2: jdk_all target includes awt and swing tests. There are some
instability
at the moment and difficult to get consistent results. Some awt tests may
required
to be run on OS console.
Issue 3: jdk_default target do not include targets such as jdk_bean1,
jdk_bean2,etc.
and those tests are good and runs without any issue. That is, using
jdk_default target
runs many fewer tests (See Issue 1, however) than we would like to run.
Therefore, I tried
1) running individual targets (wrapped in a shell script) and merging test
results
(jtreg -ro). I managed to get to run ~3600 tests.
2) selected dir.list and passed it to jtreg directly. I managed to run ~4000
tests. So,
started to publish this option.
Obviously, this is not an ideal/clean solution because tests added under a new
directory or under a new make target may not be executed. So, centrally
managed
'dir.list' (include stable directory) might help.
