On 01/16/13 02:58 AM, Stuart Marks wrote:
Jon,
I (somewhat) agree! :-)
The current state of affairs is as you say, merely an artifact of our
internal build system. It would indeed be preferable for Balchandra to
gather up the right set of tests and run them in a single run of jtreg.
The problem is that there is some information encoded in the makefile
targets that would be lost if that were done. It's not just the lists
of directories. Some of the targets run the tests in othervm mode, and
some in agentvm mode. Some of the targets call a
SharedLibraryPermissions macro that does some magic. (Hm, see big
comment at line 670 of jdk/test/Makefile.) The makefile targets also
invoke jtreg with a specific set of options, including using the
problem list as the exclude list.
I believe make targets get run during jprt build process and then RE will
create a full build. After that, I take RE build and run tests. So, I
think,
running tests with slightly different option is not necessarily a problem -
rather I would think it confirm stability/quality of the tests!
Moreover, at
least in theory, jtreg option itself should not alter test output
(except test
failure with resource/logistical issues such as memory, timeout, etc.. ).
Assuming jtreg options are managed to avoid logistical issues, any failure
probably indicates requirement of manual intervention to check whether
any improvement possibility in test or product.
I'd like to avoid having this information extracted from the makefiles
and copied into external scripts somewhere. They'll eventually
diverge, resulting in confusion.
Ideally there would be some makefile hackery to emit this information
in a form directly usable by outside scripts, but that will take some
time.
Maybe some of the information could be extracted manually and placed
into the repo itself, in a fashion that external scripts can use. This
is kind of like the duplication between jprt.properties and the
makefiles. This is unpleasant, but at least it confines the
duplication to be within the repo.
Meanwhile, in the short term, it might be possible for Balchandra to
invoke the 17(!) different makefile targets and merge the results.
There were 22 (excluding jdk_awt and jdk_swing) targets, and were
running ~3600 tests combined.
But, I see some changes in Makefile targets since my last try. I will
rerun and see how many tests it
will run.
Thanks
Balchandra
This is also unpleasant, but if it's not too much trouble, it might be
reasonable to do. On the other hand, if it turns into a real pain for
some reason or another, maybe we needn't bother.
s'marks
On 1/15/13 4:32 PM, Jonathan Gibbons wrote:
Uugh uugh uugh.
Stuart, I (somewhat) disagree. The current way of running the tests
is a
horrible artifact of our internal JPRT system, and the desire to
split the test
load up across machines. It is not something I would recommend sane
person
doing by choice.
I agree with the need to have coordination between the way the small
targets
work and a bigger target, but the way to do this is to merge the
directories
given on the jtreg command line, not try and merge the output of
multiple test
runs.
-- Jon
On 01/15/2013 04:18 PM, Stuart Marks wrote:
Hi Balchandra,
What you've done seems reasonable given that you're just trying to
get the
initial setup going.
I really think it would be preferable to use the actual makefile
targets
instead of assembling a directory list. The reason is that the makefile
targets not only have lists of directories, they may also specify
various
jtreg options differently depending upon the target. For example,
the jdk_rmi
tests are all run in othervm mode (a run mode of jtreg) which is
different
from most of the other makefile targets. This information is only
present in
the makefile targets.
Would it be difficult to run all the makefile targets individually and
combine the results? Yes, there are (I think) 17 makefile targets,
but it
wouldn't be too bad if the combination process could be automated.
My main concern is that if you end up running a different set of
tests from
our internal nightly build and internal developer builds, the
results would
be difficult to compare.
s'marks
On 1/14/13 4:02 AM, Balchandra Vaidya wrote:
I think you made all valid points. Here are some my observations:
Issue 1: As you described, jdk_all and jdk_default targets depends on
individual
targets and invoke jtreg once. The issue is, the make seems to exit
with Error
when
there is an error(or failure!) in any individual target!
Effectively, I
never got
jdk_all and jdk_default targets completed start to finish.
Issue 2: jdk_all target includes awt and swing tests. There are some
instability
at the moment and difficult to get consistent results. Some awt
tests may
required
to be run on OS console.
Issue 3: jdk_default target do not include targets such as jdk_bean1,
jdk_bean2,etc.
and those tests are good and runs without any issue. That is, using
jdk_default target
runs many fewer tests (See Issue 1, however) than we would like to
run.
Therefore, I tried
1) running individual targets (wrapped in a shell script) and
merging test
results
(jtreg -ro). I managed to get to run ~3600 tests.
2) selected dir.list and passed it to jtreg directly. I managed to
run ~4000
tests. So,
started to publish this option.
Obviously, this is not an ideal/clean solution because tests added
under a new
directory or under a new make target may not be executed. So,
centrally
managed
'dir.list' (include stable directory) might help.
