On 1/16/13 3:48 AM, Balchandra Vaidya wrote:
I believe make targets get run during jprt build process and then RE will
create a full build. After that, I take RE build and run tests. So, I think,
running tests with slightly different option is not necessarily a problem -
rather I would think it confirm stability/quality of the tests! Moreover, at
least in theory, jtreg option itself should not alter test output (except test
failure with resource/logistical issues such as memory, timeout, etc.. ).
Assuming jtreg options are managed to avoid logistical issues, any failure
probably indicates requirement of manual intervention to check whether
any improvement possibility in test or product.
It's true, most jtreg options won't affect success or failure of the tests,
they mostly affect things like reporting and output. Some things are
significant though.
The agentvm/othervm difference could cause some tests to fail, but as Alan
pointed out in another message, we should extract that logic from the makefiles
and move it elsewhere, probably TEST.ROOT.
Looking through the makefile (see the stuff regarding JTREG_BASIC_OPTIONS)
there are a number of other significant options:
-a -ea -esa -ignore:quiet -timeoutFactor:4 -J-Xmx512m
Additional args may get passed in via JAVA_ARGS and JAVA_VM_ARGS, but I'm not
sure.
It's not necessarily a fatal problem if your test runs using a different set of
options. But I don't think it necessarily helps confirm the stability or
quality of the tests, either. The thing I'm concerned about is somebody
investigating a test that fails in your test run but not on his desktop or in
our nightly build. That failure might simply have been caused by running the
test in a different mode.
The good news is that these options don't change very often, so they're
unlikely to get out of date.
Meanwhile, in the short term, it might be possible for Balchandra to invoke
the 17(!) different makefile targets and merge the results.
There were 22 (excluding jdk_awt and jdk_swing) targets, and were running ~3600
tests combined.
But, I see some changes in Makefile targets since my last try. I will rerun
and see how many tests it
will run.
Ah, the other five targets are jdk_beans[123], jdk_jdi, and jdk_sound. Those
aren't included in the "default" testset. I'm surprised you got only 3,600
tests. Our runs of the default testset (the 17 targets) includes 3,900 tests.
s'marks
