On 11/12/2013 23:10, Jonathan Gibbons wrote:
:
Yes, the high order point I was trying to make is that something is
wrong if you need to specify a long list of tests to run. While we
all may take whatever short cuts we choose to get our day to day work
done, there should be a standard set of tests[1] that we agree should
be run, and which can be run with reasonably concise command line args.
-- Jon
Ideally, "all" but maybe we're not there yet.
Suppose we create a jdk_stable (or other name) group in TEST.groups for
what "we" consider are the stable tests. Once it is defined in the
groups file then it means it can be used by anyone that runs jtreg
directly or anyone that uses the make file to run tests ("make test
TEST=jdk_stable" for example). Whether it deserves its own make file is
another question.
So suppose we create such a group then what would be the criteria to be
in that group? Clearly the test should be stable in the sense that it
should pass when we don't have a bug. It should also clean up after
itself. Things that come to mind are:
- should be usable with -agentvm? We have /othervm option for @run and
we also have othervm.dirs, the main point is that they can be run with
either in othervm or agentvm modes and they should just work. I have
deliberately not mentioned -samevm here as it's not suitable for the jdk
tests.
- needs to work with -concurrency, assuming sufficient resources. Is
that reasonable to require? We have /exclusive and exclusiveAccess.dirs
available for areas that have issues.
- needs to run in a reasonable time. I don't think we have guidelines
for what is reasonable in the jdk tests but clearly a test that runs for
more than a few minutes needs to be looked at.
- needs to run headless? Maybe this is controversial but it is somewhat
appealing to skip Xvfb or other setup. Also being selfish, I'd like to
run tests in a terminal window and not have windows dancing on my desktop.
- should not require special configuration? Maybe this is controversial
too but there are tests javax/print that fail when there isn't a printer
configured.
Anything else? If we do something like this then such a group would need
to be maintained, at least for period until all tests are stable and fast.
-Alan.
