On Thu, Dec 12, 2013 at 11:49 AM, Alan Bateman <alan.bate...@oracle.com> wrote:
> On 11/12/2013 23:10, Jonathan Gibbons wrote:
>
> :
>
> Yes, the high order point I was trying to make is that something is wrong if
> you need to specify a long list of tests to run. While we all may take
> whatever short cuts we choose to get our day to day work done, there should
> be a standard set of tests[1] that we agree should be run, and which can be
> run with reasonably concise command line args.
>
> -- Jon
>
> Ideally, "all" but maybe we're not there yet.
>
> Suppose we create a jdk_stable (or other name) group in TEST.groups for what
> "we" consider are the stable tests. Once it is defined in the groups file
> then it means it can be used by anyone that runs jtreg directly or anyone
> that uses the make file to run tests ("make test TEST=jdk_stable" for
> example). Whether it deserves its own make file is another question.
>
> So suppose we create such a group then what would be the criteria to be in
> that group? Clearly the test should be stable in the sense that it should
> pass when we don't have a bug. It should also clean up after itself. Things
> that come to mind are:
>
> - should be usable with -agentvm? We have /othervm option for @run and we
> also have othervm.dirs, the main point is that they can be run with either
> in othervm or agentvm modes and they should just work. I have deliberately
> not mentioned -samevm here as it's not suitable for the jdk tests.
>
> - needs to work with -concurrency, assuming sufficient resources. Is that
> reasonable to require? We have /exclusive and exclusiveAccess.dirs available
> for areas that have issues.
>
> - needs to run in a reasonable time. I don't think we have guidelines for
> what is reasonable in the jdk tests but clearly a test that runs for more
> than a few minutes needs to be looked at.
>
> - needs to run headless? Maybe this is controversial but it is somewhat
> appealing to skip Xvfb or other setup. Also being selfish, I'd like to run
> tests in a terminal window and not have windows dancing on my desktop.
>
What about having jdk_stable_headless, jdk_stable_headfull and
jdk_stable_all beeing the union of these two?
> - should not require special configuration? Maybe this is controversial too
> but there are tests javax/print that fail when there isn't a printer
> configured.
>
> Anything else? If we do something like this then such a group would need to
> be maintained, at least for period until all tests are stable and fast.
>
> -Alan.
>
