OK, thanks, I managed to run my first tier test. What I (sometimes) really want is to apply a test filter to my test set: - those tests that are non-flaky and don't take "too many" resources to run You could harvest both of those qualities from your many quality engineering test runs.
On Wed, Sep 16, 2015 at 6:32 PM, Joseph D. Darcy <joe.da...@oracle.com> wrote: > Hi Martin, > > On 9/16/2015 6:17 PM, Martin Buchholz wrote: > > Thanks Joe. > > I see those tier definitions in the various TEST.groups files but it's not > obvious to me how to run them all with one simple invocation (neither make > nor jtreg). > > > Make support is coming, but not there yet: > > JDK-8075571: Support tiered testing make targets > > > The command line you gave doesn't seem to tell jtreg which TEST.ROOT > directories to operate on. > > It looks like > jtreg ... langtools/test:tier1 jdk/test:tier1 > works, but I was looking for a simpler way to "run all the tier1 tests" > without enumerating all the roots (there are currently 7!) > > > That is the simple(st) way currently. > > The jdk, langtools, nashorn, and jaxp repos currently use the tiered > testing approach as defined in jtreg groups so you would "only" need four > roots. > > The tier 2 tests are pretty stable too, but we certainly have instances > were new repeatable failures are introduced, but we're getting better at > addressing them quickly :-) > > -Joe > > > > On Wed, Sep 16, 2015 at 5:25 PM, Joseph D. Darcy <joe.da...@oracle.com> > wrote: > >> Dropping the jtreg alias. >> >> >> On 9/16/2015 5:18 PM, Martin Buchholz wrote: >> >>> Just now for the first time I ran all the jtreg tests in the >>> "jdk_stable" test set using >>> make test TEST=jdk_stable >>> and I got around 50 failures out of 5000. >>> Is that expected? >>> >>> Instead, I expect that with all the work on the test suite y'all have >>> been doing, that the expected failure rate of something with a name like >>> "jdk_stable" would be 0. >>> >>> >> For JDK 9 dev, the tier 1 tests across repos should be stable. Currently, >> only the langtools and jdk repos have non-empty tier 1 tests sets defined. >> >> If you run >> >> jtreg ... -exclude:ProblemList.txt -k:intermittent :tier1 >> >> there are some tier 1 test that are known to fail intermittently, but at >> this point with very low frequency (at least with the testing I'm exposed >> to). >> >> HTH, >> >> -Joe >> >> > >
