Hi Martin,
On 9/16/2015 6:17 PM, Martin Buchholz wrote:
Thanks Joe.
I see those tier definitions in the various TEST.groups files but it's
not obvious to me how to run them all with one simple invocation
(neither make nor jtreg).
Make support is coming, but not there yet:
JDK-8075571: Support tiered testing make targets
The command line you gave doesn't seem to tell jtreg which TEST.ROOT
directories to operate on.
It looks like
jtreg ... langtools/test:tier1 jdk/test:tier1
works, but I was looking for a simpler way to "run all the tier1
tests" without enumerating all the roots (there are currently 7!)
That is the simple(st) way currently.
The jdk, langtools, nashorn, and jaxp repos currently use the tiered
testing approach as defined in jtreg groups so you would "only" need
four roots.
The tier 2 tests are pretty stable too, but we certainly have instances
were new repeatable failures are introduced, but we're getting better at
addressing them quickly :-)
-Joe
On Wed, Sep 16, 2015 at 5:25 PM, Joseph D. Darcy <joe.da...@oracle.com
<mailto:joe.da...@oracle.com>> wrote:
Dropping the jtreg alias.
On 9/16/2015 5:18 PM, Martin Buchholz wrote:
Just now for the first time I ran all the jtreg tests in the
"jdk_stable" test set using
make test TEST=jdk_stable
and I got around 50 failures out of 5000.
Is that expected?
Instead, I expect that with all the work on the test suite
y'all have been doing, that the expected failure rate of
something with a name like "jdk_stable" would be 0.
For JDK 9 dev, the tier 1 tests across repos should be stable.
Currently, only the langtools and jdk repos have non-empty tier 1
tests sets defined.
If you run
jtreg ... -exclude:ProblemList.txt -k:intermittent :tier1
there are some tier 1 test that are known to fail intermittently,
but at this point with very low frequency (at least with the
testing I'm exposed to).
HTH,
-Joe
