PS For reference, see the tiered testing policy discussion for JDK 9
earlier this year:
"Proposed new policies for JDK 9 regression tests: tiered testing,
intermittent failures, and randomness,"
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
and the subsequent refinements and amendments:
"Proposed new policies for JDK 9 regression tests: tiered testing,
intermittent failures, and randomness,"
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-April/002164.html
"Test policy follow-up, third testing tier,"
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-June/002325.html
Implementing these polices has found and corrected a number of lingering
test and product issues including:
JDK-8022224: Rare bug in JISAutodetect charset detected by
FindDecoderBugs test
JDK-6854417: TESTBUG: java/util/regex/RegExTest.java fails
intermittently
Cheers,
-Joe
On 9/16/2015 5:25 PM, Joseph D. Darcy wrote:
Dropping the jtreg alias.
On 9/16/2015 5:18 PM, Martin Buchholz wrote:
Just now for the first time I ran all the jtreg tests in the
"jdk_stable" test set using
make test TEST=jdk_stable
and I got around 50 failures out of 5000.
Is that expected?
Instead, I expect that with all the work on the test suite y'all have
been doing, that the expected failure rate of something with a name
like "jdk_stable" would be 0.
For JDK 9 dev, the tier 1 tests across repos should be stable.
Currently, only the langtools and jdk repos have non-empty tier 1
tests sets defined.
If you run
jtreg ... -exclude:ProblemList.txt -k:intermittent :tier1
there are some tier 1 test that are known to fail intermittently, but
at this point with very low frequency (at least with the testing I'm
exposed to).
HTH,
-Joe
