Hello,
As the next step of implementing "Proposed new policies for JDK 9
regression tests: tiered testing, intermittent failures, and randomness"
[1], please review this changeset which adds a "randomness" jtreg
keyword and tags 240 regression tests in the jdk repository with the
keyword:
JDK-8078334: Mark regression tests using randomness
http://cr.openjdk.java.net/~darcy/8078334.0/
The update to test/TEST.ROOT is
-# The list of keywords supported in the entire test suite
-keys=2d dnd i18n intermittent
+# The list of keywords supported in the entire test suite. The
+# "intermittent" keyword marks tests known to fail intermittently.
+# The "randomness" keyword marks tests using randomness with test
+# cases differing from run to run. (A test using a fixed random seed
+# would not count as "randomness" by this definition.) Extra care
+# should be taken to handle test failures of intermittent or
+# randomness tests.
+
+keys=2d dnd i18n intermittent randomness
To find the affected files for this fix, I grepped over the regression
tests looking for "random" and verified if a random number generated was
used in a way where its values would differ from run to run. In
particular, as noted above, a java.util.Random random number generator
with a fixed seed did *not* qualify a test for the "randomness" keyword.
The usual update to the files in this changeset is just
+ * @key randomness
The updated tests are mostly in the core libs and security libs areas
with a few serviceability tests too. The list in this bug is not quite
exhaustive; I did not include rmi tests or client libraries tests.
I verified that running
jtreg ... -l -k:randomness .
returns the expected number of tests and that the tests continue to pass
(or be ignored) when run.
One goal of marking the tests using randomness is to help root out some
remaining intermittent test failures. If one of the randomness tests is
observed to fail intermittently, if it has not already been updated to
print out the random seed and be able to accept a particular seed when
run, the test should be so modified so that future failures can log the
seed value in hopes of reproducing the failure.
Brian Burkhalter has proactively made these sorts of changes to the
java.math tests earlier in JDK 9. [2] Since most of the randomness tests
don't have a history of instability (and due to the unexpected large
number of them!), I don't think it is necessary to proactively update
all the other tests without evidence of a history of failure. However,
when an otherwise unexplained failure is seen, I recommend applying the
sort of updates Brian has done in java.math to the failing test(s).
Thanks,
-Joe
[1] http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
[2] JDK-8074460, Always print seeds used in [Splittable]Random instances
in java.math tests,
https://bugs.openjdk.java.net/browse/JDK-8074460
