Hello,
Any additional comments on marking with tests in question with a
"randomness" keyword?
Thanks,
-Joe
On 4/21/2015 7:13 PM, Joseph D. Darcy wrote:
Hello,
As the next step of implementing "Proposed new policies for JDK 9
regression tests: tiered testing, intermittent failures, and
randomness" [1], please review this changeset which adds a
"randomness" jtreg keyword and tags 240 regression tests in the jdk
repository with the keyword:
JDK-8078334: Mark regression tests using randomness
http://cr.openjdk.java.net/~darcy/8078334.0/
The update to test/TEST.ROOT is
-# The list of keywords supported in the entire test suite
-keys=2d dnd i18n intermittent
+# The list of keywords supported in the entire test suite. The
+# "intermittent" keyword marks tests known to fail intermittently.
+# The "randomness" keyword marks tests using randomness with test
+# cases differing from run to run. (A test using a fixed random seed
+# would not count as "randomness" by this definition.) Extra care
+# should be taken to handle test failures of intermittent or
+# randomness tests.
+
+keys=2d dnd i18n intermittent randomness
To find the affected files for this fix, I grepped over the regression
tests looking for "random" and verified if a random number generated
was used in a way where its values would differ from run to run. In
particular, as noted above, a java.util.Random random number generator
with a fixed seed did *not* qualify a test for the "randomness" keyword.
The usual update to the files in this changeset is just
+ * @key randomness
The updated tests are mostly in the core libs and security libs areas
with a few serviceability tests too. The list in this bug is not quite
exhaustive; I did not include rmi tests or client libraries tests.
I verified that running
jtreg ... -l -k:randomness .
returns the expected number of tests and that the tests continue to
pass (or be ignored) when run.
One goal of marking the tests using randomness is to help root out
some remaining intermittent test failures. If one of the randomness
tests is observed to fail intermittently, if it has not already been
updated to print out the random seed and be able to accept a
particular seed when run, the test should be so modified so that
future failures can log the seed value in hopes of reproducing the
failure.
Brian Burkhalter has proactively made these sorts of changes to the
java.math tests earlier in JDK 9. [2] Since most of the randomness
tests don't have a history of instability (and due to the unexpected
large number of them!), I don't think it is necessary to proactively
update all the other tests without evidence of a history of failure.
However, when an otherwise unexplained failure is seen, I recommend
applying the sort of updates Brian has done in java.math to the
failing test(s).
Thanks,
-Joe
[1]
http://mail.openjdk.java.net/pipermail/jdk9-dev/2015-March/001991.html
[2] JDK-8074460, Always print seeds used in [Splittable]Random
instances in java.math tests,
https://bugs.openjdk.java.net/browse/JDK-8074460
