Hello,
I'd like to get this changeset, or something close to it, pushed soon so
we can start taking advantage of better failure triaging.
Any further concerns?
Thanks,
-Joe
On 4/24/2015 11:04 AM, joe darcy wrote:
On 4/23/2015 10:58 PM, Alan Bateman wrote:
On 24/04/2015 02:54, Joseph D. Darcy wrote:
Hello,
Any additional comments on marking with tests in question with a
"randomness" keyword?
Thanks,
-Joe
I don't object to this keyword although I should say that most of the
(apparent) randomness that I've seen hasn't been because of tests
using Random but rather because of non-deterministic sequence of
tests running in the same VM. The switch from othervm to agentvm a
few years ago took a long time to weed out issues, same thing (even
more) with -concurrency. Another source of apparent randomness is
just having a large pool of machines and where machines are randomly
selected to execute the tests.
I agree with your comments that tests using Random should have report
the seed and a have a way to re-run with the same value. It could
help with some cases.
One thing that isn't clear to me is how this keyword will be
maintained. If I modify a test and replace the use of Random then I
guess I need to remember to also remove this label. When I add a test
that uses Random ... Just wondering if there is a tool to do the
tests analysis, identify the use of specific APIs ,and help identify
where the keyword might be missing or not needed.
The goal of the keyword is to enable better analysis of bug failures.
We (still) have some intermittent test failures in the platform, some
probably remaining from -concurrency issues and some due to
inconsistent machine config issues on the test farms. However, there
is reason to believe some of the tests also fail because of the use of
randomness in the generation of test cases. For example, the test
java/lang/invoke/MethodHandles/CatchExceptionTest.java
fails about once every three hundred runs when the random cases it
generates trigger an error condition. Setting the seed to known-bad
value causes the test to fail consistently (JDK-8055269). I have
suspicions some of our other intermittently failing tests are in a
similar but undiagnosed situation.
If a test uses randomness, I think the default assumed cause for a
test failure should switch from environmental causes (bad machine
config, etc.) to the random behavior. Therefore, when a random-using
test fails, I think the failure analysis needs to include recording of
the bad seed to see if the failure is reproducible / deterministic
with a fixed seed value.
I was very surprised by the large number of random-using tests in the
JDK regression tests. Nearly all of these have been stably passing for
many years. After the one-time cost of this large update to add the
keywords, there would be a small amount of incremental test
maintenance as new random-using tests were added or the randomness
value of a test changed. However, I would expect whether or not a test
used randomness to be a property of a test that very rarely changed.
Thanks,
-Joe
