Re: RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test
On Wed, 5 Jan 2022 08:31:54 GMT, Aleksey Shipilev wrote: >> doc/testing.html line 198: >> >>> 196: Retry failed tests up to a set number of times. Defaults to 0. >>> 197: REPEAT_COUNT >>> 198: Repeat the tests for a set number of times. Defaults to 0. >> >> Don't we need to mention that the goal of the repeat_count is to run a test >> until it fails first (to reproduce intermittent issues)? > > We can. Current docs copy-pasted the form of `RETRY_COUNT` above. Both should > be updated with some mention for use cases. Let me do that... See #6964. - PR: https://git.openjdk.java.net/jdk/pull/6720
Re: RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test
On Tue, 4 Jan 2022 23:39:45 GMT, Sergey Bylokhov wrote:
>> This adds the test repeat feature in the build system. This is convenient to
>> follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run
>> the test multiple times, until we run out of repeats or the tests fail.
>>
>> With this sample test:
>>
>>
>> /*
>> * @test
>> * @run driver IntermittentTest
>> */
>> public class IntermittentTest {
>> public static void main(String... args) {
>> if (Math.random() < 0.05) {
>> throw new IllegalStateException("Oops");
>> }
>> }
>> }
>>
>>
>> ...a lucky run looks like this:
>>
>>
>> $ CONF=linux-x86_64-server-fastdebug make run-test
>> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5"
>> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
>> Running tests using JTREG control variable 'REPEAT_COUNT=5'
>> Test selection 'sanity/IntermittentTest.java', will run:
>> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>>
>> Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>>
>> Repeating Jtreg run: 1 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>>
>> Repeating Jtreg run: 2 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>>
>> Repeating Jtreg run: 3 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>>
>> Repeating Jtreg run: 4 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>>
>> Repeating Jtreg run: 5 out of 5
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>> Test report is stored in ...
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
>>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>> 1 1 0 0
>>
>> ==
>> TEST SUCCESS
>>
>>
>> ...and unlucky run looks like this:
>>
>>
>> $ CONF=linux-x86_64-server-fastdebug make run-test
>> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50"
>> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
>> Running tests using JTREG control variable 'REPEAT_COUNT=50'
>> Test selection 'sanity/IntermittentTest.java', will run:
>> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>> ...
>> Repeating Jtreg run: 9 out of 50
>> Passed: sanity/IntermittentTest.java
>> Test results: passed: 1
>> Report written to ...
>> Results written to ...
>>
>> Repeating Jtreg run: 10 out of 50
>> --
>> TEST: sanity/IntermittentTest.java
>> TEST JDK: ...
>>
>> ...
>>
>> STDERR:
>> java.lang.IllegalStateException: Oops
>> at IntermittentTest.main(IntermittentTest.java:8)
>> at
>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
>> at java.base/java.lang.reflect.Method.invoke(Method.java:577)
>> at
>> com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
>> at java.base/java.lang.Thread.run(Thread.java:833)
>>
>> JavaTest Message: Test threw exception: java.lang.IllegalStateException
>> JavaTest Message: shutting down test
>>
>>
>> TEST RESULT: Failed. Execution failed: `main' threw exception:
>> java.lang.IllegalStateException: Oops
>> --
>> Test results: failed: 1
>> Report written to ...
>> Results written to ...
>> Error: Some tests failed or other problems occurred.
>>
>> Failures detected, no more repeats.
>> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>> Test report is stored in ...
>>
>> ==
>> Test summary
>> ==
>>TEST TOTAL PASS FAIL ERROR
>>
>>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
1 0 1 0
<<
>> ==
>> TEST FAILURE
>
> doc/testing.html line 198:
>
>> 196: Retry failed tests up to a set number of times. Defaults to 0.
>> 197: REPEAT_COUNT
>> 198: Repeat the tests for a set number of times. Defaults to 0.
>
> Don't we need to mention that the goal of the repeat_count is to run a test
> until it fails first (to reproduce intermittent issues)?
We can. Current docs copy-pasted the form of `RETRY_COUNT` above. Both should
be updated with some mention for use
Re: RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test
On Mon, 6 Dec 2021 11:12:22 GMT, Aleksey Shipilev wrote:
> This adds the test repeat feature in the build system. This is convenient to
> follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run
> the test multiple times, until we run out of repeats or the tests fail.
>
> With this sample test:
>
>
> /*
> * @test
> * @run driver IntermittentTest
> */
> public class IntermittentTest {
> public static void main(String... args) {
> if (Math.random() < 0.05) {
> throw new IllegalStateException("Oops");
> }
> }
> }
>
>
> ...a lucky run looks like this:
>
>
> $ CONF=linux-x86_64-server-fastdebug make run-test
> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=5'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>
> Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>
> Repeating Jtreg run: 1 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 2 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 3 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 4 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 5 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> 1 1 0 0
>
> ==
> TEST SUCCESS
>
>
> ...and unlucky run looks like this:
>
>
> $ CONF=linux-x86_64-server-fastdebug make run-test
> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=50'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> ...
> Repeating Jtreg run: 9 out of 50
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 10 out of 50
> --
> TEST: sanity/IntermittentTest.java
> TEST JDK: ...
>
> ...
>
> STDERR:
> java.lang.IllegalStateException: Oops
> at IntermittentTest.main(IntermittentTest.java:8)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
> at java.base/java.lang.reflect.Method.invoke(Method.java:577)
> at
> com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
> at java.base/java.lang.Thread.run(Thread.java:833)
>
> JavaTest Message: Test threw exception: java.lang.IllegalStateException
> JavaTest Message: shutting down test
>
>
> TEST RESULT: Failed. Execution failed: `main' threw exception:
> java.lang.IllegalStateException: Oops
> --
> Test results: failed: 1
> Report written to ...
> Results written to ...
> Error: Some tests failed or other problems occurred.
>
> Failures detected, no more repeats.
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>>> 1 0 1 0 <<
> ==
> TEST FAILURE
doc/testing.html line 198:
> 196: Retry failed tests up to a set number of times. Defaults to 0.
> 197: REPEAT_COUNT
> 198: Repeat the tests for a set number of times. Defaults to 0.
Don't we need to mention that the goal of the repeat_count is to run a test
until it fails first (to reproduce intermittent issues)?
-
PR: https://git.openjdk.java.net/jdk/pull/6720
Re: RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test
On Mon, 6 Dec 2021 11:12:22 GMT, Aleksey Shipilev wrote:
> This adds the test repeat feature in the build system. This is convenient to
> follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run
> the test multiple times, until we run out of repeats or the tests fail.
>
> With this sample test:
>
>
> /*
> * @test
> * @run driver IntermittentTest
> */
> public class IntermittentTest {
> public static void main(String... args) {
> if (Math.random() < 0.05) {
> throw new IllegalStateException("Oops");
> }
> }
> }
>
>
> ...a lucky run looks like this:
>
>
> $ CONF=linux-x86_64-server-fastdebug make run-test
> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=5'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>
> Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>
> Repeating Jtreg run: 1 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 2 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 3 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 4 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 5 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> 1 1 0 0
>
> ==
> TEST SUCCESS
>
>
> ...and unlucky run looks like this:
>
>
> $ CONF=linux-x86_64-server-fastdebug make run-test
> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=50'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> ...
> Repeating Jtreg run: 9 out of 50
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 10 out of 50
> --
> TEST: sanity/IntermittentTest.java
> TEST JDK: ...
>
> ...
>
> STDERR:
> java.lang.IllegalStateException: Oops
> at IntermittentTest.main(IntermittentTest.java:8)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
> at java.base/java.lang.reflect.Method.invoke(Method.java:577)
> at
> com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
> at java.base/java.lang.Thread.run(Thread.java:833)
>
> JavaTest Message: Test threw exception: java.lang.IllegalStateException
> JavaTest Message: shutting down test
>
>
> TEST RESULT: Failed. Execution failed: `main' threw exception:
> java.lang.IllegalStateException: Oops
> --
> Test results: failed: 1
> Report written to ...
> Results written to ...
> Error: Some tests failed or other problems occurred.
>
> Failures detected, no more repeats.
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>>> 1 0 1 0 <<
> ==
> TEST FAILURE
Thanks, folks!
-
PR: https://git.openjdk.java.net/jdk/pull/6720
Re: RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test
On Mon, 6 Dec 2021 11:12:22 GMT, Aleksey Shipilev wrote:
> This adds the test repeat feature in the build system. This is convenient to
> follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run
> the test multiple times, until we run out of repeats or the tests fail.
>
> With this sample test:
>
>
> /*
> * @test
> * @run driver IntermittentTest
> */
> public class IntermittentTest {
> public static void main(String... args) {
> if (Math.random() < 0.05) {
> throw new IllegalStateException("Oops");
> }
> }
> }
>
>
> ...a lucky run looks like this:
>
>
> $ CONF=linux-x86_64-server-fastdebug make run-test
> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=5'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>
> Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>
> Repeating Jtreg run: 1 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 2 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 3 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 4 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 5 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> 1 1 0 0
>
> ==
> TEST SUCCESS
>
>
> ...and unlucky run looks like this:
>
>
> $ CONF=linux-x86_64-server-fastdebug make run-test
> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=50'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> ...
> Repeating Jtreg run: 9 out of 50
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 10 out of 50
> --
> TEST: sanity/IntermittentTest.java
> TEST JDK: ...
>
> ...
>
> STDERR:
> java.lang.IllegalStateException: Oops
> at IntermittentTest.main(IntermittentTest.java:8)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
> at java.base/java.lang.reflect.Method.invoke(Method.java:577)
> at
> com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
> at java.base/java.lang.Thread.run(Thread.java:833)
>
> JavaTest Message: Test threw exception: java.lang.IllegalStateException
> JavaTest Message: shutting down test
>
>
> TEST RESULT: Failed. Execution failed: `main' threw exception:
> java.lang.IllegalStateException: Oops
> --
> Test results: failed: 1
> Report written to ...
> Results written to ...
> Error: Some tests failed or other problems occurred.
>
> Failures detected, no more repeats.
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>>> 1 0 1 0 <<
> ==
> TEST FAILURE
Marked as reviewed by erikj (Reviewer).
-
PR: https://git.openjdk.java.net/jdk/pull/6720
Re: RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test
On Mon, 6 Dec 2021 11:12:22 GMT, Aleksey Shipilev wrote:
> This adds the test repeat feature in the build system. This is convenient to
> follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run
> the test multiple times, until we run out of repeats or the tests fail.
>
> With this sample test:
>
>
> /*
> * @test
> * @run driver IntermittentTest
> */
> public class IntermittentTest {
> public static void main(String... args) {
> if (Math.random() < 0.05) {
> throw new IllegalStateException("Oops");
> }
> }
> }
>
>
> ...a lucky run looks like this:
>
>
> $ CONF=linux-x86_64-server-fastdebug make run-test
> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=5'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>
> Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
>
> Repeating Jtreg run: 1 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 2 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 3 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 4 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 5 out of 5
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> 1 1 0 0
>
> ==
> TEST SUCCESS
>
>
> ...and unlucky run looks like this:
>
>
> $ CONF=linux-x86_64-server-fastdebug make run-test
> TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50"
> Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
> Running tests using JTREG control variable 'REPEAT_COUNT=50'
> Test selection 'sanity/IntermittentTest.java', will run:
> * jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
> ...
> Repeating Jtreg run: 9 out of 50
> Passed: sanity/IntermittentTest.java
> Test results: passed: 1
> Report written to ...
> Results written to ...
>
> Repeating Jtreg run: 10 out of 50
> --
> TEST: sanity/IntermittentTest.java
> TEST JDK: ...
>
> ...
>
> STDERR:
> java.lang.IllegalStateException: Oops
> at IntermittentTest.main(IntermittentTest.java:8)
> at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
> at java.base/java.lang.reflect.Method.invoke(Method.java:577)
> at
> com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
> at java.base/java.lang.Thread.run(Thread.java:833)
>
> JavaTest Message: Test threw exception: java.lang.IllegalStateException
> JavaTest Message: shutting down test
>
>
> TEST RESULT: Failed. Execution failed: `main' threw exception:
> java.lang.IllegalStateException: Oops
> --
> Test results: failed: 1
> Report written to ...
> Results written to ...
> Error: Some tests failed or other problems occurred.
>
> Failures detected, no more repeats.
> Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
> Test report is stored in ...
>
> ==
> Test summary
> ==
>TEST TOTAL PASS FAIL ERROR
>
>jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>>> 1 0 1 0 <<
> ==
> TEST FAILURE
Looks good. Thank you for fixing this!
-
Marked as reviewed by ihse (Reviewer).
PR: https://git.openjdk.java.net/jdk/pull/6720
RFR: 8244602: Add JTREG_REPEAT_COUNT to repeat execution of a test
This adds the test repeat feature in the build system. This is convenient to
follow-up on intermittently failing tests: the `REPEAT_COUNT > 0` would run the
test multiple times, until we run out of repeats or the tests fail.
With this sample test:
/*
* @test
* @run driver IntermittentTest
*/
public class IntermittentTest {
public static void main(String... args) {
if (Math.random() < 0.05) {
throw new IllegalStateException("Oops");
}
}
}
...a lucky run looks like this:
$ CONF=linux-x86_64-server-fastdebug make run-test
TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=5"
Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
Running tests using JTREG control variable 'REPEAT_COUNT=5'
Test selection 'sanity/IntermittentTest.java', will run:
* jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
Running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
Repeating Jtreg run: 1 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...
Repeating Jtreg run: 2 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...
Repeating Jtreg run: 3 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...
Repeating Jtreg run: 4 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...
Repeating Jtreg run: 5 out of 5
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...
Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
Test report is stored in ...
==
Test summary
==
TEST TOTAL PASS FAIL ERROR
jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
1 1 0 0
==
TEST SUCCESS
...and unlucky run looks like this:
$ CONF=linux-x86_64-server-fastdebug make run-test
TEST=sanity/IntermittentTest.java JTREG="REPEAT_COUNT=50"
Building target 'run-test' in configuration 'linux-x86_64-server-fastdebug'
Running tests using JTREG control variable 'REPEAT_COUNT=50'
Test selection 'sanity/IntermittentTest.java', will run:
* jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
...
Repeating Jtreg run: 9 out of 50
Passed: sanity/IntermittentTest.java
Test results: passed: 1
Report written to ...
Results written to ...
Repeating Jtreg run: 10 out of 50
--
TEST: sanity/IntermittentTest.java
TEST JDK: ...
...
STDERR:
java.lang.IllegalStateException: Oops
at IntermittentTest.main(IntermittentTest.java:8)
at
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
at java.base/java.lang.reflect.Method.invoke(Method.java:577)
at
com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:312)
at java.base/java.lang.Thread.run(Thread.java:833)
JavaTest Message: Test threw exception: java.lang.IllegalStateException
JavaTest Message: shutting down test
TEST RESULT: Failed. Execution failed: `main' threw exception:
java.lang.IllegalStateException: Oops
--
Test results: failed: 1
Report written to ...
Results written to ...
Error: Some tests failed or other problems occurred.
Failures detected, no more repeats.
Finished running test 'jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java'
Test report is stored in ...
==
Test summary
==
TEST TOTAL PASS FAIL ERROR
jtreg:test/hotspot/jtreg/sanity/IntermittentTest.java
>> 1 0 1 0 <<
==
TEST FAILURE
-
Commit messages:
- Fix
Changes: https://git.openjdk.java.net/jdk/pull/6720/files
Webrev: https://webrevs.openjdk.java.net/?repo=jdk=6720=00
Issue: https://bugs.openjdk.java.net/browse/JDK-8244602
Stats: 28 lines in 3 files changed: 27 ins; 0 del; 1 mod
Patch: https://git.openjdk.java.net/jdk/pull/6720.diff
Fetch: git fetch https://git.openjdk.java.net/jdk pull/6720/head:pull/6720
PR: https://git.openjdk.java.net/jdk/pull/6720
