benWize commented on pull request #16253:
URL: https://github.com/apache/beam/pull/16253#issuecomment-999183055
Hi @aaltay, I'm working on BEAM-12672, but after some testing I'm getting
two cases for different configs, maybe you can give me some guidance.
I added this test to reproduce a flaky test.
https://github.com/apache/beam/blob/46af36fea099e4255bf00f02593940839ce79ca6/sdks/java/core/src/test/java/org/apache/beam/sdk/DummyTest.java#L32.
In this JavaPrecommit this test is marked as `passed`
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4407/testReport/org.apache.beam.sdk/DummyTest/,
but in the console, it shows a failure before a success
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4407/console.
The first failure has not been recorded at test results in Jenkins.
Also, the failed test
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4407/testReport/org.apache.beam.sdk.fn.data/BeamFnDataInboundObserver2Test/
was rerun several times
(https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4407/console) but
just the last one was recorded.
This happens when the option `mergeReruns`
(https://docs.gradle.org/current/userguide/java_testing.html) at
https://github.com/apache/beam/blob/46af36fea099e4255bf00f02593940839ce79ca6/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L416
is set as `true`.
When the option `mergeReruns` is not set as true, Jenkins records the
multiple reruns
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4392/testReport/org.apache.beam.sdk/DummyTest/
but the global status execution is marked as UNSTABLE
(https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4392/), which shows
a failure in Github Checks UI.
Also with the `mergeReruns` disabled, Gradle scans shows the tests as flaky
https://scans.gradle.com/s/32e4qwltztojq/tests/:sdks:java:core:test/org.apache.beam.sdk.DummyTest/testFlakiness?top-execution=1.
At this time, I'm not sure which path accomplishes better the requirements
by this task, so I want to ask you for your guidance on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
