[
https://issues.apache.org/jira/browse/BEAM-12672?focusedWorklogId=699716&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-699716
]
ASF GitHub Bot logged work on BEAM-12672:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 22/Dec/21 00:18
Start Date: 22/Dec/21 00:18
Worklog Time Spent: 10m
Work Description: benWize commented on pull request #16253:
URL: https://github.com/apache/beam/pull/16253#issuecomment-999183055
Hi @aaltay, I'm working on BEAM-12672, but after some testing I'm getting
two cases for different configs, maybe you can give me some guidance.
I added this test to reproduce a flaky test.
https://github.com/apache/beam/blob/46af36fea099e4255bf00f02593940839ce79ca6/sdks/java/core/src/test/java/org/apache/beam/sdk/DummyTest.java#L32.
In this JavaPrecommit this test is marked as `passed`
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4407/testReport/org.apache.beam.sdk/DummyTest/,
but in the console, it shows a failure before a success
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4407/console.
The first failure has not been recorded at test results in Jenkins.
Also, the failed test
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4407/testReport/org.apache.beam.sdk.fn.data/BeamFnDataInboundObserver2Test/
was rerun several times
(https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4407/console) but
just the last one was recorded.
This happens when the option `mergeReruns`
(https://docs.gradle.org/current/userguide/java_testing.html) at
https://github.com/apache/beam/blob/46af36fea099e4255bf00f02593940839ce79ca6/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L416
is set as `true`.
When the option `mergeReruns` is not set as true, Jenkins records the
multiple reruns
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4392/testReport/org.apache.beam.sdk/DummyTest/
but the global status execution is marked as UNSTABLE
(https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4392/), which shows
a failure in Github Checks UI.
Also with the `mergeReruns` disabled, Gradle scans shows the tests as flaky
https://scans.gradle.com/s/32e4qwltztojq/tests/:sdks:java:core:test/org.apache.beam.sdk.DummyTest/testFlakiness?top-execution=1.
At this time, I'm not sure which path accomplishes better the requirements
by this task, so I want to ask you for your guidance on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 699716)
Time Spent: 11h (was: 10h 50m)
> Update all Precommits to rerun failed tests K times.
> ----------------------------------------------------
>
> Key: BEAM-12672
> URL: https://issues.apache.org/jira/browse/BEAM-12672
> Project: Beam
> Issue Type: Test
> Components: testing
> Reporter: Alex Amato
> Assignee: Benjamin Gonzalez
> Priority: P2
> Time Spent: 11h
> Remaining Estimate: 0h
>
> We should configure the precommit test suites to re-run failed tests a few
> (K) times.
> (A bonus feature would be to tag certain tests as known flakey, and run
> multiple attempts in parallel when the test is first ran)
> The benefits of this are to:
> * Not have flakey tests slow down PR reviews and merges.
> * And avoiding the need to repeatedly manually rerun the precommit test with
> a PR comment.
> * Getting a clear signal that failed tests are due to the code in the PR
> * Save computing resources (Rather than needing to run the entire test suite
> every time it fails, only the failed tests will rerun).
> (Its not realistic to have 0 flakeyness, so this is desirable to improve
> productivity, as it can take several attempts to have all the tests pass in a
> single run).
> Note: This does not mean that we should ignore flakey tests, which are
> sometimes indicative of bugs. We should continue to investigate those with a
> continuous test suite.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
