DickJC123 opened a new issue #19207:
URL: https://github.com/apache/incubator-mxnet/issues/19207
## Description
This is a suggestion for a new CI feature. The problem addressed is known
to most- when one submits a PR, one often has to rerun the CI many times to get
past flaky unittests unrelated to the PR. But is a test that fails 0.1% of the
time 'flaky'? Well, if our unittest suite accumulates 100 such 'semi-flaky'
tests, and those tests are run in 10 jobs, there will be an average of 1
failure on every full CI run!
This suggestion of a remedy consists of two part:
- a script that would analyze the PR's diff, and deduce the set of newly
introduced tests
- a script that given a set of tests and a value N, runs the tests N times
[or perhaps runs the test repeatedly for N minutes]
With this proposed 'new test shake-out' facility integrated with the CI, PR
contributors would have increased confidence that their PRs, once merged, would
not make the CI unstable for the greater development community. I'm hopeful
this won't discourage creating new tests- after all, reviewers generally demand
test coverage of new features. However, it might discourage excessively long
tests- if someone adds a test that takes 5 minutes, and N=10 reps, that would
add 50 minutes to the CI runtime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
