Hi all,
My name is Carl Tsai, I’m an intern working with the Amazon MXNet engine team. For the past several weeks, I’ve been working with my mentor Hao Jin on an automatic flaky test detection system. As you may know, there’s an ongoing effort to reduce the number of flaky tests in the code base, and this flaky test detector is meant to help ensure that these efforts will have a long-lasting impact on the quality of our tests. This is an automated tool that will be run on PR checks to detect flaky tests before we merge them. It will check PRs for new and modified test cases and run them through a flakiness checker so that we can have high confidence that our tests are not flaky. There’s a design overview on confluence <https://cwiki.apache.org/confluence/display/MXNET/Automated+Flaky+Test+Detector>— I’d be happy to hear any feedback! In the meantime, one of the components, the flakiness checker, has been merged under the tools/ folder and I’d encourage those of you who are working on flaky tests or writing new tests to try it out. Documentation can be found on the confluence page for the project as well as on the Reproducing Test Results Page <https://cwiki.apache.org/confluence/display/MXNET/Reproducing+test+results> under tips and tricks, and I’m open to suggestions on how to improve it. Thanks, Carl
