That's a great question! Since we record branch-coverage as well, we would
now reliably notice these test-gaps and could make targeted tests.
Additionally, we could define test-coverage thresholds for certain files to
enforce this - e.g. 100% for all operators.

While the stochastic approach certainly covers a wider area, failures are
currently rather ignored and rerun instead of properly analyzed. Proper
tooling like a consistent test-coverage would then enable us to properly
investigate these inconsistencies if we encounter them.

-Marco

On Sun, Jan 20, 2019 at 5:55 PM Chris Olivier <cjolivie...@gmail.com> wrote:

> wouldn’t abandoning stochastic testing mean that execution paths would no
> longer be tested and therefore increase issues found in production?
>
> On Sun, Jan 20, 2019 at 5:13 AM Marco de Abreu <marco.g.ab...@gmail.com>
> wrote:
>
> > Hello everyone,
> >
> > a few months ago, I have enabled test coverage recording for the MXNet
> > repository [1]. The next step would be to report the coverage changes to
> > pull requests so we can make them part of the review. For this, I've
> > created a pull request here at [2].
> >
> > Before this feature can be enabled though, we have to stabilize the test
> > coverage. Due to the extensive usage of random inputs in our tests, the
> > execution takes different paths and thus, the coverage results undergo a
> > heavy variance. If we would enable the coverage report for PRs, they
> would
> > be marked as increased/decreased coverage without actually doing so. To
> get
> > a few examples, visit [3].
> >
> > I'd like to provide some detailed examples or screenshots, but with big
> > reports [4], CodeCov's webinterface tends to time out. I currently have
> an
> > open ticket with CodeCov that will hopefully improve the situation soon,
> > but we have to live with the timeouts until then.
> >
> > My proposal to improve the situation would be to divide the work across
> the
> > community in two stages:
> > 1. Replace random inputs in tests with deterministic ones if applicable.
> > 2. If randomness cannot be prevented or this 'flakiness' persists, we
> could
> > run the same test multiple times with coverage, look at the coverage diff
> > and then write targeted unittests for the inconsistent parts.
> >
> > If there is more interest towards test coverage, I'd be happy to write a
> > guide that explains how to measure test coverage and detect flaky
> coverage
> > areas so everybody can help contribute towards a stable test coverage.
> > Please let me know what you think.
> >
> > Best regards,
> > Marco
> >
> > [1]: https://codecov.io/gh/apache/incubator-mxnet
> > [2]: https://github.com/apache/incubator-mxnet/pull/12648
> > [3]: https://codecov.io/gh/apache/incubator-mxnet/pulls
> > [4]: https://codecov.io/gh/apache/incubator-mxnet/pull/13849
> >
>

Reply via email to