Agree with Bhavin's arguments 100%. Please don't compromise the
stability of CI with Flaky tests. Address the root cause of why these
tests are failing / not deterministic as per propper engineering
standards.
Hence, my non-binding vote is:
-1 for proposal #1 for re-enabling flaky tests.
+1 for
we haven’t seemed to touch on this yet, but what’s the vision on how we
“encourage” people to fix their tests (assuming we have a rough idea who is
responsible)? honor system? complaining on dev? blocking PR merges? prayer?
it’s been pointed out a few times that not one disabled test has been
Sheng, could you provide a list of tests which you would cover with the
flaky-plugin? I totally agree with the point that we should not create a
release if we have reduced test coverage and it should be our highest
priority to restore it properly. I'd propose that if a test takes less than
5
Hi Bhavin,
Thank you for the support. Running it nightly is a great idea in that it
doesn't compromise the coverage and we can still get notified fairly soon when
things are breaking. Is there a way to subscribe to its result report?
-sz
On 2018-01-14 12:28, Bhavin Thaker
Hi Bhavin,
Thanks for sharing your thoughts. Regarding the usage of 'flaky' plugin for
retrying flaky tests, it's proposed as a compromise, given that it will take
time to properly fix the tests and we still need coverage in the meantime.
I'm not sure if releasing before these tests are
Hi Sheng,
Thank you for your efforts and this proposal to improve the tests. Here are
my thoughts.
Shouldn’t the focus be to _engineer_ each test to be reliable instead of
compromising and discussing the relative tradeoffs in re-enabling flaky
tests? Is the test failure probability really 10%?
Hello Sheng,
thanks a lot for leading this task!
+1 for both points. Additionally, I'd propose to add the requirement to
specify a reason if a new test takes more than X seconds (say 10) or adds
an external dependency.
Looking forward to getting these tests fixed :)
Best regards,
Marco
On
Hi MXNet community,
Thanks to the efforts of several community members, we identified many
flaky tests. These tests are currently disabled to ensure the smooth
execution of continuous integration (CI). As a result, we lost coverage on
those features. They need fixing and to be re-enabled to
