I have a couple of questions: 1) What would be the criteria for removing `FLAKY` label from a test? Who will take care of removing this label? 2) Do we expect that most of our tests will eventually get `FLAKY` label?
On Thu, Mar 29, 2018 at 7:35 PM, Meng Zhu <m...@mesosphere.com> wrote: > +1, the advantages are appealing. > > Though I am afraid that this will probably reduce the incentive to fix > flaky tests. > > -Meng > > On Thu, Mar 29, 2018 at 9:45 AM, Benno Evers <bev...@mesosphere.com> > wrote: > > > Hi all, > > > > if you're regularly running Mesos unit tests, e.g. because you've set up > a > > CI system, you probably noticed that there is a lot of noise in the > results > > due to flaky tests. > > > > As a measure to ease the pain, what do you think about adding a `FLAKY` > > label to known flaky unit tests, similar to how we have `ROOT`, > `INTERNET`, > > `DISABLED`, etc. right now? > > > > The advantages, in my opinion, would be: > > - Looking at test results, it would be immediately visible whether a > test > > failure was known flaky or not without going to JIRA > > - People who want to reduce noise can disable all known flaky tests by a > > simple gtest filter > > - People who want to can still run the flaky tests easier than if they > get > > disabled outright > > - With a little bit of scripting, it would be possible to add logic like > > "for flaky tests, run them 10 times and only report a failure if more > than > > x% of the runs fail." > > > > What do you think? > > > > Best regards, > > -- > > Benno Evers > > Software Engineer, Mesosphere > > >
