Would the CI run FLAKY tests or will it filter it out? I'm assuming it still does based on your observation above.
What are the other reasons tests are DISABLED today? On Thu, Mar 29, 2018 at 10:35 AM, Meng Zhu <m...@mesosphere.com> wrote: > +1, the advantages are appealing. > > Though I am afraid that this will probably reduce the incentive to fix > flaky tests. > > -Meng > > On Thu, Mar 29, 2018 at 9:45 AM, Benno Evers <bev...@mesosphere.com> > wrote: > > > Hi all, > > > > if you're regularly running Mesos unit tests, e.g. because you've set up > a > > CI system, you probably noticed that there is a lot of noise in the > results > > due to flaky tests. > > > > As a measure to ease the pain, what do you think about adding a `FLAKY` > > label to known flaky unit tests, similar to how we have `ROOT`, > `INTERNET`, > > `DISABLED`, etc. right now? > > > > The advantages, in my opinion, would be: > > - Looking at test results, it would be immediately visible whether a > test > > failure was known flaky or not without going to JIRA > > - People who want to reduce noise can disable all known flaky tests by a > > simple gtest filter > > - People who want to can still run the flaky tests easier than if they > get > > disabled outright > > - With a little bit of scripting, it would be possible to add logic like > > "for flaky tests, run them 10 times and only report a failure if more > than > > x% of the runs fail." > > > > What do you think? > > > > Best regards, > > -- > > Benno Evers > > Software Engineer, Mesosphere > > >
