There's a JUnit annotation in geode-junit called @IgnoreUntil which you provide a deadline for. After the deadline, it starts running again. Unfortunately the test is not run while it is ignored whereas any test with the FlakyTest category is still run.
It's up to you when to remove FlakyTest category from any given test, but if the test has any thread sleeps or uses non-thread-safe test hooks then it's still fundamentally flaky. In theory, removal of FlakyTest category should require fixing what made it flaky in the first place. If you think you've fixed a test, then by all means remove the category. On Fri, Nov 3, 2017 at 10:48 AM, Nabarun Nag <[email protected]> wrote: > I think majority of the flaky test tags were put in one shot in one commit. > So the timer will expire on all tests in one shot. > Also we have stopped marking things flaky, if something fails in CI, we > immediately try to fix it. If there is a flakiness element in the test, the > test is immediately modified. And slowly we are also cleaning up the > existing flaky tests. > > Regards > Naba > > > > > > > On Fri, Nov 3, 2017 at 10:33 AM Patrick Rhomberg <[email protected]> > wrote: > > > Hello, all! > > > > I was considering doing some git archeology centered around identifying > > how long a any given test class containing a @Flaky has had that > > annotation. Ultimately, I think it would be good to add a test that > would > > fail when any one test has been flaky for too long. I feel like many of > > our flaky tests have fallen by the wayside, and this could provide the > > impetus to resolve these issues in a timely fashion. > > This leads naturally to the question: How long should a test be allowed > > to remain marked Flaky? Certainly, flaky tests are most often of the > > non-deterministic, hard-to-reproduce variety, so some leeway is deserved. > > Two weeks? One month? > > Thoughts? > > > > Imagination is Change. > > ~Patrick Rhomberg > > >
