Thanks Kirk! ~/code/incubator-geode (develop)$ grep -ro "FlakyTest.class" . | grep -v Binary | wc -l | xargs echo "Flake factor:" Flake factor: 136
Anthony > On Apr 25, 2016, at 9:45 PM, William Markito <[email protected]> wrote: > > +1 > > Are we also planning to automate the additional build task somehow ? > > I'd also suggest creating a wiki page with some stats (like how many > FlakyTests we currently have) and the idea behind this effort so we can > keep track and see how it's evolving over time. > > On Mon, Apr 25, 2016 at 6:54 PM, Kirk Lund <[email protected]> wrote: > >> After completing GEODE-1233, all currently known flickering tests are now >> annotated with our FlakyTest JUnit Category. >> >> In an effort to divide our build up into multiple build pipelines that are >> sequential and dependable, we could consider excluding FlakyTests from the >> primary integrationTest and distributedTest tasks. An additional build task >> would then execute all of the FlakyTests separately. This would hopefully >> help us get to a point where we can depend on our primary testing tasks >> staying green 100% of the time. We would then prioritize fixing the >> FlakyTests and one by one removing the FlakyTest category from them. >> >> I would also suggest that we execute the FlakyTests with "forkEvery 1" to >> give each test a clean JVM or set of DistributedTest JVMs. That would >> hopefully decrease the chance of a GC pause or test pollution causing >> flickering failures. >> >> Having reviewed lots of test code and failure stacks, I believe that the >> primary causes of FlakyTests are timing sensitivity (thread sleeps or >> nothing that waits for async activity, timeouts or sleeps that are >> insufficient on busy CPU or I/O or during due GC pause) and random ports >> via AvailablePort (instead of using zero for ephemeral port). >> >> Opinions or ideas? Hate it? Love it? >> >> -Kirk >> > > > > -- > > ~/William
signature.asc
Description: Message signed with OpenPGP using GPGMail
