How about if we make a script for determining flakey status, based on rate of failing in either post-commit or in the pre-existing branch check for precommit. The script could live in dev-support and work by effectively grabbing the test reports from those jobs.
We can then run that script in a job and save the results as a build artifact. We can then build the changes in your original email: ignoring the list entries in our normal builds and rechecking just them in a dedicated job. On Tue, Mar 29, 2016 at 6:30 PM, Sean Busbey <bus...@cloudera.com> wrote: > Where/How do we maintain the list of tests that are flagged as flakes? > > On Tue, Mar 29, 2016 at 6:25 PM, Apekshit Sharma <a...@cloudera.com> wrote: >> Proposal: >> Maintain a list of flaky tests which can be ignored in main builds (for >> patches) and setup a new job which will periodically run those flaky tests. >> >> Benefits: >> - Cleaner main builds. Less runs per patch, ideally no re-runs because of >> timeouts and other flakiness. Less frustration. Increased developer >> productivity. >> - Logs from various runs available when anyone tries to fix these tests. >> >> Possible con: We start demoting tests to flaky list which never get fixed. >> >> How? >> Internally, we have setup up two jenkins jobs, one for main build and one >> for flaky. In execute shell step, they curl to get the list of flaky tests >> and use surefire plugin flags to ignore/run particular tests. >> I volunteer to set them up. However, not sure if this approach will work >> since we have yetus upstream. >> >> - Appy > > > > -- > busbey -- busbey
