Here's a list I started, with a brief rationale for each: https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AjwxZmhDIclsdFdiV2c3dFhGYWpfckhqT1N1ZWpUY1E&usp=sharing
Certainly things could change per image here, this is just to give us an easy way to make a list. On Mon, Nov 4, 2013 at 10:53 AM, Francis Ginther < [email protected]> wrote: > A possible, first step solution would be to add a very simple retry to > our CI test runner scripts and if the retry passes, the test passes. > The harder part would be to identify those test cases which failed on > the original attempt and making those visible. This could be done by > mining the jenkins data itself and could probably be made smart enough > to file bugs when it finds tests that passed the retry, although it's > hard to determine why the tests failed (i.e. maybe it's a flaky test > or maybe unity8 crashed, etc.). > > Francis > > On Mon, Nov 4, 2013 at 3:43 AM, Vincent Ladeuil <[email protected]> > wrote: > >>>>>> Evan Dandrea <[email protected]> writes: > > > > > Can someone provide the set of tests we're running that are flaky? > > > > There is a whole spectrum of flaky tests, and identifying them is an > > art. That's the theory ;) > > > > In practice, there are several ways to automatically identify flake > > tests. > > > > Each failing test could be retried and considered flaky only if it fails > > again in the same context. > > > > Ha, ouch, "same context" is darn hard to guarantee, what if the test is > > flaky only when some other tests run before it ? > > > > What if a flaky test always succeeds when run alone ? > > > > What if the flakiness is caused by a specific hardware (and by that I > > don't mean a brand or a product but a unique piece of hardware) ? > > > > > > > If we don't have a good way of getting this list, could we wire up > > > job retries and log when that occurs? > > > > While I've encountered the above ones in real life, those are not the > > majority, so re-trying the test itself only requires a specific test > > runner (having all projects use such a test runner is achievable, > > slowly, but who will refuse to be protected against flaky tests). We can > > log such occurrences but the log will be disconnected from the test > > results. > > > > Alternatively, a job can be re-run with only the failing tests and it > > becomes slightly easiest to both report and process the flaky > > tests. This doesn't require a specific test runner but a subunit test > > result ;) And a specific test loader to select only the failing tests > > from a previous run[1]. > > > > Even jenkins should be able to display such results as long as we tweak > > the flaky test names (don't ever try to display the details of a test > > failure in jenkins if the test you're interested in is named like a > > previous failure, you'll always get the later ;). And even the dashboard > > could be taught to display green(pass)/yellow(flaky)/red(errors) instead > > of just a %. > > > > Note that unique test names, while not enforced by python's unittest, is > > a simple constraint we want for multiple reasons, the jenkins one > > mentioned above ; this is not a bug a far as I'm concerned, there is > > just no solution to ambiguous names, what if I tell you: 'test X' is > > failing and you're looking at the code of test X (the other one) ? A > > unique name is also needed when you want to select a single test to run. > > > > Now, there are a few risks associated with automatically re-trying flaky > > tests: > > > > - people will care less about them (in the same way any dev start > > ignoring warnings when too many happen making it impossible to notice > > the new ones), > > > > - the root cause of the flakiness can be in the ci engine, the test > > infrastructure or the code itself. The right people should be involved > > to fix them and as of today, all causes are inter-mixed so nobody has > > a clear view of what *he* should fix. > > > > So there is more than automatic retries to get rid of the flaky tests, > > people should be involved to track and fix them. > > > > Vincent > > > > > Thanks! > > > > > On 31 October 2013 11:23, Julien Funk <[email protected]> > wrote: > > >> so, it would be great if we coul get a list of flaky tests re: > discussion > > >> today. I don't mind rerunning the tests automatically when they > fail, but I > > >> think action should be mandatory on any flaky tests we discover > and we > > >> should maintain a list of them somewhere with a process to deal > > >> with them. > > > > Yeah, that's the social part ;) At least as important and probably far > > more than the technical part ;) > > > > Vincent > > > > > > [1]: Test loaders, filters, results and runners already exist in > > lp:selenium-simple-test (and lp:subunit of course). Teaching the test > > runner to re-run failing tests could be added easily. > > > > -- > > Mailing list: https://launchpad.net/~canonical-ci-engineering > > Post to : [email protected] > > Unsubscribe : https://launchpad.net/~canonical-ci-engineering > > More help : https://help.launchpad.net/ListHelp > > > > -- > Francis Ginther > Canonical - Ubuntu Engineering - Continuous Integration Team > > -- > Mailing list: https://launchpad.net/~canonical-ci-engineering > Post to : [email protected] > Unsubscribe : https://launchpad.net/~canonical-ci-engineering > More help : https://help.launchpad.net/ListHelp >
-- Mailing list: https://launchpad.net/~canonical-ci-engineering Post to : [email protected] Unsubscribe : https://launchpad.net/~canonical-ci-engineering More help : https://help.launchpad.net/ListHelp
