Never a good time to stop feature work entirely and fix what amounts to a race prone set of tests.
But I would advocate building in some practices to improve the situation incrementally: - fixing one major issue per team per week - promoting all issues which fail CI more than x times per week to Critical - blocking all branches on fridays except for fixes for bugs on the top issues list - or some other similar policy Over time over time any of the above policies will bring the total number of test failures down significantly, and would still allow progress on feature work. On Mon, Mar 28, 2016 at 1:05 PM, Nate Finch <nate.fi...@canonical.com> wrote: > I'll just note that we've had flaky tests for as long as I've been working > on Juju, and there's never a "good" time to fix them. :) > > On Mon, Mar 28, 2016 at 11:48 AM Aaron Bentley < > aaron.bent...@canonical.com> wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA256 >> >> On 2016-03-28 09:03 AM, Katherine Cox-Buday wrote: >> > Generally +1 on this, but I'm also intrigued by Martin's >> > statistic... do we currently weight test failures by how likely >> > they are to fail (i.e. how likely they are flaky)? That seems like >> > it would be a great metric to use to decide which to fix first. >> >> We don't do it on the likelihood of failure, but we do it on the >> frequency of failure. >> >> http://reports.vapour.ws/releases/top-issues >> >> I report on these on the cross-team call, and once the 2.0 settles >> down, I'll be reporting them on the release call again. >> >> Aaron >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v2 >> >> iQEcBAEBCAAGBQJW+VJcAAoJEK84cMOcf+9hWrwH/0JradfscIE0wnt+yCW9nNCR >> 9hTHI2U19v1VuP6pWI4UiC7srfojPI8EXXEXrrAhF9rT8tpVK4EcJRJK9RvWvvz5 >> BEquHMS0+eROFOqDJFavEB8hU7BKHErzkSwSG8uKq7JuwHs9gNtQO9z9fIhVKjnr >> aP4z2IliCqbYfXbupfSTD8TmqhI0AipQymTg3QB4C3sJdXzc5GjzIIckUo/X7aJj >> zH1tEtlwOdP0c9F+8ZVs1j6AAkb+uDGc/1Qr4MT1kInqGkli2UNF4TOX/AihNPyH >> iwYgq6O7uOkijFTrL9obRfbXxIFw1WCc9cYzxbRYnGfQff47Dyj7/BUStPPH0i0= >> =8FQ6 >> -----END PGP SIGNATURE----- >> >> -- >> Juju-dev mailing list >> Juju-dev@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju-dev >> > > -- > Juju-dev mailing list > Juju-dev@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > >
-- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
