On Tue, Mar 29, 2016 at 10:42 AM, Mark Ramm-Christensen (Canonical.com) <[email protected]> wrote: > Never a good time to stop feature work entirely and fix what amounts to a > race prone set of tests. > > > But I would advocate building in some practices to improve the situation > incrementally: > > fixing one major issue per team per week
SGTM. How do we know which of the millions of private lists of bugs are the critical ones? Which of the hundred "critical", "papercut", "urgent" LP tags are the critical ones? > promoting all issues which fail CI more than x times per week to Critical > blocking all branches on fridays except for fixes for bugs on the top issues > list Which timezone is friday ? > or some other similar policy > > Over time over time any of the above policies will bring the total number of > test failures down significantly, and would still allow progress on feature > work. > > On Mon, Mar 28, 2016 at 1:05 PM, Nate Finch <[email protected]> > wrote: >> >> I'll just note that we've had flaky tests for as long as I've been working >> on Juju, and there's never a "good" time to fix them. :) >> >> On Mon, Mar 28, 2016 at 11:48 AM Aaron Bentley >> <[email protected]> wrote: >>> >>> -----BEGIN PGP SIGNED MESSAGE----- >>> Hash: SHA256 >>> >>> On 2016-03-28 09:03 AM, Katherine Cox-Buday wrote: >>> > Generally +1 on this, but I'm also intrigued by Martin's >>> > statistic... do we currently weight test failures by how likely >>> > they are to fail (i.e. how likely they are flaky)? That seems like >>> > it would be a great metric to use to decide which to fix first. >>> >>> We don't do it on the likelihood of failure, but we do it on the >>> frequency of failure. >>> >>> http://reports.vapour.ws/releases/top-issues >>> >>> I report on these on the cross-team call, and once the 2.0 settles >>> down, I'll be reporting them on the release call again. >>> >>> Aaron >>> -----BEGIN PGP SIGNATURE----- >>> Version: GnuPG v2 >>> >>> iQEcBAEBCAAGBQJW+VJcAAoJEK84cMOcf+9hWrwH/0JradfscIE0wnt+yCW9nNCR >>> 9hTHI2U19v1VuP6pWI4UiC7srfojPI8EXXEXrrAhF9rT8tpVK4EcJRJK9RvWvvz5 >>> BEquHMS0+eROFOqDJFavEB8hU7BKHErzkSwSG8uKq7JuwHs9gNtQO9z9fIhVKjnr >>> aP4z2IliCqbYfXbupfSTD8TmqhI0AipQymTg3QB4C3sJdXzc5GjzIIckUo/X7aJj >>> zH1tEtlwOdP0c9F+8ZVs1j6AAkb+uDGc/1Qr4MT1kInqGkli2UNF4TOX/AihNPyH >>> iwYgq6O7uOkijFTrL9obRfbXxIFw1WCc9cYzxbRYnGfQff47Dyj7/BUStPPH0i0= >>> =8FQ6 >>> -----END PGP SIGNATURE----- >>> >>> -- >>> Juju-dev mailing list >>> [email protected] >>> Modify settings or unsubscribe at: >>> https://lists.ubuntu.com/mailman/listinfo/juju-dev >> >> >> -- >> Juju-dev mailing list >> [email protected] >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju-dev >> > > > -- > Juju-dev mailing list > [email protected] > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju-dev > -- Juju-dev mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
