I think that's a fair assessment. Perhaps the easiest fix is to install a switch QA could throw to change the required merge message to something like !!ThisFixesCI!!
On Tue, Jul 15, 2014 at 9:57 AM, William Reade <[email protected]> wrote: > On Tue, Jul 15, 2014 at 2:51 PM, Wayne Witzel <[email protected]> > wrote: > >> If we aren't stopping the line when our CI is in the red, then what is >> the point of even having CI at all? If we are not prepared to adjust the >> culture of our development. To truly halt forward progress in favor of >> chasing down regressions then I struggle to find the benefit that CI and >> testing is giving us at all. Just confirming that master is broken and we >> are still ignoring it? If master is broken, we all our broken. No >> development you are doing should proceed that is based on a broken master. >> That work cannot at any point be considered in good working condition. A >> problem in master is everyone's problem. >> >> Bugs that are found throughout the normal operations and usage of juju >> should be assigned a priority and queued, but regression is a sign of a >> greater problem that should be resolved immediately. Allowing regressions >> to not stop the line is taking the stance that we don't care about >> deterioration in our code base. >> > > +100 to this. Regressions are a Big Deal and should be treated as such; > leaving other merges queued until such a time as the regression is fixed > (or backed out for rework) is entirely reasonable (and I think we've got > enough evidence that the alternative really doesn't fly effectively). > > Cheers > William > > >> >> >> On Tue, Jul 15, 2014 at 9:37 AM, Nate Finch <[email protected]> >> wrote: >> >>> I don't think we need to stop the world to get these things fixed. It >>> is the responsibility of the team leads to make sure someone's actively >>> working on fixes for regressions. If they're not getting fixed, it's our >>> fault. We should have one of the team leads pick up the regression and >>> assign someone to work on it, just like any other high priority bug. >>> >>> >>> >>> On Mon, Jul 14, 2014 at 3:05 PM, Curtis Hovey-Canonical < >>> [email protected]> wrote: >>> >>>> Devel has been broken for weeks because of regressions. We cannot >>>> release devel. The stable 1.20.0 that we release is actually older >>>> than it appears because we had to search CI for an older revision that >>>> worked. >>>> >>>> We have a systemic problem: once a regression is introduced, it blocks >>>> the release for weeks, and we build on top of the regression. We often >>>> see many regressions.The regression mutate as people merge more >>>> branches. >>>> >>>> The current two regressions are: >>>> * win juju client still broken with unknown >>>> from 2014-06-27 which has varied as a compilation >>>> problem or panic during execution. >>>> https://bugs.launchpad.net/juju-core/+bug/1335328 >>>> >>>> * FAIL: managedstorage_test trusty ppc64 >>>> from 2014-06-30 which had a secondary bug that broke compilation. >>>> https://bugs.launchpad.net/juju-core/+bug/1336089 >>>> >>>> I think the problem is engineers are focused on there feature. They >>>> don't see the fallout from their changes. They may hope the fix will >>>> arrive soon, and that maybe someone else will fix it. >>>> >>>> I propose a change in policy. When a there is a regression in CI, no >>>> new branches can be merged except those that link to the blocking bug. >>>> This will encourage engineers to fix the regression. One way to fix >>>> the regression is to identify and revert the commit that broken CI. >>>> >>>> >>>> -- >>>> Curtis Hovey >>>> Canonical Cloud Development and Operations >>>> http://launchpad.net/~sinzui >>>> >>>> -- >>>> Juju-dev mailing list >>>> [email protected] >>>> Modify settings or unsubscribe at: >>>> https://lists.ubuntu.com/mailman/listinfo/juju-dev >>>> >>> >>> >>> -- >>> Juju-dev mailing list >>> [email protected] >>> Modify settings or unsubscribe at: >>> https://lists.ubuntu.com/mailman/listinfo/juju-dev >>> >>> >> >> >> -- >> Wayne Witzel III >> [email protected] >> >> -- >> Juju-dev mailing list >> [email protected] >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju-dev >> >> >
-- Juju-dev mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
