Re: Devel is broken, we cannot release

2014-07-15 Thread Nate Finch
I don't think we need to stop the world to get these things fixed. It is the responsibility of the team leads to make sure someone's actively working on fixes for regressions. If they're not getting fixed, it's our fault. We should have one of the team leads pick up the regression and assign

Re: Devel is broken, we cannot release

2014-07-15 Thread Wayne Witzel
If we aren't stopping the line when our CI is in the red, then what is the point of even having CI at all? If we are not prepared to adjust the culture of our development. To truly halt forward progress in favor of chasing down regressions then I struggle to find the benefit that CI and testing is

Re: Devel is broken, we cannot release

2014-07-15 Thread William Reade
On Tue, Jul 15, 2014 at 2:51 PM, Wayne Witzel wayne.wit...@canonical.com wrote: If we aren't stopping the line when our CI is in the red, then what is the point of even having CI at all? If we are not prepared to adjust the culture of our development. To truly halt forward progress in favor of

Re: Devel is broken, we cannot release

2014-07-15 Thread Nate Finch
I think that's a fair assessment. Perhaps the easiest fix is to install a switch QA could throw to change the required merge message to something like !!ThisFixesCI!! On Tue, Jul 15, 2014 at 9:57 AM, William Reade william.re...@canonical.com wrote: On Tue, Jul 15, 2014 at 2:51 PM, Wayne

Re: Devel is broken, we cannot release

2014-07-15 Thread Curtis Hovey-Canonical
On Tue, Jul 15, 2014 at 1:29 AM, John Meinel j...@arbash-meinel.com wrote: It seems worthy to just run go test github.com/juju/... as our CI testing, isn't it? (i.e., run all unit tests across all packages that we write on all platforms), rather than *just* github.com/juju/juju. Ah!. That

Re: Devel is broken, we cannot release

2014-07-15 Thread John Meinel
My concern was with the speed of response. I'm happy to have a QA Team switch that must be fixed (with an associated email to juju - dev so everyone knows why their patch won't land). I *would* like us to be tracking stuff like how long we go into regression mode, etc. I think ideally the process

Re: Devel is broken, we cannot release

2014-07-15 Thread Tim Penhey
On 16/07/14 01:57, William Reade wrote: On Tue, Jul 15, 2014 at 2:51 PM, Wayne Witzel wayne.wit...@canonical.com mailto:wayne.wit...@canonical.com wrote: If we aren't stopping the line when our CI is in the red, then what is the point of even having CI at all? If we are not prepared

Devel is broken, we cannot release

2014-07-14 Thread Curtis Hovey-Canonical
Devel has been broken for weeks because of regressions. We cannot release devel. The stable 1.20.0 that we release is actually older than it appears because we had to search CI for an older revision that worked. We have a systemic problem: once a regression is introduced, it blocks the release

Re: Devel is broken, we cannot release

2014-07-14 Thread Wayne Witzel
+1 I've experienced this type of policy and it leads to few things. More tests, better tests, and better self reviews and developer QA. I believe borrowing the other ideas from lean and agile but not having the big stop button when a defect is found is an unsustainable approach to development

Re: Devel is broken, we cannot release

2014-07-14 Thread Ian Booth
I think the problem is engineers are focused on there feature. They don't see the fallout from their changes. They may hope the fix will arrive soon, and that maybe someone else will fix it. I propose a change in policy. When a there is a regression in CI, no new branches can be merged

Re: Devel is broken, we cannot release

2014-07-14 Thread Ian Booth
* FAIL: managedstorage_test trusty ppc64 from 2014-06-30 which had a secondary bug that broke compilation. https://bugs.launchpad.net/juju-core/+bug/1336089 This bug brings up another issue. The code concerned has now been moved off to a juju sub project - blobstorage. So the juju-core

Re: Devel is broken, we cannot release

2014-07-14 Thread Tim Penhey
On 15/07/14 15:48, Ian Booth wrote: * FAIL: managedstorage_test trusty ppc64 from 2014-06-30 which had a secondary bug that broke compilation. https://bugs.launchpad.net/juju-core/+bug/1336089 This bug brings up another issue. The code concerned has now been moved off to a juju sub

Re: Devel is broken, we cannot release

2014-07-14 Thread Ian Booth
On 15/07/14 14:17, Tim Penhey wrote: On 15/07/14 15:48, Ian Booth wrote: * FAIL: managedstorage_test trusty ppc64 from 2014-06-30 which had a secondary bug that broke compilation. https://bugs.launchpad.net/juju-core/+bug/1336089 This bug brings up another issue. The code concerned