Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-28 Thread Thierry Carrez
James E. Blair wrote: [...] Most of these bugs are not failures of the test system; they are real bugs. Many of them have even been in OpenStack for a long time, but are only becoming visible now due to improvements in our tests. That's not much help to developers whose patches are being

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-28 Thread Daniel P. Berrange
On Mon, Jul 28, 2014 at 02:28:56PM +0200, Thierry Carrez wrote: James E. Blair wrote: [...] Most of these bugs are not failures of the test system; they are real bugs. Many of them have even been in OpenStack for a long time, but are only becoming visible now due to improvements in our

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-28 Thread Eoghan Glynn
Sean Dague wrote: To be clear, the functional tests will not be Tempest tests. This is a different class of testing, it's really another tox target that needs a devstack to run. A really good initial transition would be things like the CLI testing. Also, the Tempest team has gone out

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-26 Thread Jay Pipes
On 07/24/2014 06:36 PM, John Dickinson wrote: On Jul 24, 2014, at 3:25 PM, Sean Dague s...@dague.net wrote: On 07/24/2014 06:15 PM, Angus Salkeld wrote: We do this in Solum and I really like it. It's nice for the same reviewers to see the functional tests and the code the implements a feature.

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-25 Thread Daniel P. Berrange
On Thu, Jul 24, 2014 at 04:01:39PM -0400, Sean Dague wrote: On 07/24/2014 12:40 PM, Daniel P. Berrange wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: ==Future changes== ===Fixing Faster=== We introduce bugs to OpenStack at some constant rate, which piles up

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-25 Thread Steven Hardy
On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: snip * Put the burden for a bunch of these tests back on the projects as functional tests. Basically a custom devstack environment that a project can create with a set of services that they minimally need to do their

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-25 Thread David Kranz
On 07/25/2014 10:01 AM, Steven Hardy wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: snip * Put the burden for a bunch of these tests back on the projects as functional tests. Basically a custom devstack environment that a project can create with a set of

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-25 Thread Sean Dague
On 07/25/2014 10:01 AM, Steven Hardy wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: snip * Put the burden for a bunch of these tests back on the projects as functional tests. Basically a custom devstack environment that a project can create with a set of

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-25 Thread Steve Baker
On 25/07/14 11:18, Sean Dague wrote: On 07/25/2014 10:01 AM, Steven Hardy wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: snip * Put the burden for a bunch of these tests back on the projects as functional tests. Basically a custom devstack environment that a

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-25 Thread Joe Gordon
On Thu, Jul 24, 2014 at 3:54 PM, Sean Dague s...@dague.net wrote: On 07/24/2014 05:57 PM, Matthew Treinish wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: OpenStack has a substantial CI system that is core to its development process. The goals of the system are to

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-25 Thread Matthew Treinish
On Thu, Jul 24, 2014 at 06:54:38PM -0400, Sean Dague wrote: On 07/24/2014 05:57 PM, Matthew Treinish wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: OpenStack has a substantial CI system that is core to its development process. The goals of the system are to

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-25 Thread Robert Collins
On 26 July 2014 08:20, Matthew Treinish mtrein...@kortar.org wrote: This is also more of a pragmatic organic approach to figuring out the interfaces we need to lock down. When one projects breaks depending on an interface in another project, that should trigger this kind of contract growth,

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Chmouel Boudjnah
Hello, Thanks for writing this summary, I like all those ideas and thanks working hard on fixing this. * For all non gold standard configurations, we'll dedicate a part of our infrastructure to running them in a continuous background loop, as well as making these configs available

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Sean Dague
On 07/24/2014 06:06 AM, Chmouel Boudjnah wrote: Hello, Thanks for writing this summary, I like all those ideas and thanks working hard on fixing this. * For all non gold standard configurations, we'll dedicate a part of our infrastructure to running them in a continuous background

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Russell Bryant
On 07/23/2014 05:39 PM, James E. Blair wrote: ==Final thoughts== The current rate of test failures and subsequent rechecks is not sustainable in the long term. It's not good for contributors, reveiewers, or the overall project quality. While these bugs do need to be addressed, it's

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Daniel P. Berrange
On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: ==Future changes== ===Fixing Faster=== We introduce bugs to OpenStack at some constant rate, which piles up over time. Our systems currently treat all changes as equally risky and important to the health of the system, which

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Joshua Harlow
A potentially brilliant idea ;-) Aren't all the machines the gate runs tests on VMs running via OpenStack APIs? OpenStack supports snapshotting (last time I checked). So instead of providing back a whole bunch of log files, provide back a snapshot of the machine/s that ran the tests; let

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Anita Kuno
On 07/24/2014 12:40 PM, Daniel P. Berrange wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: ==Future changes== ===Fixing Faster=== We introduce bugs to OpenStack at some constant rate, which piles up over time. Our systems currently treat all changes as equally

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Joshua Harlow
On Jul 24, 2014, at 12:08 PM, Anita Kuno ante...@anteaya.info wrote: On 07/24/2014 12:40 PM, Daniel P. Berrange wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: ==Future changes== ===Fixing Faster=== We introduce bugs to OpenStack at some constant rate, which

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Sean Dague
On 07/24/2014 02:51 PM, Joshua Harlow wrote: A potentially brilliant idea ;-) Aren't all the machines the gate runs tests on VMs running via OpenStack APIs? OpenStack supports snapshotting (last time I checked). So instead of providing back a whole bunch of log files, provide back a

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Sean Dague
On 07/24/2014 12:40 PM, Daniel P. Berrange wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: ==Future changes== ===Fixing Faster=== We introduce bugs to OpenStack at some constant rate, which piles up over time. Our systems currently treat all changes as equally

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Joshua Harlow
On Jul 24, 2014, at 12:54 PM, Sean Dague s...@dague.net wrote: On 07/24/2014 02:51 PM, Joshua Harlow wrote: A potentially brilliant idea ;-) Aren't all the machines the gate runs tests on VMs running via OpenStack APIs? OpenStack supports snapshotting (last time I checked). So instead

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Matthew Treinish
On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: OpenStack has a substantial CI system that is core to its development process. The goals of the system are to facilitate merging good code, prevent regressions, and ensure that there is at least one configuration of upstream

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Angus Salkeld
On Wed, 2014-07-23 at 14:39 -0700, James E. Blair wrote: OpenStack has a substantial CI system that is core to its development process. The goals of the system are to facilitate merging good code, prevent regressions, and ensure that there is at least one configuration of upstream OpenStack

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Sean Dague
On 07/24/2014 06:15 PM, Angus Salkeld wrote: On Wed, 2014-07-23 at 14:39 -0700, James E. Blair wrote: OpenStack has a substantial CI system that is core to its development process. The goals of the system are to facilitate merging good code, prevent regressions, and ensure that there is at

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread John Dickinson
On Jul 24, 2014, at 3:25 PM, Sean Dague s...@dague.net wrote: On 07/24/2014 06:15 PM, Angus Salkeld wrote: On Wed, 2014-07-23 at 14:39 -0700, James E. Blair wrote: OpenStack has a substantial CI system that is core to its development process. The goals of the system are to facilitate

Re: [openstack-dev] Thoughts on the patch test failure rate and moving forward

2014-07-24 Thread Sean Dague
On 07/24/2014 05:57 PM, Matthew Treinish wrote: On Wed, Jul 23, 2014 at 02:39:47PM -0700, James E. Blair wrote: OpenStack has a substantial CI system that is core to its development process. The goals of the system are to facilitate merging good code, prevent regressions, and ensure that