Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-12 Thread Eoghan Glynn
So I'm not sure that this should be a mandatory thing, but an opt-in. My real concern is the manpower, who is going to take the time to write all the test suites for all of the projects. I think it would be better to add that on-demand as the extra testing is required. That being said, I

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-10 Thread Eoghan Glynn
Note that the notifications that capture these resource state transitions are a long-standing mechanism in openstack that ceilometer has depended on from the very outset. I don't think it's realistic to envisage these interactions will be replaced by REST APIs any time soon. I wasn't

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-10 Thread Matthew Treinish
On Thu, Jul 10, 2014 at 08:37:40AM -0400, Eoghan Glynn wrote: Note that the notifications that capture these resource state transitions are a long-standing mechanism in openstack that ceilometer has depended on from the very outset. I don't think it's realistic to envisage these

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-10 Thread Thierry Carrez
Hi! There is a lot of useful information in that post (even excluding the part brainstorming solutions) and it would be a shame if it was lost in a sub-thread. Do you plan to make a blog post, or reference wiki page, out of this ? Back to the content, I think a more layered testing approach (as

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-10 Thread David Kranz
On 07/10/2014 08:53 AM, Matthew Treinish wrote: On Thu, Jul 10, 2014 at 08:37:40AM -0400, Eoghan Glynn wrote: Note that the notifications that capture these resource state transitions are a long-standing mechanism in openstack that ceilometer has depended on from the very outset. I don't think

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-10 Thread David Kranz
On 07/10/2014 09:47 AM, Thierry Carrez wrote: Hi! There is a lot of useful information in that post (even excluding the part brainstorming solutions) and it would be a shame if it was lost in a sub-thread. Do you plan to make a blog post, or reference wiki page, out of this ? Back to the

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-10 Thread Sean Dague
On 07/10/2014 09:48 AM, Matthew Treinish wrote: On Wed, Jul 09, 2014 at 09:16:01AM -0400, Sean Dague wrote: I think we need to actually step back a little and figure out where we are, how we got here, and what the future of validation might need to look like in OpenStack. Because I think there

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-10 Thread Doug Hellmann
On Thu, Jul 10, 2014 at 11:56 AM, Sean Dague s...@dague.net wrote: On 07/10/2014 09:48 AM, Matthew Treinish wrote: On Wed, Jul 09, 2014 at 09:16:01AM -0400, Sean Dague wrote: I think we need to actually step back a little and figure out where we are, how we got here, and what the future of

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-09 Thread Matthew Treinish
On Wed, Jul 09, 2014 at 05:41:10AM -0400, Eoghan Glynn wrote: TL;DR: branchless Tempest shouldn't impact on backporting policy, yet makes it difficult to test new features not discoverable via APIs Folks, At the project/release status meeting yesterday[1], I raised the issue that

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-09 Thread Sean Dague
I think we need to actually step back a little and figure out where we are, how we got here, and what the future of validation might need to look like in OpenStack. Because I think there has been some communication gaps. (Also, for people I've had vigorous conversations about this before, realize

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-09 Thread Eoghan Glynn
Thanks for the response Matt, some comments inline. At the project/release status meeting yesterday[1], I raised the issue that featureful backports to stable are beginning to show up[2], purely to facilitate branchless Tempest. We had a useful exchange of views on IRC but ran out of

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-09 Thread Eoghan Glynn
I think we need to actually step back a little and figure out where we are, how we got here, and what the future of validation might need to look like in OpenStack. Because I think there has been some communication gaps. (Also, for people I've had vigorous conversations about this before,

Re: [openstack-dev] [qa][all] Branchless Tempest beyond pure-API tests, impact on backporting policy

2014-07-09 Thread Matthew Treinish
On Wed, Jul 09, 2014 at 01:44:33PM -0400, Eoghan Glynn wrote: Thanks for the response Matt, some comments inline. At the project/release status meeting yesterday[1], I raised the issue that featureful backports to stable are beginning to show up[2], purely to facilitate branchless