Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-27 Thread Malini Kamalambal
The beginning of this thread largely came from the fact that Marconi clearly doing most of their QA not in an upstream way. To be integrated, that needs to change. Marconi has very good test coverage within the project. These tests guarantee functionality at a project level (i.e the API works

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-27 Thread Joe Gordon
On Wed, Mar 26, 2014 at 9:02 PM, Maru Newby ma...@redhat.com wrote: On Mar 26, 2014, at 12:59 PM, Joe Gordon joe.gord...@gmail.com wrote: On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby ma...@redhat.com wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-27 Thread Christopher Yeoh
On Wed, 26 Mar 2014 21:02:58 -0700 Maru Newby ma...@redhat.com wrote: If an API is made and documented properly what domain expertise would be needed to use it? The opposite is true for tempest and the tests themselves. The tempest team focuses on just tests so they know how to write

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Maglana, Mark
This is really interesting discussion but was thrown off by the different use of ‘functional testing.’ I decided to reconcile it with my understanding and ended up with this two-pager. Sharing it in case it helps:

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Joe Gordon
On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby ma...@redhat.com wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com]

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Sean Dague
On 03/25/2014 04:49 AM, Maru Newby wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Maru Newby
On Mar 26, 2014, at 12:59 PM, Joe Gordon joe.gord...@gmail.com wrote: On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby ma...@redhat.com wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-26 Thread Maru Newby
On Mar 26, 2014, at 1:52 PM, Sean Dague s...@dague.net wrote: On 03/25/2014 04:49 AM, Maru Newby wrote: On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-25 Thread Maru Newby
On Mar 21, 2014, at 9:01 AM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-25 Thread Malini Kamalambal
We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2. System Tests - 'System' referring to ( limited to), all the components that make up the project. These are also the functional tests for the project. 3.

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-24 Thread Malini Kamalambal
On 3/21/14 3:49 PM, Rochelle.RochelleGrober rochelle.gro...@huawei.com wrote: From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] snip We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2.

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-24 Thread Joe Gordon
On Fri, Mar 21, 2014 at 9:38 AM, Malini Kamalambal malini.kamalam...@rackspace.com wrote: On 3/21/14 12:01 PM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-21 Thread Malini Kamalambal
I have an etherpad started to document QA requirements https://etherpad.openstack.org/p/Tempest-Graduation-Criteria I hope Sean and the rest of QA team can add their thoughts here. I am also looking for inputs from the Sahara team, while the path to graduation is still fresh in their minds. We can

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-21 Thread David Kranz
On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system,

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-21 Thread Malini Kamalambal
On 3/21/14 12:01 PM, David Kranz dkr...@redhat.com wrote: On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-21 Thread Rochelle.RochelleGrober
From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] snip We are talking about different levels of testing, 1. Unit tests - which everybody agrees should be in the individual project itself 2. System Tests - 'System' referring to ( limited to), all the components that make

[openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Malini Kamalambal
Hello all, I have been working on adding tests in Tempest for Marconi, for the last few months. While there are many amazing people to work with, the process has been more difficult than I expected. Couple of pain-points and suggestions to make the process easier for myself future

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Matthew Treinish
On Thu, Mar 20, 2014 at 11:35:15AM +, Malini Kamalambal wrote: Hello all, I have been working on adding tests in Tempest for Marconi, for the last few months. While there are many amazing people to work with, the process has been more difficult than I expected. Couple of

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Malini Kamalambal
Thanks Matt for your response !! It has clarified some of the 'cloudy areas' ;) So having only looked at the Marconi ML thread and not the actual TC meeting minutes I might be missing the whole picture. But, from what I saw when I looked at both a marconi commit and a tempest commit is that

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Sean Dague
I will agree that the TC language is not as strong as it should be (and really should be clarified, but I don't think that's going to happen until the release is looking solid). Honestly, though, I think Sahara is a good example here of the level of that we expect. They have actively engaged with

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Rochelle.RochelleGrober
-Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context is treating Marconi as a complete system, making Marconi API calls verifying the response -

Re: [openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest

2014-03-20 Thread Joe Gordon
On Thu, Mar 20, 2014 at 1:19 PM, Rochelle.RochelleGrober rochelle.gro...@huawei.com wrote: -Original Message- From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] Sent: Thursday, March 20, 2014 12:13 PM 'project specific functional testing' in the Marconi context