This is really interesting discussion but was thrown off by the different use 
of ‘functional testing.’ I decided to reconcile it with my understanding and 
ended up with this two-pager. Sharing it in case it helps:

https://docs.google.com/document/d/1ILxfoJlov9lBfuuZtvwW_7bmlGaYYw0UsE1R9lo6XwQ/edit


Regards,

Mark Maglana
QA/Release Engineer
Nexus IS, Inc.
Single Number Reach: 424-225-1309






On Mar 25, 2014, at 4:55 AM, Malini Kamalambal 
<malini.kamalam...@rackspace.com> wrote:

> 
> 
> 
> We are talking about different levels of testing,
> 
> 1. Unit tests - which everybody agrees should be in the individual project
> itself
> 2. System Tests - 'System' referring to (& limited to), all the components
> that make up the project. These are also the functional tests for the
> project.
> 3. Integration Tests - This is to verify that the OS components interact
> well and don't break other components -Keystone being the most obvious
> example. This is where I see getting the maximum mileage out of Tempest.
> 
> "Its not easy to detect what the integration points with other projects are, 
> any project can use any stable API from any other project. Because of this 
> all OpenStack APIs should fit into this category. "
> 
> Any project can use any stable API –but that does not make all API tests , 
> Integration Tests.
> A test becomes Integration test when it has two or more projects interacting 
> in the test.
> 
> Individual projects should be held accountable to make sure that their API's 
> work – no matter who consumes them.
> We should be able to treat the project as a complete system, make API calls 
> and validate that the response matches the API definitions.
> Identifying issues earlier in the pipeline reduces the Total Cost of Quality.
> 
> I agree that Integration Testing is hard. It is complicated because it 
> requires knowledge of how systems interact with each other – and knowledge 
> comes from a lot of time spent on analysis.
> It requires people with project expertise to talk to each other & identify 
> possible test cases.
> openstack-qa is the ideal forum to do this.
> Holding projects responsible for their functionality will help the QA team 
> focus on complicated integration tests.
> 
> "Having a second group writing tests to Nova's public APIs has been really 
> helpful in keeping us honest as well."
> 
> Sounds like a testimonial for more project level testing :)
> 
> 
> I see value in projects taking ownership of the System Tests - because if
> the project is not 'functionally ready', it is not ready to integrate with
> other components of Openstack.
> 
> "What do you mean by not ready?"
> 
> 'Functionally Ready' - The units that make up a project can work together as 
> a system,  all API's have been exercised with positive & negative test cases 
> by treating the project as a complete system.
> There are no known critical bugs. The point here being identify as many 
> issues as possible, earlier in the game.
>  
> But for this approach to be successful, projects should have diversity in
> the team composition - we need more testers who focus on creating these
> tests.
> This will keep the teams honest in their quality standards.
> 
> As long as individual projects cannot guarantee functional test coverage,
> we will need more tests in Tempest.
> But that will shift focus away from Integration Testing, which can be done
> ONLY in Tempest.
> 
> Regardless of whatever we end up deciding, it will be good to have these
> discussions sooner than later.
> This will help at least the new projects to move in the right direction.
> 
> -Malini
> 
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to