On Wed, Mar 26, 2014 at 9:02 PM, Maru Newby <ma...@redhat.com> wrote:
> > On Mar 26, 2014, at 12:59 PM, Joe Gordon <joe.gord...@gmail.com> wrote: > > > > > > > > > On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby <ma...@redhat.com> wrote: > > > > On Mar 21, 2014, at 9:01 AM, David Kranz <dkr...@redhat.com> wrote: > > > > > On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote: > > >> > > >>> -----Original Message----- > > >>> From: Malini Kamalambal [mailto:malini.kamalam...@rackspace.com] > > >>> Sent: Thursday, March 20, 2014 12:13 PM > > >>> > > >>> 'project specific functional testing' in the Marconi context is > > >>> treating > > >>> Marconi as a complete system, making Marconi API calls & verifying > the > > >>> response - just like an end user would, but without keystone. If one > of > > >>> these tests fail, it is because there is a bug in the Marconi code , > > >>> and > > >>> not because its interaction with Keystone caused it to fail. > > >>> > > >>> "That being said there are certain cases where having a project > > >>> specific > > >>> functional test makes sense. For example swift has a functional test > > >>> job > > >>> that > > >>> starts swift in devstack. But, those things are normally handled on a > > >>> per > > >>> case > > >>> basis. In general if the project is meant to be part of the larger > > >>> OpenStack > > >>> ecosystem then Tempest is the place to put functional testing. That > way > > >>> you know > > >>> it works with all of the other components. The thing is in openstack > > >>> what > > >>> seems > > >>> like a project isolated functional test almost always involves > another > > >>> project > > >>> in real use cases. (for example keystone auth with api requests) > > >>> > > >>> " > > > > > > > > >>> > > >>> One of the concerns we heard in the review was 'having the functional > > >>> tests elsewhere (I.e within the project itself) does not count and > they > > >>> have to be in Tempest'. > > >>> This has made us as a team wonder if we should migrate all our > > >>> functional > > >>> tests to Tempest. > > >>> But from Matt's response, I think it is reasonable to continue in our > > >>> current path & have the functional tests in Marconi coexist along > with > > >>> the tests in Tempest. > > >>> > > >> I think that what is being asked, really is that the functional tests > could be a single set of tests that would become a part of the tempest > repository and that these tests would have an ENV variable as part of the > configuration that would allow either "no Keystone" or "Keystone" or some > such, if that is the only configuration issue that separates running the > tests isolated vs. integrated. The functional tests need to be as much as > possible a single set of tests to reduce duplication and remove the > likelihood of two sets getting out of sync with each other/development. If > they only run in the integrated environment, that's ok, but if you want to > run them isolated to make debugging easier, then it should be a > configuration option and a separate test job. > > >> > > >> So, if my assumptions are correct, QA only requires functional tests > for integrated runs, but if the project QAs/Devs want to run isolated for > dev and devtest purposes, more power to them. Just keep it a single set of > functional tests and put them in the Tempest repository so that if a > failure happens, anyone can find the test and do the debug work without > digging into a separate project repository. > > >> > > >> Hopefully, the tests as designed could easily take a new > configuration directive and a short bit of work with OS QA will get the > integrated FTs working as well as the isolated ones. > > >> > > >> --Rocky > > > This issue has been much debated. There are some active members of our > community who believe that all the functional tests should live outside of > tempest in the projects, albeit with the same idea that such tests could be > run either as part of today's "real" tempest runs or mocked in various ways > to allow component isolation or better performance. Maru Newby posted a > patch with an example of one way to do this but I think it expired and I > don't have a pointer. > > > > I think the best place for functional api tests to be maintained is in > the projects themselves. The domain expertise required to write api tests > is likely to be greater among project resources, and they should be tasked > with writing api tests pre-merge. The current 'merge-first, test-later' > procedure of maintaining api tests in the Tempest repo makes that > impossible. Worse, the cost of developing functional api tests is higher > in the integration environment that is the Tempest default. > > > > > > If an API is made and documented properly what domain expertise would be > needed to use it? The opposite is true for tempest and the tests > themselves. The tempest team focuses on just tests so they know how to > write good tests and are able to leverage common underlying framework code. > > Given that documentation is typically finalized only late in the cycle, > are you suggesting that we forego api testing until possibly well after the > code has been written? Plus, it is a rare api that doesn't have to evolve > in response to real-world experience with early implementations. The > sooner we can write functional api tests, the sooner we can identify > shortcomings that need to be addressed - and the less costly they will be > to fix. > I agree with this sentiment, but I think this can be solved via tempest as well. > As to expertise, would you agree that we have a much greater need for QA > expertise than the existing team can hope to meet? I would argue that this > is the case, and believe that the more we can diffuse a testing mentality > from the QA team to the projects, the better off OpenStack will be. > Tempest has proven a very effective guardian against regression, but that > role can only grow more difficult with time and a lack of focus will > complicate the effort unnecessarily. > If Tempest doesn't have enough human resources, then teams should work closer with tempest to fix that. > > As to framework, the talk of 'libifying' Tempest promises to allow the > projects the same benefit of common framework that Tempest currently enjoys. > > > > > Yes 'merge-first, test-later' is bad. But There are other ways of fixing > this then moving the tests out of tempest. Hopefully cross project > dependencies in zuul will help make this workflow easier. > > > > > > > > The patch in question [1] proposes allowing pre-merge functional api > test maintenance and test reuse in an integration environment. > > > > > > m. > > > > 1: https://review.openstack.org/#/c/72585/ > > > > > IMO there are valid arguments on both sides, but I hope every one > could agree that functional tests should not be arbitrarily split between > projects and tempest as they are now. The Tempest README states a desire > for "complete coverage of the OpenStack API" but Tempest is not close to > that. We have been discussing and then ignoring this issue for some time > but I think the recent action to say that Tempest will be used to determine > if something can use the OpenStack trademark will force more completeness > on tempest (more tests, that is). I think we need to resolve this issue but > it won't be easy and modifying existing api tests to be more flexible will > be a lot of work. But at least new projects could get on the right path > sooner. > > > > > > This touches on another point that I forgot to bring up before. As a > cloud deployer I want a test suite that allows me to exercise all of my > APIs to make sure everything is working. Although OpenStack runs extensive > tests upstream, upstream cannot test every possible configuration, so as a > deployer I would want rerun the same test suite. This is exactly why > TripleO is getting ready to run tempest. Does this test suite have to just > tempest? Perhaps not, but if we go down the road of moving more API tests > into individual projects I think we still need easy way for a cloud > deployer to run all the API tests against a live cloud from a single > interface. > > I don't think project-maintained functional api tests would in any way > preclude Tempest from continuing to be the go-to test suite for > deployers/operators. There is nothing stopping project-maintained > functional api tests from being consumed by Tempest. Tempest could > explicitly depend on the projects, either the whole package or a special > package that contained only the api tests. Or Tempest could copy the api > tests, as is done with oslo incubator. In either case, the process could > be purely mechanical. > I agree with you here, if we do decide to go down the route you are proposing (functional tests outside of tempest repo), then having this mechanism sorted out should be a pre-requisite. > > > m. > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev