On Thu, Oct 30, 2014 at 6:30 AM, Eoghan Glynn <egl...@redhat.com> wrote:
> > > > Hi everyone, > > > > Before we start the larger discussion at summit next week about the > future of > > testing in OpenStack - specifically about spinning up functional testing > and > > how > > it relates to tempest - I would like to share some of my thoughts on how > we > > can > > get things started and how I think they'll eventually come together. > > > > Currently in tempest we have a large number of tests (mostly api-focused) > > which are probably a better fit for a project's functional test suite. > The > > best > > example I can think of is the nova flavors tests. Validation of flavor > > manipulation doesn't need to run in the integrated test suite on every > commit > > to > > every project because it only requires Nova. A simple win for initiating > > in-tree > > functional testing would be to move these kinds of tests into the > projects > > and > > run the tests from the project repos instead of from tempest. > > > > This would have the advantage of making tempest slimmer for every project > > and begin the process of getting projects to take responsibility for > their > > functional testing rather than relying on tempest. As tests are moved > tempest > > can start to become the integration test suite it was meant to be. It > would > > retain only tests that involve multiple projects and stop being the > OpenStack > > black box testing suite. I think that this is the right direction for > tempest > > moving forward, especially as we move to having project-specific > functional > > testing. > > > > Doing this migration is dependent on some refactors in tempest and moving > > the required bits to tempest-lib so they can be easily consumed by the > > other projects. This will be discussed at summit, is being planned > > for implementation this cycle, and is similar to what is currently in > > progress > > for the cli tests. > > > > The only reason this testing existed in tempest in the first place was as > > mechanism to block and then add friction against breaking api changes. > > Tempest's > > api testing has been been pretty successful at achieving these goals. > We'll > > want > > to ensure that migrated tests retain these characteristics. If we are > using > > clients from tempest-lib we should get this automatically since to break > > the api you'd have to change the api client. Another option proposed was > to > > introduce a hacking rule that would block changes to api tests at the > same > > time > > other code was being changed. > > > > There is also a concern for external consumers of tempest if we move the > > tests > > out of the tempest tree (I'm thinking refstack). I think the solution is > > to maintain a load_tests discovery method inside of tempest or elsewhere > that > > will run the appropriate tests from the other repos for something like > > refstack. > > Assuming that things are built in a compatible way using the same > framework > > then > > running the tests from separate repos should be a simple matter of > pointing > > the > > test runner in the right direction. > > > > I also want to comment on the role of functional testing. What I've > proposed > > here is only one piece of what project specific functional testing > should be > > and just what I feel is a good/easy start. I don't feel that this should > be > > the only testing done in the projects. I'm suggesting this as a first > > step because the tests already exist and it should be a relatively simple > > task. > > I also feel that using tempest-lib like this shouldn't be a hard > requirement. > > Ideally the client definitions shouldn't have to live externally, or if > they > > did > > they would be the official clients, but I am suggesting this as a first > step > > to > > start a migration out of tempest. > > > > I don't want anyone to feel that they need block their functional testing > > efforts until tempest-lib becomes more useable. The larger value from > > functional > > testing is actually in enabling testing more tightly coupled to the > projects > > (e.g. whitebox testing). I feel that any work necessary to enable > functional > > testing should be prioritized. > > Thanks Matt for getting the ball rolling on this conversation in advance > of summit. > > Probably stating the obvious here, but I feel we should make a concious > effort to keep the approaches to in-tree functional testing as consistent > as possible across projects. > > Towards that end, it would be good for folks with an interest in this area > to attend each other's sessions where possible: > > +1 > Cross-project: Tue, 12:05 [1] > Heat: Wed, 13:50 [2] > Nova: Wed, 16:30 [3] > Ceilometer: Wed, 17:20 [4] > QA: Wed, 17:20 [5] I think Keystone was planning on dedicating some time to this on Friday, so our dev/hackathon day. I'm interested in the process that will be in place for tracking status of the "functional test migration" if there will be one. > Unfortunately there's a clash there between the QA tempest-lib session > and the ceilo session. I'm not sure how reasonable it would be to make > a last-minute schedule change to avoid that clash. > > Cheers, > Eoghan > > [1] > http://kilodesignsummit.sched.org/event/575938e4837e8293615845582d7e3e7f > [2] > http://kilodesignsummit.sched.org/event/eb261fb08b18ec1eaa2c67492e7fc385 > [3] > http://kilodesignsummit.sched.org/event/271a9075c1ced6c1269100ff4b8efc37 > [4] > http://kilodesignsummit.sched.org/event/daf63526a1883e84cec107c70cc6cad3 > [5] > http://kilodesignsummit.sched.org/event/a45e9af23c2c3fc17c12cc22e8bb3704 > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
