Re: [openstack-dev] [qa] In-tree functional test vision
On Mon, 25 Aug 2014, Joe Gordon wrote: On Mon, Aug 4, 2014 at 3:29 AM, Chris Dent chd...@redhat.com wrote: For constraints: Will tempest be available as a stable library? Is using tempest (or other same library across all projects) a good or bad thing? Seems there's some disagreement on both of these. Yes, there is a separate thread on spinning out a tempest-lib (not sure on what final name will be yet) that functional tests can use. Although I think there is a lot to be done before needing the tempest-lib. What's the status of tempest-lib? Looking at the repo it appears that other things may be taking priority at the moment. As I said in notifications thread: With summit approaching and kilo open for business, now seems to be talking about what kinds of structure we want to apply to in-tree functional testing. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] In-tree functional test vision
On Tue, Oct 07, 2014 at 04:28:23PM +0100, Chris Dent wrote: On Mon, 25 Aug 2014, Joe Gordon wrote: On Mon, Aug 4, 2014 at 3:29 AM, Chris Dent chd...@redhat.com wrote: For constraints: Will tempest be available as a stable library? Is using tempest (or other same library across all projects) a good or bad thing? Seems there's some disagreement on both of these. Yes, there is a separate thread on spinning out a tempest-lib (not sure on what final name will be yet) that functional tests can use. Although I think there is a lot to be done before needing the tempest-lib. What's the status of tempest-lib? Looking at the repo it appears that other things may be taking priority at the moment. Things were blocked briefly by global requirements freeze. I'll hopefully be cutting the first release this week, just waiting on one patch to go through. We already have most of the pieces in place already to start using it on ci jobs so it should just be a matter of getting: https://review.openstack.org/#/c/122478/ landed before you can import it anywhere. That being said it's still a lower priority dev item compared to some other tasks, mostly because we don't have too many interfaces in tempest which are considered stable enough to migrate yet. The long term scope of the library hasn't been clearly defined yest, so not entirely clear what things are worth migrating into the library at this point. Next up I expect to migrate the auth layer and the base rest client. This will allow projects to build service specific clients out of tree, which I've seen commonly done for projects prior to incubation. This won't include the service clients in tempest though, because we expect to make some changes to that interface soon. That being said I'm not sure how long the additional library migrations will take. As I said in notifications thread: With summit approaching and kilo open for business, now seems to be talking about what kinds of structure we want to apply to in-tree functional testing. There is nothing stopping you from doing this. Tempest-lib is not a requirement for doing this at all, and should really only be useful for some specific types of testing. It shouldn't be required for functional testing in general. Also, at this point tempest-lib only contains the CLI testing framework which is hardly useful for spinning up project specific functional testing. -Matt Treinish pgpHt8yHxOH3G.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] In-tree functional test vision
On Mon, Aug 4, 2014 at 3:29 AM, Chris Dent chd...@redhat.com wrote: In the Thoughts on the patch test failure rate and moving forward thread[1] there's discussion of moving some of the burden for functional testing to the individual projects. This seems like a good idea to me, but also seems like it could be a source of confusion so I thought I'd start another thread to focus on the details of just this topic, separate from the gate-oriented discussion in the other. In a couple of messages[2] Sean mentions the vision. Is there a wiki page or spec or other kind of document where this nascent vision is starting to form? Even if we can't quite get started just yet, it would be good to have an opporunity to think about the constraints and goals that we'll be working with. There is no single document on this that I know of. But two good places to start are the functional testing systems we have for swift and neutron: http://logs.openstack.org/70/116570/3/check/check-swift-dsvm-functional/4894055/console.html http://logs.openstack.org/70/116570/3/check/check-neutron-dsvm-functional/bf68c61/console.html http://git.openstack.org/cgit/openstack/neutron/tree/tox.ini#n29 Not just the goal of moving tests around, but what, for example, makes a good functional test? Here is a non-exhaustive list of some differences between what belongs in tempest and functional tests * functional tests should ideally not need external services (external services makes it closer to an integration test) * functional tests can be whitebox tests * a functional test environment should be easier to set up then devstack For constraints: Will tempest be available as a stable library? Is using tempest (or other same library across all projects) a good or bad thing? Seems there's some disagreement on both of these. Yes, there is a separate thread on spinning out a tempest-lib (not sure on what final name will be yet) that functional tests can use. Although I think there is a lot to be done before needing the tempest-lib. Personally I'm quite eager to to vastly increase the amount of testing I can do on my own machine(s) before letting the gate touch my code. Why can't you run devstack locally? Maybe there are some changes we can make so its easier to run devstack locally first. [1] http://lists.openstack.org/pipermail/openstack-dev/2014- July/thread.html#41057 [2] http://lists.openstack.org/pipermail/openstack-dev/2014-July/041188.html http://lists.openstack.org/pipermail/openstack-dev/2014-July/041252.html -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] In-tree functional test vision
On Mon, 25 Aug 2014, Joe Gordon wrote: [Other stuff snipped, thanks for that, good to have some pointers.] Why can't you run devstack locally? Maybe there are some changes we can make so its easier to run devstack locally first. I do run a local devstack, and throw in some tempest and grenade every now and again too. But in terms of automated local testing in the project tree there are places it is difficult for clean unit tests to reach. Sure we can make really hairy mocks, but that results in tests which a) make no sense b) it is hard to have any confidence in. Thus in tree functional tests: * to reach places unit tests won't go * to not have the noise of all that mock and OO mess * to have some faith in the end to end The sorts of things that require provisioning of temporary datastores, interception of wsgi apps, in process message queues... -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] In-tree functional test vision
On Mon, Aug 25, 2014 at 2:50 PM, Chris Dent chd...@redhat.com wrote: On Mon, 25 Aug 2014, Joe Gordon wrote: [Other stuff snipped, thanks for that, good to have some pointers.] Why can't you run devstack locally? Maybe there are some changes we can make so its easier to run devstack locally first. I do run a local devstack, and throw in some tempest and grenade every now and again too. But in terms of automated local testing in the project tree there are places it is difficult for clean unit tests to reach. Sure we can make really hairy mocks, but that results in tests which a) make no sense b) it is hard to have any confidence in. Thus in tree functional tests: * to reach places unit tests won't go * to not have the noise of all that mock and OO mess * to have some faith in the end to end The sorts of things that require provisioning of temporary datastores, interception of wsgi apps, in process message queues... agreed -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
