On 10/27/2015 9:55 PM, Emilien Macchi wrote:
As a user[1], I would like to functionally test OpenStack services. I'm using Tempest (which is AFIK [2] the official OpenStack project for functional testing) and am able to validate that Puppet OpenStack module actually deploy services & make them work together which is the goal of Puppet OpenStack Integration testing [3]. Until now I was happy, until this bug [4] (TL;DR: Aodh can't be testing with Tempest which is a bug I'm working on, and not really related to this thread). I realized Aodh [5] (and apparently some other projects like Ceilometer) were using something else (gabbi [6]) for testing. How come some big tent projects do not use Tempest anymore for functional testing? I thought there was/is a move with tempest plugins that will help projects to host their tempest tests in their repos. Am I missing something? Any official decision taken? Is gabbi supported by OpenStack? I feel like there is currently 2 paths that try to do the same thing and as a user, I'm not happy. Please help me to understand, Thank you. [1] a user who deploy Puppet OpenStack modules in OpenStack Infra for CI purposes [2] http://goo.gl/sgI2D8 http://goo.gl/DTR1cL [3] https://github.com/openstack/puppet-openstack-integration#overview [4] https://bugs.launchpad.net/tempest/+bug/1509885 [5] https://github.com/openstack/aodh [6] https://pypi.python.org/pypi/gabbi/ __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
This might be helpful. I noticed recently that nova didn't have any API functional testing for a wrinkle in one of it's APIs. I opened a nova bug to add the functional testing (in nova's tree) since it only relies on having a wsgi server for the API call, and I think a sqlite database for stubbing out some resources.
A patch was proposed to Tempest [1] which I -1'ed since I thought we could just contain that code in nova's tree since there wasn't any cross-project interaction needed (nova functional tests don't run against a live devstack so we don't have glance/cinder/neutron).
mtreinish elaborated on the reason for not having it in Tempest in the review:
"Yeah, for something like this I feel it's more appropriate inside nova itself. It'll be far more efficient to test it there. This only really needs to be a tempest test if you want it to be externally covered.
Since it can be more effectively tested inside of nova, and is self contained in nova, the rule of thumb I use for a case like this is basically: if this is something you want to enforce every cloud provider to do correctly for all time. (in both directions, since deployments aren't always running the latest) You also have to weigh that against the cost of enforcing this on a change to every project's commits.
This is because the primary advantage tempest gives you over an in tree functional test is external visibility. Tempest is used a ton of different places to test real clouds, not just for gating. (things like defcore, CD setups, etc) So if we can test it more effectively inside of nova's functional tests the only major reason to add it to tempest would to take advantage of that."
Hopefully that helps. I think that statement is something that we should get into the Tempest docs, i.e. a section on 'when is a test appropriate for tempest'.
[1] https://review.openstack.org/#/c/233808/ -- Thanks, Matt Riedemann __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
