On 10/17/2013 10:46 AM, Matthew Treinish wrote:
On Thu, Oct 17, 2013 at 10:10:15AM -0400, Jay Pipes wrote:
On 10/17/2013 08:46 AM, Salvatore Orlando wrote:
Hi,
in the discussion for patch https://review.openstack.org/#/c/50880/ Sean
asked a very reasonable question:
"so are all these [Neutron] extensions always loaded on all
environments? If not, how are we detecting which are?"
So far we've been relying on devstack-gate setup, I think. If the
extension is enabled by devstack-gate, then you can add the test. This
would make sense if tempest is used exclusively by the upstream gate
jobs, which however appears not to be the case. It should therefore be
possible to select which tests should be run according to currently
enabled Neutron extensions, which depend on core and service plugins
One way to solve this issue would be to use tempest configuration
variables; while this is the simplest approach, there are already 25
extensions in Neutron, not counting plugin-specific extensions. This
number can only grow in future releases, so I have a few concerns about
maintainability of this approach because of the high number of
configuration variables.
Another way is to use Neutron API to query available extensions, by
querying /v2.0/extensions
This is actually how we used to control for Nova extensions. I just
went to look at the latest code, though, and while I see the
tempest.services.compute.json.extensions_client.ExtensionsClientJSON.is_enabled()
method exists, I don't see it called any more :(
https://github.com/openstack/tempest/blob/master/tempest/services/compute/json/extensions_client.py#L36
Instead, it looks like a bunch of variables have been added to the
Tempest configuration file that are then checked to see if an
extension should be tested:
https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_flavors_extra_specs.py#L39
This will return the list of currently loaded extensions, from which
extension aliases can be easily extracted; these values can then be used
to conditionally skip tests. This Neutron query can either be done each
time an API test class is initialised or just once - by storing in
memory the list of enabled extensions. If you deem the last approach
reasonable, I'd be happy to provide some code for it. I see many patches
pushed into tempest without an accompanying blueprint/bug; should I
register anyway a blueprint for this work?
I would fully support getting rid of tempest configuration file
variables that "enable" these tests and instead just use the
is_enabled() method that checks a memoized list of extensions pulled
from the /extensions resource to determine whether a test should be
run.
We've actually been going the opposite direction in tempest on purpose. The
concern with auto discovery is that if there is a bug we might end up masking
it if something were to go wrong. With auto discovery we can't tell exactly
what is supposed to be run, so if a test is skipped in the >1000 tests because
something went wrong with auto-discovery it'd be very difficult to tell. By
explicitly stating up front in the config what is enabled we know what is
expected to run. Granted this does shift the burden to devstack (or whatever
else you're using to configure tempest) to set this up properly.
Jay, I originally agreed with you that we can auto-discover things nicely and
trim down the config file. But, we had a couple cases where we found things not
running as expected in the gate and when we fixed it the tests didn't work.
Hi Matt, thanks for sharing the above history. I can see your point and
understand the rationale behind the change in direction.
One thing that might be nice is to make the tempest configuration
variables that switch on/off these extension tests consistently named.
It's tough right now to scan the tempest config file and try to
determine which config options refer to extensions... some use
"ENABLED", others use "AVAILABLE", none of them mention extension or
have "extension" or "ext" prefixes, etc.
Best,
-jay
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
