On 7/13/2016 10:18 PM, Ken'ichi Ohmichi wrote:
Hi Matt,
2016-07-14 11:55 GMT+09:00 Matt Riedemann <[email protected]>:
There are several changes in Tempest right now trying to add response schema
validation for the 2.26 microversion which added server tags to the server
GET response. This is needed for anything testing a microversion >=2.26,
which several people are trying to add.
We have a similar issue with the 2.3 microversion which is really a bug, but
only exposed in jobs that have run_validation=True which is only in a
non-voting job right now.
I've mostly been debating this in this change:
https://review.openstack.org/#/c/233176/
I've added an item to the nova midcycle meetup agenda to talk about the plan
for handling microversion testing in tempest for nova changes, specifically
around API response validation.
I agree that nova doesn't test response schema validation in tree, so doing
it in tempest is good.
But I'm not sure that we need a new set of tempest tests for every
microversion change in nova, e.g. if it's only touching the API and
database, like server tags, we can test that in nova.
I think it is nice to implement every microversion tests in Tempest
for preparing the base/minimum microversion bump in the future.
On API microversions mechanism, we have defined the minimum
microversion(now version 2.1) can be increased.
Every microversion test will provide an option when we want to bump it
to verify a new minimum microversion is stable by Tempest as an
integrated test.
We're pretty far out from raising the minimum required microversion in
Nova I think, like several releases at least.
The issue I have really is a few releases ago there was a push to pair
down how much is tested in Tempest in the integrated gate, and if tests
only hit the API for a single service, those tests should live in the
project/service that's being tested. This is why the CLI tests moved out
and a bunch of things that only hit the API and database.
We'd be going back on that now if we need to implement a test for every
microversion that shows up.
Maybe Nova needs to think about a Tempest plugin for stuff like that?
It's also not great having several changes in flight at the same time to
tempest trying to add the same 2.26 response schema because it wasn't added
the at the same time the 2.26 API merged.
I also wonder what it means if someone configures max_microversion in
tempest.conf to something we don't test, like say 2.11, what blows up? For
example, we know that we don't have response validation for 2.3 so some
tests are broken when you run with ssh validation and microversion>=2.3.
Ideally, the response schema also should be implemented on Tempest.
but it is difficult to catch up because the microversion bumping is
fast on Nova side.
The max_microversion option is useful to operate the same Tempest to
master/stable branches which are different microversions.
Thanks
Ken Ohmichi
---
So I'm thinking we should:
1. Always add a schema change to Tempest if a microversion changes a
response.
2. Only add tests to Tempest for a microversion change if it can't be tested
within nova, e.g. you actually need to create a server, or use other
services like glance/cinder/neutron.
mtreinish and sdague will be at the nova midcycle so hopefully they can
represent for the QA team.
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
