There are several changes in Tempest right now trying to add response
schema validation for the 2.26 microversion which added server tags to
the server GET response. This is needed for anything testing a
microversion >=2.26, which several people are trying to add.
We have a similar issue with the 2.3 microversion which is really a bug,
but only exposed in jobs that have run_validation=True which is only in
a non-voting job right now.
I've mostly been debating this in this change:
https://review.openstack.org/#/c/233176/
I've added an item to the nova midcycle meetup agenda to talk about the
plan for handling microversion testing in tempest for nova changes,
specifically around API response validation.
I agree that nova doesn't test response schema validation in tree, so
doing it in tempest is good.
But I'm not sure that we need a new set of tempest tests for every
microversion change in nova, e.g. if it's only touching the API and
database, like server tags, we can test that in nova.
It's also not great having several changes in flight at the same time to
tempest trying to add the same 2.26 response schema because it wasn't
added the at the same time the 2.26 API merged.
I also wonder what it means if someone configures max_microversion in
tempest.conf to something we don't test, like say 2.11, what blows up?
For example, we know that we don't have response validation for 2.3 so
some tests are broken when you run with ssh validation and
microversion>=2.3.
So I'm thinking we should:
1. Always add a schema change to Tempest if a microversion changes a
response.
2. Only add tests to Tempest for a microversion change if it can't be
tested within nova, e.g. you actually need to create a server, or use
other services like glance/cinder/neutron.
mtreinish and sdague will be at the nova midcycle so hopefully they can
represent for the QA team.
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
