Thanks Matt for bringing this up.
On Thu, Jul 14, 2016 at 11:55 AM, Matt Riedemann <mrie...@linux.vnet.ibm.com> wrote: > There are several changes in Tempest right now trying to add response schema > validation for the 2.26 microversion which added server tags to the server > GET response. This is needed for anything testing a microversion >=2.26, > which several people are trying to add. > > We have a similar issue with the 2.3 microversion which is really a bug, but > only exposed in jobs that have run_validation=True which is only in a > non-voting job right now. > > I've mostly been debating this in this change: > > https://review.openstack.org/#/c/233176/ > > I've added an item to the nova midcycle meetup agenda to talk about the plan > for handling microversion testing in tempest for nova changes, specifically > around API response validation. > > I agree that nova doesn't test response schema validation in tree, so doing > it in tempest is good. > > But I'm not sure that we need a new set of tempest tests for every > microversion change in nova, e.g. if it's only touching the API and > database, like server tags, we can test that in nova. > > It's also not great having several changes in flight at the same time to > tempest trying to add the same 2.26 response schema because it wasn't added > the at the same time the 2.26 API merged. > > I also wonder what it means if someone configures max_microversion in > tempest.conf to something we don't test, like say 2.11, what blows up? For > example, we know that we don't have response validation for 2.3 so some > tests are broken when you run with ssh validation and microversion>=2.3. you mean if we increase min_microversion? For max_microversion capping, It should just skip the tests which are going out of configured boundary. But yes there will be failure if we increase the min_microversion to some higher version which schema is missing (as tests is not added). 2.3 schema was missed as this tests was not added yet in tempest. So we might have same case in future also. I am trying to add all version schema needed till Mitaka max supported version which is 2.25 in https://review.openstack.org/#/c/287605/ After that, we should have all lower version schema available for other version tests(<2.25). This should make things easy for other new tests. > > So I'm thinking we should: > > 1. Always add a schema change to Tempest if a microversion changes a > response. We should not actually. Schema are more error prone if we add manually. We have to be very careful on all attributes, their type and range etc which can be verified with tests only. Adding schema in Tempest may end up adding wrong schema in Tempest. > > 2. Only add tests to Tempest for a microversion change if it can't be tested > within nova, e.g. you actually need to create a server, or use other > services like glance/cinder/neutron. Almost +1. Actually This is fine till we do not increase the min_microversion. Any new version tests coming in Tempest should add lower version schema also if needed. This is mentioned very clearly in doc  also. But if anytime we increase the minimum version (which is almost impossible) then yes, we should have tests for all microversion so that no failure with schema things. Honestly saying, I would like to ignore this case as of now as very rare to rare that nova will bump minimum version. I am agree on this But also want to tests each microversion changes through schema which does not mean we need to add tests for each microversion. Higher microversion tests(which are valid for Tempest like interacting with other services etc) can verify same API's lower version changes through schema automatically. Tempest testing which does schema validation are very useful to find API bugs. We have found many in past and recently also . How about this: - Do not add tests for changes which can be tested in Nova as mentioned by you in 2nd point with below exception: - Exception- If that version need schema change and those schema changes are not covered by any higher version tests then add tests in Tempest. And this we can done at end of each cycle to make sure each version changes in this cycle are tested thorough schema validation (is change in schema) in Tempest. > > mtreinish and sdague will be at the nova midcycle so hopefully they can > represent for the QA team. .. http://docs.openstack.org/developer/tempest/microversion_testing.html#notes-about-compute-microversion-tests .. https://bugs.launchpad.net/nova/+bug/1602044 > > -- > > Thanks, > > Matt Riedemann > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev