On 04/27/2014 10:02 PM, Matthew Treinish wrote:
On Mon, Apr 28, 2014 at 01:01:00AM +0000, Kenichi Oomichi wrote:
Hi,
Sorry for my late response, but I'd like to discuss this again.
Now we are working for adding Nova API responses checks to Tempest[1] to
block backward incompatible changes.
With this work, Tempest checks each response(status code, response body)
and raises a test failure exception if detecting something unexpected.
For example if some API parameter, which is defined as 'required' Tempest
side, does not exist in response body, Tempest test fails.
We are defining API parameters as 'required' if they are not API extensions
or they are not depended on Nova configuration. In addition now Tempest
allows additional API parameters, that means Tempest does not fail even if
Nova response includes unexpected API parameters. Because I think the removal
of API parameter causes backward incompatible issue but the addition does not
cause it.
So, AIUI we can only add parameters to an API with a new extension. The API
change guidelines also say that adding new properties must be conditional:
https://wiki.openstack.org/wiki/APIChangeGuidelines#Generally_Considered_OK
I just wanted to note that the original referenced wiki page, assembled
by markmc and myself, did not specify the need for an extension in order
to add a value to a return dict or add a value to a dict argument if the
existing api would ignore it. This was changed (and honestly I did not
notice this change at the time) last August to require an extension:
https://wiki.openstack.org/w/index.php?title=APIChangeGuidelines&diff=prev&oldid=28593.
Is there any trace left of discussions around that change?
The original wording allowed the api to evolve as long as a "reasonable"
application would not be broken. Essentially the extra value becomes
optional and new client or server code can check for it. The new
definition is quite strict and similar to what leads to many Windows
APIs having names like CreateWindowEx. Not saying being strict is bad,
but it will require a lot of changes to the client libraries as well as
tempest because there are a lot of extensions that are not checked by
either.
-David
Adding or removing a parameter to an API is a backwards incompatible change IMO
for the exact reasons you mentioned here. If we have to worry about it in
tempest then end users do as well.
This is also why there are a bunch of nova v2 extensions that just add
properties to an existing API. I think in v3 the proposal was to do this with
microversioning of the plugins. (we don't have a way to configure
microversioned v3 api plugins in tempest yet, but we can cross that bridge when
the time comes) Either way it will allow tempest to have in config which
behavior to expect.
In this situation, there is a problem related to branchless Tempest.
When we define new API parameter as 'required', Tempest against old release
would fail.
So I feel that if we are marking something in the API as required in tempest
makes the test fail in a previous release than that should be considered a bug
in the old release (assuming it was correct to mark it as required), and that
should be backportable fix.
I think we need to define new parameters, which do not depended on the
configuration, as 'required' in Tempest when we have added them in the
development cycle because of blocking backward incompatible changes in
the future. However these parameters are new and old releases don't contain
them, so the Tempest change causes failures against old releases tests.
I agree we should mark the required parameters as those that are used without
any extensions. If we can also conditionally check those not marked as required
based on the enabled extensions or features in the tempest config that would
provide the best coverage.
So for master branch development on tempest I think we should only concern
ourselves with getting these things to work against Juno and Icehouse. I think
Havana support using master is a lost cause at this point so we'll keep the
stable branch around until it's EOL. So hopefully we can lock down things in
tempest with the new api attribute tests quickly so we can block the Juno
additions that would violate the stability guidelines. It would be a shame if
we managed to allow a breaking API change into an API since the release. (but
hopefully it would be an easy backport or revert)
Case: add new parameter 'A' in Juno cycle
Icehouse Juno K L
--*-------------------*-------------------*-------------------*--
Nova: new parameter 'A'
Tempest: define 'A' as 'required'
block 'A' removal block ..
test fails due to non-existent 'A'
So in this example I feel that parameter 'A' can only be added as an extension.
(or some other condition) If it's not then it's a breaking api change which will
be caught which is the desired behavior from tempest here.
Thanks,
Matt Treinish
I have not found enough idea for this yet.
Thanks
Ken'ichi Ohmichi
---
[1]: https://blueprints.launchpad.net/tempest/+spec/nova-api-attribute-test
-----Original Message-----
From: Sean Dague [mailto:s...@dague.net]
Sent: Monday, April 14, 2014 10:22 PM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [all] Branchless Tempest QA Spec - final draft
As we're coming up on the stable/icehouse release the QA team is looking
pretty positive at no longer branching Tempest. The QA Spec draft for
this is here -
http://docs-draft.openstack.org/77/86577/2/check/gate-qa-specs-docs/3f84796/doc/build/html/specs/branchless-tempest.
html
and hopefully address a lot of the questions we've seen so far.
Additional comments are welcome on the review -
https://review.openstack.org/#/c/86577/
or as responses on this ML thread.
-Sean
--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
