Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Ghanshyam Mann]

On Mon, Mar 13, 2017 at 7:37 PM, Andrea Frittoli 
wrote:

> On Sat, Mar 11, 2017 at 10:45 PM Matt Riedemann 
> wrote:
>
> On 3/10/2017 3:02 PM, Andrea Frittoli wrote:
> >
> > We had a couple of sessions related to this topic at the PTG [0][1].
> >
> > We agreed that we want to still maintain integration tests only in
> > Tempest, which means that API micro versions that have no integration
> > impact can be tested via functional tests.
>
> To be clear, "integration" here means tests that span operations across
> multiple services, correct? Like a compute test that first creates a
> port in the networking service and a volume in the block storage service
> and then uses those to create a server and maybe take a snapshot of it
> which is then verified was uploaded to the image service.
>
> Yes, indeed.
>
>
>
> The non-integration things are self-contained in a single service, like
> if all you need to do is create an aggregate, show it's details and
> validate the response, at a particular microversion, we can just do that
> in nova functional tests, and it's not necessary in Tempest.
>
> It might be worth having a definition of this policy in the Tempest docs
> so when people ask this question again you can just point at the docs.
>
>
> I agree, we need to go a better job at documenting this.
> I just want to avoid having a black and white rule that would not work.
>
> To me tests that should be in Tempest are tests that make sense to run
> against any change in any repo, and the reasons for that can be many:
> - the test involves multiple services (more that "it uses a token though")
> - the test covers a feature which is key for interoperability
> - the test must be reliable and not take too long
>
> ​+1. I added those in https://review.openstack.org/#/c/444727/​



> Based on this I don't it's possible to completely avoid the discussion
> about
> which test should in Tempest and which not, it's something to be considered
> on a test by test basis, based on a set of guidelines.
>
> We have an initial definition of scope in [0], but it's probably worth to
> elaborate
> more on it. I'll put up a patch so that the discussion on it can continue
> in
> gerrit.
>
> [0] https://docs.openstack.org/developer/tempest/test-
> removal.html#tempest-scope
>
>
> >
> > In terms of which versions we test in the gate, for nova we always run
> > with min_microversion = None and max_microversion = latest, which means
> > that all tests will be executed.
> > Since micro versions are incremental, and each micro version usually
> > involves no or one test on Tempest side, I think it will be a while
> > before this becomes an issue for the common gate.
>
> We test max_microversion=latest only on master. On the devstack stable
> branch we cap the max_microversion, e.g.:
>
> Thanks, good point.
>
>
>
> https://github.com/openstack-dev/devstack/blob/stable/
> newton/lib/tempest#L339
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Ken'ichi Ohmichi]

2017-03-13 12:23 GMT-07:00 Jim Rollenhagen :
> On Mon, Mar 13, 2017 at 12:58 PM, Chris Friesen
>  wrote:
>>
>> On 03/10/2017 01:37 PM, John Griffith wrote:
>>
>>> Now that micro-versions are *the API versioning scheme to rule them all*
>>> one
>>> question I've not been able to find an answer for is what we're going to
>>> promise
>>> here for support and testing.  My understanding thus far is that the
>>> "community"
>>> approach here is "nothing is ever deprecated, and everything is supported
>>> forever".
>>
>>
>> Nova has so far taken this approach, but there has been talk of bumping
>> the minimum required microversion at every dev gathering.  It hasn't
>> happened yet, but if the support costs of maintaining the compat code
>> becomes too high then it could happen.
>
>
> Indeed. We discussed this at the PTG a bit[0], and plan to use ironic as an
> experiment for this. It's an admin-only API, so the API users should be the
> same (or in contact with) the folks deploying it, and so it shouldn't be as
> surprising. We hope to get some feedback and find out if doing this is as
> terrible as we keep saying.

That seems a nice plan.
The effective scope of bumping minimum microversion could be small and
it would be easier to get feedback from administrators by comparing
normal API consumers.

Thanks

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Jim Rollenhagen]

On Mon, Mar 13, 2017 at 12:58 PM, Chris Friesen  wrote:

> On 03/10/2017 01:37 PM, John Griffith wrote:
>
> Now that micro-versions are *the API versioning scheme to rule them all*
>> one
>> question I've not been able to find an answer for is what we're going to
>> promise
>> here for support and testing.  My understanding thus far is that the
>> "community"
>> approach here is "nothing is ever deprecated, and everything is supported
>> forever".
>>
>
> Nova has so far taken this approach, but there has been talk of bumping
> the minimum required microversion at every dev gathering.  It hasn't
> happened yet, but if the support costs of maintaining the compat code
> becomes too high then it could happen.
>

Indeed. We discussed this at the PTG a bit[0], and plan to use ironic as an
experiment for this. It's an admin-only API, so the API users should be the
same (or in contact with) the folks deploying it, and so it shouldn't be as
surprising. We hope to get some feedback and find out if doing this is as
terrible as we keep saying.

// jim

[0] line 146 here:
https://etherpad.openstack.org/p/ptg-architecture-workgroup
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Chris Friesen]

On 03/10/2017 01:37 PM, John Griffith wrote:


Now that micro-versions are *the API versioning scheme to rule them all* one
question I've not been able to find an answer for is what we're going to promise
here for support and testing.  My understanding thus far is that the "community"
approach here is "nothing is ever deprecated, and everything is supported 
forever".


Nova has so far taken this approach, but there has been talk of bumping the 
minimum required microversion at every dev gathering.  It hasn't happened yet, 
but if the support costs of maintaining the compat code becomes too high then it 
could happen.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Andrea Frittoli]

On Mon, Mar 13, 2017 at 6:52 AM Ghanshyam Mann 
wrote:

> On Sun, Mar 12, 2017 at 7:40 AM, Matt Riedemann 
> wrote:
>
> On 3/10/2017 3:02 PM, Andrea Frittoli wrote:
>
>
> We had a couple of sessions related to this topic at the PTG [0][1].
>
> We agreed that we want to still maintain integration tests only in
> Tempest, which means that API micro versions that have no integration
> impact can be tested via functional tests.
>
>
> To be clear, "integration" here means tests that span operations across
> multiple services, correct? Like a compute test that first creates a port
> in the networking service and a volume in the block storage service and
> then uses those to create a server and maybe take a snapshot of it which is
> then verified was uploaded to the image service.
>
> The non-integration things are self-contained in a single service, like if
> all you need to do is create an aggregate, show it's details and validate
> the response, at a particular microversion, we can just do that in nova
> functional tests, and it's not necessary in Tempest.
>
> It might be worth having a definition of this policy in the Tempest docs
> so when people ask this question again you can just point at the docs.
>
>
> ​Yes, that will be helpful to have those agreement in doc​. I am adding
> those in existing microversion testing doc [0] and later will be
> refactoring to give them better and more visible shape.
>
> Tempest maintain the info about what microversion tests are is implemented
> on Tempest side which will be helpful for projects to check the coverage
> [1]. Cinder one was missed which I added in [2].
>
> ​As summary:
> Tempest Scope:
> - Only Integration tests for microversion is allowed in Tempest
>
If a microversion is important for interoperability it may end up in
Tempest even if doesn't involve too much integration


> - Exception for non integration tests to fill the schema gap if exists.
>
Yeah when we fill schema gaps we need at least one test for the top version
to verify that the latest implemented schema is valid


>
> Project Scope:
> - Remaining tests coverage of microversion​ should be on Projects side.
>   - Project can cover those as functional tests etc.
>   - Nova currently have at least one functional tests per each
> microversion and plan to extend coverage like [3]
>   - IMO, only running tests with 'latest' microversion is not enough,
> each microversion should be tested with positive and negative testing
> (w.r.t at least immediate previous microversion).
>
>
>
>
> In terms of which versions we test in the gate, for nova we always run
> with min_microversion = None and max_microversion = latest, which means
> that all tests will be executed.
> Since micro versions are incremental, and each micro version usually
> involves no or one test on Tempest side, I think it will be a while
> before this becomes an issue for the common gate.
>
>
> We test max_microversion=latest only on master. On the devstack stable
> branch we cap the max_microversion, e.g.:
>
>
> https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339
>
> --
>
> Thanks,
>
> Matt
>
>
>
>
> ..0 https://review.openstack.org/#/c/444727/1
> ..1
> https://docs.openstack.org/developer/tempest/microversion_testing.html#microversion-tests-implemented-in-tempest
>
> ..2 https://review.openstack.org/#/c/444711/ ​
> ​..3
> https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests
>  ​
>
> ​​Thanks
> gmann​
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Andrea Frittoli]

On Sat, Mar 11, 2017 at 10:45 PM Matt Riedemann  wrote:

On 3/10/2017 3:02 PM, Andrea Frittoli wrote:
>
> We had a couple of sessions related to this topic at the PTG [0][1].
>
> We agreed that we want to still maintain integration tests only in
> Tempest, which means that API micro versions that have no integration
> impact can be tested via functional tests.

To be clear, "integration" here means tests that span operations across
multiple services, correct? Like a compute test that first creates a
port in the networking service and a volume in the block storage service
and then uses those to create a server and maybe take a snapshot of it
which is then verified was uploaded to the image service.

Yes, indeed.



The non-integration things are self-contained in a single service, like
if all you need to do is create an aggregate, show it's details and
validate the response, at a particular microversion, we can just do that
in nova functional tests, and it's not necessary in Tempest.

It might be worth having a definition of this policy in the Tempest docs
so when people ask this question again you can just point at the docs.


I agree, we need to go a better job at documenting this.
I just want to avoid having a black and white rule that would not work.

To me tests that should be in Tempest are tests that make sense to run
against any change in any repo, and the reasons for that can be many:
- the test involves multiple services (more that "it uses a token though")
- the test covers a feature which is key for interoperability
- the test must be reliable and not take too long

Based on this I don't it's possible to completely avoid the discussion about
which test should in Tempest and which not, it's something to be considered
on a test by test basis, based on a set of guidelines.

We have an initial definition of scope in [0], but it's probably worth to
elaborate
more on it. I'll put up a patch so that the discussion on it can continue in
gerrit.

[0]
https://docs.openstack.org/developer/tempest/test-removal.html#tempest-scope


>
> In terms of which versions we test in the gate, for nova we always run
> with min_microversion = None and max_microversion = latest, which means
> that all tests will be executed.
> Since micro versions are incremental, and each micro version usually
> involves no or one test on Tempest side, I think it will be a while
> before this becomes an issue for the common gate.

We test max_microversion=latest only on master. On the devstack stable
branch we cap the max_microversion, e.g.:

Thanks, good point.



https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339

--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Ghanshyam Mann]

On Sun, Mar 12, 2017 at 7:40 AM, Matt Riedemann  wrote:

> On 3/10/2017 3:02 PM, Andrea Frittoli wrote:
>
>>
>> We had a couple of sessions related to this topic at the PTG [0][1].
>>
>> We agreed that we want to still maintain integration tests only in
>> Tempest, which means that API micro versions that have no integration
>> impact can be tested via functional tests.
>>
>
> To be clear, "integration" here means tests that span operations across
> multiple services, correct? Like a compute test that first creates a port
> in the networking service and a volume in the block storage service and
> then uses those to create a server and maybe take a snapshot of it which is
> then verified was uploaded to the image service.
>
> The non-integration things are self-contained in a single service, like if
> all you need to do is create an aggregate, show it's details and validate
> the response, at a particular microversion, we can just do that in nova
> functional tests, and it's not necessary in Tempest.
>
> It might be worth having a definition of this policy in the Tempest docs
> so when people ask this question again you can just point at the docs.
>
>
​Yes, that will be helpful to have those agreement in doc​. I am adding
those in existing microversion testing doc [0] and later will be
refactoring to give them better and more visible shape.

Tempest maintain the info about what microversion tests are is implemented
on Tempest side which will be helpful for projects to check the coverage
[1]. Cinder one was missed which I added in [2].

​As summary:
Tempest Scope:
- Only Integration tests for microversion is allowed in Tempest
- Exception for non integration tests to fill the schema gap if exists.

Project Scope:
- Remaining tests coverage of microversion​ should be on Projects side.
  - Project can cover those as functional tests etc.
  - Nova currently have at least one functional tests per each
microversion and plan to extend coverage like [3]
  - IMO, only running tests with 'latest' microversion is not enough,
each microversion should be tested with positive and negative testing
(w.r.t at least immediate previous microversion).



>
>> In terms of which versions we test in the gate, for nova we always run
>> with min_microversion = None and max_microversion = latest, which means
>> that all tests will be executed.
>> Since micro versions are incremental, and each micro version usually
>> involves no or one test on Tempest side, I think it will be a while
>> before this becomes an issue for the common gate.
>>
>
> We test max_microversion=latest only on master. On the devstack stable
> branch we cap the max_microversion, e.g.:
>
> https://github.com/openstack-dev/devstack/blob/stable/newton
> /lib/tempest#L339
>
> --
>
> Thanks,
>
> Matt
>
>


..0 https://review.openstack.org/#/c/444727/1
..1
https://docs.openstack.org/developer/tempest/microversion_testing.html#microversion-tests-implemented-in-tempest

..2 https://review.openstack.org/#/c/444711/ ​
​..3
https://blueprints.launchpad.net/nova/+spec/nova-microversion-functional-tests
 ​

​​Thanks
gmann​



> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Matt Riedemann]

On 3/10/2017 3:02 PM, Andrea Frittoli wrote:


We had a couple of sessions related to this topic at the PTG [0][1].

We agreed that we want to still maintain integration tests only in
Tempest, which means that API micro versions that have no integration
impact can be tested via functional tests.


To be clear, "integration" here means tests that span operations across 
multiple services, correct? Like a compute test that first creates a 
port in the networking service and a volume in the block storage service 
and then uses those to create a server and maybe take a snapshot of it 
which is then verified was uploaded to the image service.


The non-integration things are self-contained in a single service, like 
if all you need to do is create an aggregate, show it's details and 
validate the response, at a particular microversion, we can just do that 
in nova functional tests, and it's not necessary in Tempest.


It might be worth having a definition of this policy in the Tempest docs 
so when people ask this question again you can just point at the docs.




In terms of which versions we test in the gate, for nova we always run
with min_microversion = None and max_microversion = latest, which means
that all tests will be executed.
Since micro versions are incremental, and each micro version usually
involves no or one test on Tempest side, I think it will be a while
before this becomes an issue for the common gate.


We test max_microversion=latest only on master. On the devstack stable 
branch we cap the max_microversion, e.g.:


https://github.com/openstack-dev/devstack/blob/stable/newton/lib/tempest#L339

--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Ken'ichi Ohmichi]

2017-03-10 13:32 GMT-08:00 Matthew Treinish :
> On Fri, Mar 10, 2017 at 12:34:31PM -0800, Ken'ichi Ohmichi wrote:
>> Hi John,
>>
>> Now Tempest is testing microversions only for Nova and contains some
>> testing framework for re-using for another projects.
>> On this framework, we can implement necessary microversions tests as
>> we want and actually many microversions of Nova are not tested by
>> Tempest.
>> We can see the tested microversion of Nova on
>> https://github.com/openstack/tempest/blob/master/doc/source/microversion_testing.rst#microversion-tests-implemented-in-tempest
>>
>> Before implementing microversion testing for Cinder, we will implement
>> JSON-Schema validation for API responses for Cinder.
>> The validation will be helpful for testing base microversion of Cinder
>> API and we will be able to implement the microversion tests based on
>> that.
>> This implementation is marked as 7th priority in this Pike cycle as
>> https://etherpad.openstack.org/p/pike-qa-priorities
>>
>> In addition, now Cinder V3 API is not tested. So we are going to
>> enable v3 tests with some restructure of Tempest in this cycle.
>> The detail is described on the part of "Volume API" of
>> https://etherpad.openstack.org/p/tempest-api-versions-in-pike
>
>
> Umm, I don't know what you're basing that on, but there have been cinder v3
> tests and cinder microversion support in Tempest since Newton. It was 
> initially
> added in this patch: https://review.openstack.org/#/c/300639/

Yeah, that is for v3 but that is only I think at this time.

>> 2017-03-10 11:37 GMT-08:00 John Griffith :
>> > Hey Everyone,
>> >
>> > So along the lines of an earlier thread that went out regarding testing of
>> > deprecated API's and Tempest etc [1].
>> >
>> > Now that micro-versions are *the API versioning scheme to rule them all* 
>> > one
>> > question I've not been able to find an answer for is what we're going to
>> > promise here for support and testing.  My understanding thus far is that 
>> > the
>> > "community" approach here is "nothing is ever deprecated, and everything is
>> > supported forever".
>> >
>> > That's sort of a tall order IMO, but ok.  I've already had some questions
>> > from folks about implementing an explicit Tempest test for every
>> > micro-versioned implementation of an API call also.  My response has been
>> > "nahh, just always test latest available".  This kinda means that we're not
>> > testing/supporting the previous versions as promised though.
>> >
>> > Anyway; I'm certain that between Nova and the API-WG this has come up and 
>> > is
>> > probably addressed, just wondering if somebody can point me to some
>> > documentation or policies in this respect.
>> >
>> > Thanks,
>> > John
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Matthew Treinish]

On Fri, Mar 10, 2017 at 12:34:31PM -0800, Ken'ichi Ohmichi wrote:
> Hi John,
> 
> Now Tempest is testing microversions only for Nova and contains some
> testing framework for re-using for another projects.
> On this framework, we can implement necessary microversions tests as
> we want and actually many microversions of Nova are not tested by
> Tempest.
> We can see the tested microversion of Nova on
> https://github.com/openstack/tempest/blob/master/doc/source/microversion_testing.rst#microversion-tests-implemented-in-tempest
> 
> Before implementing microversion testing for Cinder, we will implement
> JSON-Schema validation for API responses for Cinder.
> The validation will be helpful for testing base microversion of Cinder
> API and we will be able to implement the microversion tests based on
> that.
> This implementation is marked as 7th priority in this Pike cycle as
> https://etherpad.openstack.org/p/pike-qa-priorities
> 
> In addition, now Cinder V3 API is not tested. So we are going to
> enable v3 tests with some restructure of Tempest in this cycle.
> The detail is described on the part of "Volume API" of
> https://etherpad.openstack.org/p/tempest-api-versions-in-pike


Umm, I don't know what you're basing that on, but there have been cinder v3
tests and cinder microversion support in Tempest since Newton. It was initially
added in this patch: https://review.openstack.org/#/c/300639/

-Matt Treinish


> 
> 2017-03-10 11:37 GMT-08:00 John Griffith :
> > Hey Everyone,
> >
> > So along the lines of an earlier thread that went out regarding testing of
> > deprecated API's and Tempest etc [1].
> >
> > Now that micro-versions are *the API versioning scheme to rule them all* one
> > question I've not been able to find an answer for is what we're going to
> > promise here for support and testing.  My understanding thus far is that the
> > "community" approach here is "nothing is ever deprecated, and everything is
> > supported forever".
> >
> > That's sort of a tall order IMO, but ok.  I've already had some questions
> > from folks about implementing an explicit Tempest test for every
> > micro-versioned implementation of an API call also.  My response has been
> > "nahh, just always test latest available".  This kinda means that we're not
> > testing/supporting the previous versions as promised though.
> >
> > Anyway; I'm certain that between Nova and the API-WG this has come up and is
> > probably addressed, just wondering if somebody can point me to some
> > documentation or policies in this respect.
> >
> > Thanks,
> > John


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Andrea Frittoli]

On Fri, Mar 10, 2017 at 8:39 PM Ken'ichi Ohmichi 
wrote:

> Hi John,
>
> Now Tempest is testing microversions only for Nova and contains some
> testing framework for re-using for another projects.
> On this framework, we can implement necessary microversions tests as
> we want and actually many microversions of Nova are not tested by
> Tempest.
> We can see the tested microversion of Nova on
>
> https://github.com/openstack/tempest/blob/master/doc/source/microversion_testing.rst#microversion-tests-implemented-in-tempest
>
> Before implementing microversion testing for Cinder, we will implement
> JSON-Schema validation for API responses for Cinder.
> The validation will be helpful for testing base microversion of Cinder
> API and we will be able to implement the microversion tests based on
> that.
> This implementation is marked as 7th priority in this Pike cycle as
> https://etherpad.openstack.org/p/pike-qa-priorities
>
> In addition, now Cinder V3 API is not tested. So we are going to
> enable v3 tests with some restructure of Tempest in this cycle.
> The detail is described on the part of "Volume API" of
> https://etherpad.openstack.org/p/tempest-api-versions-in-pike
>
> Thanks
> Ken Ohmichi
>
> ---
>
> 2017-03-10 11:37 GMT-08:00 John Griffith :
> > Hey Everyone,
> >
> > So along the lines of an earlier thread that went out regarding testing
> of
> > deprecated API's and Tempest etc [1].
> >
> > Now that micro-versions are *the API versioning scheme to rule them all*
> one
> > question I've not been able to find an answer for is what we're going to
> > promise here for support and testing.  My understanding thus far is that
> the
> > "community" approach here is "nothing is ever deprecated, and everything
> is
> > supported forever".
> >
> > That's sort of a tall order IMO, but ok.  I've already had some questions
> > from folks about implementing an explicit Tempest test for every
> > micro-versioned implementation of an API call also.  My response has been
> > "nahh, just always test latest available".  This kinda means that we're
> not
> > testing/supporting the previous versions as promised though.
>

We had a couple of sessions related to this topic at the PTG [0][1].

We agreed that we want to still maintain integration tests only in Tempest,
which means that API micro versions that have no integration impact can be
tested via functional tests.

Since we do schema validation, when someone wants to develop an integration
tests for a specific micro version, there may be a gap in the schemas
implemented on Tempest side and the current one - we had this case already
in nova.
We decided we shall monitor this gap, and fill it with schema definition at
least at the end of each cycle, or more often if required.

Tests can define which range or micro versions they are applicable to.
Initial tests for v3 in cinder will have no min_version, and they may get a
max version if they're behaviour is not valid anymore beyond a certain
microversion. Tests developed specifically for a micro versions will have
min_version set accordingly.

In terms of which versions we test in the gate, for nova we always run with
min_microversion = None and max_microversion = latest, which means that all
tests will be executed.
Since micro versions are incremental, and each micro version usually
involves no or one test on Tempest side, I think it will be a while before
this becomes an issue for the common gate.

andrea

[0] https://etherpad.openstack.org/p/qa-ptg-pike-microversion
[1] https://etherpad.openstack.org/p/qa-ptg-pike-schema-validation
[2] https://docs.openstack.org/developer/tempest/microversion_testing.html

> >
> > Anyway; I'm certain that between Nova and the API-WG this has come up
> and is
> > probably addressed, just wondering if somebody can point me to some
> > documentation or policies in this respect.
> >
> > Thanks,
> > John
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[John Griffith]

On Fri, Mar 10, 2017 at 1:34 PM, Ken'ichi Ohmichi 
wrote:

> Hi John,
>
> Now Tempest is testing microversions only for Nova and contains some
> testing framework for re-using for another projects.
> On this framework, we can implement necessary microversions tests as
> we want and actually many microversions of Nova are not tested by
> Tempest.
> We can see the tested microversion of Nova on
> https://github.com/openstack/tempest/blob/master/doc/
> source/microversion_testing.rst#microversion-tests-implemented-in-tempest
>
> Before implementing microversion testing for Cinder, we will implement
> JSON-Schema validation for API responses for Cinder.
> The validation will be helpful for testing base microversion of Cinder
> API and we will be able to implement the microversion tests based on
> that.
> This implementation is marked as 7th priority in this Pike cycle as
> https://etherpad.openstack.org/p/pike-qa-priorities
>
> In addition, now Cinder V3 API is not tested. So we are going to
> enable v3 tests with some restructure of Tempest in this cycle.
> The detail is described on the part of "Volume API" of
> https://etherpad.openstack.org/p/tempest-api-versions-in-pike
>
> Thanks
> Ken Ohmichi
>
> ---
>
> 2017-03-10 11:37 GMT-08:00 John Griffith :
> > Hey Everyone,
> >
> > So along the lines of an earlier thread that went out regarding testing
> of
> > deprecated API's and Tempest etc [1].
> >
> > Now that micro-versions are *the API versioning scheme to rule them all*
> one
> > question I've not been able to find an answer for is what we're going to
> > promise here for support and testing.  My understanding thus far is that
> the
> > "community" approach here is "nothing is ever deprecated, and everything
> is
> > supported forever".
> >
> > That's sort of a tall order IMO, but ok.  I've already had some questions
> > from folks about implementing an explicit Tempest test for every
> > micro-versioned implementation of an API call also.  My response has been
> > "nahh, just always test latest available".  This kinda means that we're
> not
> > testing/supporting the previous versions as promised though.
> >
> > Anyway; I'm certain that between Nova and the API-WG this has come up
> and is
> > probably addressed, just wondering if somebody can point me to some
> > documentation or policies in this respect.
> >
> > Thanks,
> > John
> >
> > 
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
> unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Thanks for the pointer to the doc Ken, that helps a lot.  I do have
concerns about supportability, test sprawl and life-cycle with this, but
maybe it's unwarranted.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[Ken'ichi Ohmichi]

Hi John,

Now Tempest is testing microversions only for Nova and contains some
testing framework for re-using for another projects.
On this framework, we can implement necessary microversions tests as
we want and actually many microversions of Nova are not tested by
Tempest.
We can see the tested microversion of Nova on
https://github.com/openstack/tempest/blob/master/doc/source/microversion_testing.rst#microversion-tests-implemented-in-tempest

Before implementing microversion testing for Cinder, we will implement
JSON-Schema validation for API responses for Cinder.
The validation will be helpful for testing base microversion of Cinder
API and we will be able to implement the microversion tests based on
that.
This implementation is marked as 7th priority in this Pike cycle as
https://etherpad.openstack.org/p/pike-qa-priorities

In addition, now Cinder V3 API is not tested. So we are going to
enable v3 tests with some restructure of Tempest in this cycle.
The detail is described on the part of "Volume API" of
https://etherpad.openstack.org/p/tempest-api-versions-in-pike

Thanks
Ken Ohmichi

---

2017-03-10 11:37 GMT-08:00 John Griffith :
> Hey Everyone,
>
> So along the lines of an earlier thread that went out regarding testing of
> deprecated API's and Tempest etc [1].
>
> Now that micro-versions are *the API versioning scheme to rule them all* one
> question I've not been able to find an answer for is what we're going to
> promise here for support and testing.  My understanding thus far is that the
> "community" approach here is "nothing is ever deprecated, and everything is
> supported forever".
>
> That's sort of a tall order IMO, but ok.  I've already had some questions
> from folks about implementing an explicit Tempest test for every
> micro-versioned implementation of an API call also.  My response has been
> "nahh, just always test latest available".  This kinda means that we're not
> testing/supporting the previous versions as promised though.
>
> Anyway; I'm certain that between Nova and the API-WG this has come up and is
> probably addressed, just wondering if somebody can point me to some
> documentation or policies in this respect.
>
> Thanks,
> John
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[John Griffith]

On Fri, Mar 10, 2017 at 12:37 PM, John Griffith 
wrote:

> Hey Everyone,
>
> So along the lines of an earlier thread that went out regarding testing of
> deprecated API's and Tempest etc [1].
>
> Now that micro-versions are *the API versioning scheme to rule them all*
> one question I've not been able to find an answer for is what we're going
> to promise here for support and testing.  My understanding thus far is that
> the "community" approach here is "nothing is ever deprecated, and
> everything is supported forever".
>
> That's sort of a tall order IMO, but ok.  I've already had some questions
> from folks about implementing an explicit Tempest test for every
> micro-versioned implementation of an API call also.  My response has been
> "nahh, just always test latest available".  This kinda means that we're not
> testing/supporting the previous versions as promised though.
>
> Anyway; I'm certain that between Nova and the API-WG this has come up and
> is probably addressed, just wondering if somebody can point me to some
> documentation or policies in this respect.
>
> Thanks,
> John
>
​Ooops:
  [1]:
http://lists.openstack.org/pipermail/openstack-dev/2017-March/113727.html​
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [api][qa][tc][nova][cinder] Testing of a microversioned world

[John Griffith]

Hey Everyone,

So along the lines of an earlier thread that went out regarding testing of
deprecated API's and Tempest etc [1].

Now that micro-versions are *the API versioning scheme to rule them all*
one question I've not been able to find an answer for is what we're going
to promise here for support and testing.  My understanding thus far is that
the "community" approach here is "nothing is ever deprecated, and
everything is supported forever".

That's sort of a tall order IMO, but ok.  I've already had some questions
from folks about implementing an explicit Tempest test for every
micro-versioned implementation of an API call also.  My response has been
"nahh, just always test latest available".  This kinda means that we're not
testing/supporting the previous versions as promised though.

Anyway; I'm certain that between Nova and the API-WG this has come up and
is probably addressed, just wondering if somebody can point me to some
documentation or policies in this respect.

Thanks,
John
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev