Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Matthew Treinish]

On Wed, May 03, 2017 at 11:09:32PM -0400, Monty Taylor wrote:
> On 05/02/2017 11:49 AM, Sean McGinnis wrote:
> > On Tue, May 02, 2017 at 03:36:20PM +0200, Jordan Pittier wrote:
> > > On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann 
> > > wrote:
> > > 
> > > > In Cinder, there are many features/APIs which are backend specific and
> > > > will return 405 or 501 if same is not implemented on any backend [1].
> > > > If such tests are implemented in Tempest, then it will break some gate
> > > > where that backend job is voting. like ceph job in glance_store gate.
> > > > 
> > > > There been many such cases recently where ceph jobs were broken due to
> > > > such tests and recently it is for force-delete backup feature[2].
> > > > Reverting force-delete tests in [3]. To resolve such cases at some
> > > > extend, Jon is going to add a white/black list of tests which can run
> > > > on ceph job [4] depends on what all feature ceph implemented. But this
> > > > does not resolve it completely due to many reason like
> > > > 1. External use of Tempest become difficult where user needs to know
> > > > what all tests to skip for which backend
> > > > 2. Tempest tests become too specific to backend.
> > > > 
> > > > Now there are few options to resolve this:
> > > > 1. Tempest should not tests such API/feature which are backend
> > > > specific like mentioned by api-ref like[1].
> > > > 
> > > So basically, if one of the 50 Cinder driver doesn't support a feature, we
> > > should never test that feature ? What about the 49 other drivers ? If a
> > > feature exists and can be tested in the Gate (with whatever default
> > > config/driver is shipped) then I think we should test it.
> > > 
> > 50? Over 100 as of Ocata.
> > 
> > Well, is tempest's purpose in life to provide complete gate test coverage,
> > or is tempest's purpose in life to give operators a tool to validate that
> > their deployment is working as expected?
> 
> I'd actually like to suggest that such a scenario actually points out a
> thing that is ultimately potential pain passed to the end user in the real
> world, so this question about what/how to test this in tempest is a good one
> to have.
> 
> If there is a feature which is only provisionally available depending on the
> backend driver such that it's hard to test in tempest without an out of band
> config - then it's a feature that a user will have no clue whether it works
> on a given cloud.
> 
> As we find these, I'd love it if we could expose discovery in the API for
> viability of the feature. Like:
> 
> GET /capabilities
> 
> {
>   "capabilities": {
> "has_force_delete": true
>   }
> }
> 
> (I know we've talked about that concept generally, but this is a specific
> example)
> 
> If such a thing existed, then the user can know whether they can use a thing
> .. and so can tempest. A tempest test to validate force_delete working could
> check the capability reported by the API and validate that if the API says
> "true" that the feature work as expected, and if it says "false" validate
> that attempting to call it returns a 405 (or whatever is appropriate)
> 
> Ultimately, every config we need to feed to tempest is potentially a place
> where an end user is unable to know whether or not to expect a call to work
> - and an opportunity for us to provide our API consumers with a richer
> experience.

Heh, well I've been saying all of these things for years. In fact I got so tired
of repeating it all the time I just put it in the tempest configuration guide:
(although I remember it being a lot snarkier)

https://docs.openstack.org/developer/tempest/configuration.html#service-feature-configuration

So, I'll be very happy when the capabilities work actually becomes a thing you
can use. But, it feels like we've been talking about around the problem for a
very long time...

-Matt Treinish


> 
> > In attempting to do things in the past, I've received push back based on
> > the argument that it was the latter. For this reason, in-tree tempest tests
> > were added to Cinder to give us a way to get better test coverage for our
> > own sake.
> > 
> > Now that this is all in place, I think it's working well and I would like
> > to see it continue that way. IMO, tempest proper should not have anything
> > that isn't universally applicable to real world deployments. Not just for
> > things like Ceph, but things like the manage/unmanage backend specific
> > tests that were added and broke a large majority of third party CI.
> > 
> > Backend specific things should not be part of tempest in my opinion. We
> > should cover those things through in-tree tempest plugins and our own
> > testing.
> > > 
> > > > 2. Tempest test can be disabled/skip based on backend. - This is not
> > > > good idea as it increase config options and overhead of setting those.
> > > > 
> > > Using regex and blacklist, any 3rd party CI can skip any test based on the
> > > test ID. Without introducing a config flag. 

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Fei Long Wang]

On 04/05/17 15:01, Ghanshyam Mann wrote:
>
> On Wed, May 3, 2017 at 21:57 Andrea Frittoli
> > wrote:
>
> On Tue, May 2, 2017 at 2:41 PM Jordan Pittier
> >
> wrote:
>
> On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann
> > wrote:
>
> In Cinder, there are many features/APIs which are backend
> specific and
> will return 405 or 501 if same is not implemented on any
> backend [1].
> If such tests are implemented in Tempest, then it will
> break some gate
> where that backend job is voting. like ceph job in
> glance_store gate.
>
> There been many such cases recently where ceph jobs were
> broken due to
> such tests and recently it is for force-delete backup
> feature[2].
> Reverting force-delete tests in [3]. To resolve such cases
> at some
> extend, Jon is going to add a white/black list of tests
> which can run
> on ceph job [4] depends on what all feature ceph
> implemented. But this
> does not resolve it completely due to many reason like
> 1. External use of Tempest become difficult where user
> needs to know
> what all tests to skip for which backend
> 2. Tempest tests become too specific to backend.
>
> Now there are few options to resolve this:
> 1. Tempest should not tests such API/feature which are backend
> specific like mentioned by api-ref like[1].
>
> So basically, if one of the 50 Cinder driver doesn't support a
> feature, we should never test that feature ? What about the 49
> other drivers ? If a feature exists and can be tested in the
> Gate (with whatever default config/driver is shipped) then I
> think we should test it.
>  
>
> 2. Tempest test can be disabled/skip based on backend. -
> This is not
> good idea as it increase config options and overhead of
> setting those.
>
> Using regex and blacklist, any 3rd party CI can skip any test
> based on the test ID. Without introducing a config flag.
> See: 
> https://github.com/openstack-infra/project-config/blob/1cea31f402b6b047cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871
>
>
> This way each 3rd party system has to maintain its own list, which
> has the advantage that
> different teams maintain their own list (which is nice from an
> ownership and scale pov).
>
> However I think such list of tests are not as re-usable as having
> a devstack plugin (or an
> ansible or puppet module) changing a few tempest config options. 
>
>
> Humm, I am little bit hesitate to go that way. For gate and CI it
> might be good solution but for production cloud testing it makes
> Tenpest difficult to use.
>
> If I use Tempest to test my cloud, few tests going to fail as those
> were not supported by cinder driver my cloud has or going to have. 
> I do not have any way to configure something so that test can be
> disabled. Instead I need to maintain list of tests to run or skip. And
> that list is not static, it grows dynamically. 
> This makes Tempest difficult to use. 
>
>

Agree. We (Catalyst Cloud based in NZ) are using Tempest as the
monitoring tool and CI/CD gate for our cloud. But we do have to maintain
a white list of test cases because there are some cases are not fitting
our cloud.

>
>
>  
>
> 3. Tempest test can verify behavior with if else condition
> as per
> backend. This is bad idea and lose the test strength.
>
> Yeah, that's bad. 
>
>
> IMO options 1 is better options. More feedback are welcome. 
>
>
> ..1
> 
> https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
> ..2 https://bugs.launchpad.net/glance/+bug/1687538
> ..3 https://review.openstack.org/#/c/461625/
> ..4
> 
> http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html
>
> -gmann
>
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> 
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> 
> 

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Ghanshyam Mann]

On Wed, May 3, 2017 at 22:35 Sean McGinnis  wrote:

> On Wed, May 03, 2017 at 01:14:18PM +, Andrea Frittoli wrote:
> >
> > > >
> >
> > > Now that this is all in place, I think it's working well and I would
> like
> > > to see it continue that way. IMO, tempest proper should not have
> anything
> > > that isn't universally applicable to real world deployments. Not just
> for
> > > things like Ceph, but things like the manage/unmanage backend specific
> > > tests that were added and broke a large majority of third party CI.
> > >
> >
> > Is there a policy in Cinder that a backend must implement a certain set
> of
> > APIs? If so we could think of testing only that set of APIs in Tempest,
> so
> > that
> > any app developer knows that he/she can rely on that minimum set of APIs.
> >


+1.


>
> Yes, there is a minimum feature set that all backend drivers must support.
> That base functionality can be found here [0] and here [1].
>
> [0]
> https://docs.openstack.org/developer/cinder/devref/drivers.html#core-functionality
> [1]
> https://github.com/openstack/cinder/blob/master/cinder/interface/volume_driver.py
>
> The issue we've had with some of the tempest tests isn't actually around
> whether the given backend supports the API. It's the way the API is used
> that becomes a challenge.


This is good way to solve the most part of issue. Tempest can test the
cinder MUST list Features. This provides the consistent way of testing the
basic and consistent features.



>
> I'll use the manage/unmanage snapshot one as an example. This is not one
> of the required APIs, but many backends do support it. But the way this
> API works is you are basically saying "manage this snapshot that you
> identify as X", where "X" can be different things depending on the volume
> driver being used. It's the storage device's native way of identifying
> a snapshot, which may or may not be our Cinder snapshot ID.
>
> So that test was added based on the way the LVM driver works for this.
> That part is great, we get some more code coverage in the gate. But then
> each volume driver that uses a different ID had to first a) start failing
> CI, b) troubleshoot what happened to cause this new failure, c) reconfig
> their CI to skip this test. Repeat cycle for each test added that does
> something specific to how one or a small subset of backends work.


Yea, we should test the consistent part of behavior. I agree snapshot
manage test needs to be modified to work for all backends. We welcome if
you or someone from cinder team can suggest the generic way to identify the
managed snapshot.

For all cinder MUST features in all backend we should avoid the
storage/backend specific logic in tests.



>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
-gmann
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Monty Taylor]

On 05/02/2017 11:49 AM, Sean McGinnis wrote:

On Tue, May 02, 2017 at 03:36:20PM +0200, Jordan Pittier wrote:

On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann 
wrote:


In Cinder, there are many features/APIs which are backend specific and
will return 405 or 501 if same is not implemented on any backend [1].
If such tests are implemented in Tempest, then it will break some gate
where that backend job is voting. like ceph job in glance_store gate.

There been many such cases recently where ceph jobs were broken due to
such tests and recently it is for force-delete backup feature[2].
Reverting force-delete tests in [3]. To resolve such cases at some
extend, Jon is going to add a white/black list of tests which can run
on ceph job [4] depends on what all feature ceph implemented. But this
does not resolve it completely due to many reason like
1. External use of Tempest become difficult where user needs to know
what all tests to skip for which backend
2. Tempest tests become too specific to backend.

Now there are few options to resolve this:
1. Tempest should not tests such API/feature which are backend
specific like mentioned by api-ref like[1].


So basically, if one of the 50 Cinder driver doesn't support a feature, we
should never test that feature ? What about the 49 other drivers ? If a
feature exists and can be tested in the Gate (with whatever default
config/driver is shipped) then I think we should test it.


50? Over 100 as of Ocata.

Well, is tempest's purpose in life to provide complete gate test coverage,
or is tempest's purpose in life to give operators a tool to validate that
their deployment is working as expected?


I'd actually like to suggest that such a scenario actually points out a 
thing that is ultimately potential pain passed to the end user in the 
real world, so this question about what/how to test this in tempest is a 
good one to have.


If there is a feature which is only provisionally available depending on 
the backend driver such that it's hard to test in tempest without an out 
of band config - then it's a feature that a user will have no clue 
whether it works on a given cloud.


As we find these, I'd love it if we could expose discovery in the API 
for viability of the feature. Like:


GET /capabilities

{
  "capabilities": {
"has_force_delete": true
  }
}

(I know we've talked about that concept generally, but this is a 
specific example)


If such a thing existed, then the user can know whether they can use a 
thing .. and so can tempest. A tempest test to validate force_delete 
working could check the capability reported by the API and validate that 
if the API says "true" that the feature work as expected, and if it says 
"false" validate that attempting to call it returns a 405 (or whatever 
is appropriate)


Ultimately, every config we need to feed to tempest is potentially a 
place where an end user is unable to know whether or not to expect a 
call to work - and an opportunity for us to provide our API consumers 
with a richer experience.



In attempting to do things in the past, I've received push back based on
the argument that it was the latter. For this reason, in-tree tempest tests
were added to Cinder to give us a way to get better test coverage for our
own sake.

Now that this is all in place, I think it's working well and I would like
to see it continue that way. IMO, tempest proper should not have anything
that isn't universally applicable to real world deployments. Not just for
things like Ceph, but things like the manage/unmanage backend specific
tests that were added and broke a large majority of third party CI.

Backend specific things should not be part of tempest in my opinion. We
should cover those things through in-tree tempest plugins and our own
testing.



2. Tempest test can be disabled/skip based on backend. - This is not
good idea as it increase config options and overhead of setting those.


Using regex and blacklist, any 3rd party CI can skip any test based on the
test ID. Without introducing a config flag. See:
https://github.com/openstack-infra/project-config/blob/1cea31f402b6b047cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871



3. Tempest test can verify behavior with if else condition as per
backend. This is bad idea and lose the test strength.


Yeah, that's bad.



IMO options 1 is better options. More feedback are welcome.




..1 https://developer.openstack.org/api-ref/block-storage/v3/?
expanded=force-delete-a-backup-detail#force-delete-a-backup
..2 https://bugs.launchpad.net/glance/+bug/1687538
..3 https://review.openstack.org/#/c/461625/
..4 http://lists.openstack.org/pipermail/openstack-dev/2017-
April/115229.html

-gmann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Ghanshyam Mann]

On Wed, May 3, 2017 at 21:57 Andrea Frittoli 
wrote:

> On Tue, May 2, 2017 at 2:41 PM Jordan Pittier 
> wrote:
>
>> On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann 
>> wrote:
>>
>>> In Cinder, there are many features/APIs which are backend specific and
>>> will return 405 or 501 if same is not implemented on any backend [1].
>>> If such tests are implemented in Tempest, then it will break some gate
>>> where that backend job is voting. like ceph job in glance_store gate.
>>>
>>> There been many such cases recently where ceph jobs were broken due to
>>> such tests and recently it is for force-delete backup feature[2].
>>> Reverting force-delete tests in [3]. To resolve such cases at some
>>> extend, Jon is going to add a white/black list of tests which can run
>>> on ceph job [4] depends on what all feature ceph implemented. But this
>>> does not resolve it completely due to many reason like
>>> 1. External use of Tempest become difficult where user needs to know
>>> what all tests to skip for which backend
>>> 2. Tempest tests become too specific to backend.
>>>
>>> Now there are few options to resolve this:
>>> 1. Tempest should not tests such API/feature which are backend
>>> specific like mentioned by api-ref like[1].
>>>
>> So basically, if one of the 50 Cinder driver doesn't support a feature,
>> we should never test that feature ? What about the 49 other drivers ? If a
>> feature exists and can be tested in the Gate (with whatever default
>> config/driver is shipped) then I think we should test it.
>>
>>
>>> 2. Tempest test can be disabled/skip based on backend. - This is not
>>> good idea as it increase config options and overhead of setting those.
>>>
>> Using regex and blacklist, any 3rd party CI can skip any test based on
>> the test ID. Without introducing a config flag. See:
>> https://github.com/openstack-infra/project-config/blob/1cea31f402b6b047cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871
>>
>
> This way each 3rd party system has to maintain its own list, which has the
> advantage that
> different teams maintain their own list (which is nice from an ownership
> and scale pov).
>
> However I think such list of tests are not as re-usable as having a
> devstack plugin (or an
> ansible or puppet module) changing a few tempest config options.
>

Humm, I am little bit hesitate to go that way. For gate and CI it might be
good solution but for production cloud testing it makes Tenpest difficult
to use.

If I use Tempest to test my cloud, few tests going to fail as those were
not supported by cinder driver my cloud has or going to have.
I do not have any way to configure something so that test can be disabled.
Instead I need to maintain list of tests to run or skip. And that list is
not static, it grows dynamically.
This makes Tempest difficult to use.




>
>>
>>> 3. Tempest test can verify behavior with if else condition as per
>>> backend. This is bad idea and lose the test strength.
>>>
>> Yeah, that's bad.
>>
>>>
>>> IMO options 1 is better options. More feedback are welcome.
>>
>>
>>> ..1
>>> https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
>>> ..2 https://bugs.launchpad.net/glance/+bug/1687538
>>> ..3 https://review.openstack.org/#/c/461625/
>>> ..4
>>> http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html
>>>
>>> -gmann
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
-gmann
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Ghanshyam Mann]

On Wed, May 3, 2017 at 21:49 Andrea Frittoli 
wrote:

> On Tue, May 2, 2017 at 6:46 AM Ghanshyam Mann 
> wrote:
>
>> In Cinder, there are many features/APIs which are backend specific and
>> will return 405 or 501 if same is not implemented on any backend [1].
>> If such tests are implemented in Tempest, then it will break some gate
>> where that backend job is voting. like ceph job in glance_store gate.
>>
>
> Having a test in Tempest is important for interoperability and API
> backward compatibility.
> As long as available features are discoverable and reported in a
> consistent way, it
> is possible for app developer to write application that will work fine
> against
> different backends.
>

But in this case, features are not discoverable. From API status code only
we can get to know whether it is implemented in particular backend or not.
It has same issue for interoperability as Tempest facing now.


>
>>
>> There been many such cases recently where ceph jobs were broken due to
>> such tests and recently it is for force-delete backup feature[2].
>> Reverting force-delete tests in [3]. To resolve such cases at some
>> extend, Jon is going to add a white/black list of tests which can run
>> on ceph job [4] depends on what all feature ceph implemented. But this
>> does not resolve it completely due to many reason like
>> 1. External use of Tempest become difficult where user needs to know
>> what all tests to skip for which backend
>> 2. Tempest tests become too specific to backend.
>>
>> Now there are few options to resolve this:
>> 1. Tempest should not tests such API/feature which are backend
>> specific like mentioned by api-ref like[1].
>> 2. Tempest test can be disabled/skip based on backend. - This is not
>> good idea as it increase config options and overhead of setting those.
>>
>
> Tempest has many options because we decide not to rely on discovery, i.e.
> we configure what we expect to find in the target cloud. I don't think we
> can use
> this as a factor to influence the decision in this case.
>

>
>> 3. Tempest test can verify behavior with if else condition as per
>> backend. This is bad idea and lose the test strength.
>>
>> IMO options 1 is better options. More feedback are welcome.
>>
>> ..1
>> https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
>> ..2 https://bugs.launchpad.net/glance/+bug/1687538
>> ..3 https://review.openstack.org/#/c/461625/
>> ..4
>> http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html
>>
>> -gmann
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
-gmann
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Sean McGinnis]

On Wed, May 03, 2017 at 01:14:18PM +, Andrea Frittoli wrote:
> 
> > >
> 
> > Now that this is all in place, I think it's working well and I would like
> > to see it continue that way. IMO, tempest proper should not have anything
> > that isn't universally applicable to real world deployments. Not just for
> > things like Ceph, but things like the manage/unmanage backend specific
> > tests that were added and broke a large majority of third party CI.
> >
> 
> Is there a policy in Cinder that a backend must implement a certain set of
> APIs? If so we could think of testing only that set of APIs in Tempest, so
> that
> any app developer knows that he/she can rely on that minimum set of APIs.
> 

Yes, there is a minimum feature set that all backend drivers must support.
That base functionality can be found here [0] and here [1].

[0] 
https://docs.openstack.org/developer/cinder/devref/drivers.html#core-functionality
[1] 
https://github.com/openstack/cinder/blob/master/cinder/interface/volume_driver.py

The issue we've had with some of the tempest tests isn't actually around
whether the given backend supports the API. It's the way the API is used
that becomes a challenge.

I'll use the manage/unmanage snapshot one as an example. This is not one
of the required APIs, but many backends do support it. But the way this
API works is you are basically saying "manage this snapshot that you
identify as X", where "X" can be different things depending on the volume
driver being used. It's the storage device's native way of identifying
a snapshot, which may or may not be our Cinder snapshot ID.

So that test was added based on the way the LVM driver works for this.
That part is great, we get some more code coverage in the gate. But then
each volume driver that uses a different ID had to first a) start failing
CI, b) troubleshoot what happened to cause this new failure, c) reconfig
their CI to skip this test. Repeat cycle for each test added that does
something specific to how one or a small subset of backends work.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Andrea Frittoli]

On Tue, May 2, 2017 at 4:56 PM Sean McGinnis  wrote:

> On Tue, May 02, 2017 at 03:36:20PM +0200, Jordan Pittier wrote:
> > On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann 
> > wrote:
> >
> > > In Cinder, there are many features/APIs which are backend specific and
> > > will return 405 or 501 if same is not implemented on any backend [1].
> > > If such tests are implemented in Tempest, then it will break some gate
> > > where that backend job is voting. like ceph job in glance_store gate.
> > >
> > > There been many such cases recently where ceph jobs were broken due to
> > > such tests and recently it is for force-delete backup feature[2].
> > > Reverting force-delete tests in [3]. To resolve such cases at some
> > > extend, Jon is going to add a white/black list of tests which can run
> > > on ceph job [4] depends on what all feature ceph implemented. But this
> > > does not resolve it completely due to many reason like
> > > 1. External use of Tempest become difficult where user needs to know
> > > what all tests to skip for which backend
> > > 2. Tempest tests become too specific to backend.
> > >
> > > Now there are few options to resolve this:
> > > 1. Tempest should not tests such API/feature which are backend
> > > specific like mentioned by api-ref like[1].
> > >
> > So basically, if one of the 50 Cinder driver doesn't support a feature,
> we
> > should never test that feature ? What about the 49 other drivers ? If a
> > feature exists and can be tested in the Gate (with whatever default
> > config/driver is shipped) then I think we should test it.
> >
> 50? Over 100 as of Ocata.
>
> Well, is tempest's purpose in life to provide complete gate test coverage,
> or is tempest's purpose in life to give operators a tool to validate that
> their deployment is working as expected?
>

Neither.

Tempest is used for several different purposes, but I would say it was never
meant to ensure 100% coverage of the API. It is used by many operators to
validate their deployments, even if that part is better achieved via the
"scenario" tests, as opposed to the "API" tests.

Main use cases for Tempest are:
- integration (cross-service) testing in the gate
- help to ensure API backward compatibility / stability
- home for interoperability tests


> In attempting to do things in the past, I've received push back based on
> the argument that it was the latter. For this reason, in-tree tempest tests
> were added to Cinder to give us a way to get better test coverage for our
> own sake.
>

There are several use cases for in-tree tests.
Some examples:

Test that provide very little cross-service validation (e.g. most API
negative
tests) do not need to be in Tempest. Running a full cloud is expensive
resource and time-wise, and it's not the best test environment to run a
large
combination of negative test cases.

Many tests cannot be driven via API. It's very hard for instance to test
any transient resource state via API, since there's not enough control via
the API.

Scheduler tests are not best implemented via API as well, since they often
require
several nodes and resources when executed in a full cloud environment.


> Now that this is all in place, I think it's working well and I would like
> to see it continue that way. IMO, tempest proper should not have anything
> that isn't universally applicable to real world deployments. Not just for
> things like Ceph, but things like the manage/unmanage backend specific
> tests that were added and broke a large majority of third party CI.
>

Is there a policy in Cinder that a backend must implement a certain set of
APIs? If so we could think of testing only that set of APIs in Tempest, so
that
any app developer knows that he/she can rely on that minimum set of APIs.

If the list of APIs is not constrained on cinder side, the next driver
could come
along that does not support an API, and then we would have to stop testing
it
in Tempest - which is not an option.

Another point is that API that rely on services other than nova are best
tested
in Tempest, so that the tests run in the gate of other services as well -
or at
least the cinder functional test job should run against the other services
as well.


>
> Backend specific things should not be part of tempest in my opinion. We
> should cover those things through in-tree tempest plugins and our own
> testing.
> >
> > > 2. Tempest test can be disabled/skip based on backend. - This is not
> > > good idea as it increase config options and overhead of setting those.
> > >
> > Using regex and blacklist, any 3rd party CI can skip any test based on
> the
> > test ID. Without introducing a config flag. See:
> >
> https://github.com/openstack-infra/project-config/blob/1cea31f402b6b047cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871
> >
> >
> > > 3. Tempest test can verify behavior with if else condition as per
> > > backend. This is bad idea and lose the test strength.
> > >
> 

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Andrea Frittoli]

On Tue, May 2, 2017 at 2:41 PM Jordan Pittier 
wrote:

> On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann 
> wrote:
>
>> In Cinder, there are many features/APIs which are backend specific and
>> will return 405 or 501 if same is not implemented on any backend [1].
>> If such tests are implemented in Tempest, then it will break some gate
>> where that backend job is voting. like ceph job in glance_store gate.
>>
>> There been many such cases recently where ceph jobs were broken due to
>> such tests and recently it is for force-delete backup feature[2].
>> Reverting force-delete tests in [3]. To resolve such cases at some
>> extend, Jon is going to add a white/black list of tests which can run
>> on ceph job [4] depends on what all feature ceph implemented. But this
>> does not resolve it completely due to many reason like
>> 1. External use of Tempest become difficult where user needs to know
>> what all tests to skip for which backend
>> 2. Tempest tests become too specific to backend.
>>
>> Now there are few options to resolve this:
>> 1. Tempest should not tests such API/feature which are backend
>> specific like mentioned by api-ref like[1].
>>
> So basically, if one of the 50 Cinder driver doesn't support a feature, we
> should never test that feature ? What about the 49 other drivers ? If a
> feature exists and can be tested in the Gate (with whatever default
> config/driver is shipped) then I think we should test it.
>
>
>> 2. Tempest test can be disabled/skip based on backend. - This is not
>> good idea as it increase config options and overhead of setting those.
>>
> Using regex and blacklist, any 3rd party CI can skip any test based on the
> test ID. Without introducing a config flag. See:
> https://github.com/openstack-infra/project-config/blob/1cea31f402b6b047cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871
>

This way each 3rd party system has to maintain its own list, which has the
advantage that
different teams maintain their own list (which is nice from an ownership
and scale pov).

However I think such list of tests are not as re-usable as having a
devstack plugin (or an
ansible or puppet module) changing a few tempest config options.


>
>> 3. Tempest test can verify behavior with if else condition as per
>> backend. This is bad idea and lose the test strength.
>>
> Yeah, that's bad.
>
>>
>> IMO options 1 is better options. More feedback are welcome.
>
>
>> ..1
>> https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
>> ..2 https://bugs.launchpad.net/glance/+bug/1687538
>> ..3 https://review.openstack.org/#/c/461625/
>> ..4
>> http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html
>>
>> -gmann
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Andrea Frittoli]

On Tue, May 2, 2017 at 6:46 AM Ghanshyam Mann 
wrote:

> In Cinder, there are many features/APIs which are backend specific and
> will return 405 or 501 if same is not implemented on any backend [1].
> If such tests are implemented in Tempest, then it will break some gate
> where that backend job is voting. like ceph job in glance_store gate.
>

Having a test in Tempest is important for interoperability and API backward
compatibility.
As long as available features are discoverable and reported in a consistent
way, it
is possible for app developer to write application that will work fine
against
different backends.


>
> There been many such cases recently where ceph jobs were broken due to
> such tests and recently it is for force-delete backup feature[2].
> Reverting force-delete tests in [3]. To resolve such cases at some
> extend, Jon is going to add a white/black list of tests which can run
> on ceph job [4] depends on what all feature ceph implemented. But this
> does not resolve it completely due to many reason like
> 1. External use of Tempest become difficult where user needs to know
> what all tests to skip for which backend
> 2. Tempest tests become too specific to backend.
>
> Now there are few options to resolve this:
> 1. Tempest should not tests such API/feature which are backend
> specific like mentioned by api-ref like[1].
> 2. Tempest test can be disabled/skip based on backend. - This is not
> good idea as it increase config options and overhead of setting those.
>

Tempest has many options because we decide not to rely on discovery, i.e.
we configure what we expect to find in the target cloud. I don't think we
can use
this as a factor to influence the decision in this case.


> 3. Tempest test can verify behavior with if else condition as per
> backend. This is bad idea and lose the test strength.
>
> IMO options 1 is better options. More feedback are welcome.
>
> ..1
> https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
> ..2 https://bugs.launchpad.net/glance/+bug/1687538
> ..3 https://review.openstack.org/#/c/461625/
> ..4
> http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html
>
> -gmann
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Jeremy Stanley]

On 2017-05-02 10:49:55 -0500 (-0500), Sean McGinnis wrote:
[...]
> > So basically, if one of the 50 Cinder driver doesn't support a
> > feature, we should never test that feature ? What about the 49
> > other drivers ? If a feature exists and can be tested in the
> > Gate (with whatever default config/driver is shipped) then I
> > think we should test it.
> 
> 50? Over 100 as of Ocata.
[...]

The cover slide here just keeps getting truer and truer:

https://www.openstack.org/videos/vancouver-2015/openstack-is-doomed-and-it-is-your-fault

(also Thingee looks dashing in that scarf!)
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Sean McGinnis]

On Tue, May 02, 2017 at 03:36:20PM +0200, Jordan Pittier wrote:
> On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann 
> wrote:
> 
> > In Cinder, there are many features/APIs which are backend specific and
> > will return 405 or 501 if same is not implemented on any backend [1].
> > If such tests are implemented in Tempest, then it will break some gate
> > where that backend job is voting. like ceph job in glance_store gate.
> >
> > There been many such cases recently where ceph jobs were broken due to
> > such tests and recently it is for force-delete backup feature[2].
> > Reverting force-delete tests in [3]. To resolve such cases at some
> > extend, Jon is going to add a white/black list of tests which can run
> > on ceph job [4] depends on what all feature ceph implemented. But this
> > does not resolve it completely due to many reason like
> > 1. External use of Tempest become difficult where user needs to know
> > what all tests to skip for which backend
> > 2. Tempest tests become too specific to backend.
> >
> > Now there are few options to resolve this:
> > 1. Tempest should not tests such API/feature which are backend
> > specific like mentioned by api-ref like[1].
> >
> So basically, if one of the 50 Cinder driver doesn't support a feature, we
> should never test that feature ? What about the 49 other drivers ? If a
> feature exists and can be tested in the Gate (with whatever default
> config/driver is shipped) then I think we should test it.
> 
50? Over 100 as of Ocata.

Well, is tempest's purpose in life to provide complete gate test coverage,
or is tempest's purpose in life to give operators a tool to validate that
their deployment is working as expected?

In attempting to do things in the past, I've received push back based on
the argument that it was the latter. For this reason, in-tree tempest tests
were added to Cinder to give us a way to get better test coverage for our
own sake.

Now that this is all in place, I think it's working well and I would like
to see it continue that way. IMO, tempest proper should not have anything
that isn't universally applicable to real world deployments. Not just for
things like Ceph, but things like the manage/unmanage backend specific
tests that were added and broke a large majority of third party CI.

Backend specific things should not be part of tempest in my opinion. We
should cover those things through in-tree tempest plugins and our own
testing.
> 
> > 2. Tempest test can be disabled/skip based on backend. - This is not
> > good idea as it increase config options and overhead of setting those.
> >
> Using regex and blacklist, any 3rd party CI can skip any test based on the
> test ID. Without introducing a config flag. See:
> https://github.com/openstack-infra/project-config/blob/1cea31f402b6b047cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871
> 
> 
> > 3. Tempest test can verify behavior with if else condition as per
> > backend. This is bad idea and lose the test strength.
> >
> Yeah, that's bad.
> 
> >
> > IMO options 1 is better options. More feedback are welcome.
> 
> 
> > ..1 https://developer.openstack.org/api-ref/block-storage/v3/?
> > expanded=force-delete-a-backup-detail#force-delete-a-backup
> > ..2 https://bugs.launchpad.net/glance/+bug/1687538
> > ..3 https://review.openstack.org/#/c/461625/
> > ..4 http://lists.openstack.org/pipermail/openstack-dev/2017-
> > April/115229.html
> >
> > -gmann

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Eric Harney]

On 05/02/2017 01:42 AM, Ghanshyam Mann wrote:
> In Cinder, there are many features/APIs which are backend specific and
> will return 405 or 501 if same is not implemented on any backend [1].
> If such tests are implemented in Tempest, then it will break some gate
> where that backend job is voting. like ceph job in glance_store gate.
> 
> There been many such cases recently where ceph jobs were broken due to
> such tests and recently it is for force-delete backup feature[2].

This problem was detected on the initial patch [5], where the ceph gate
failed with the 405 error, but the patch was merged anyway.  Why are
there "many such cases" of these jobs getting broken?

[5] https://review.openstack.org/#/c/332670/

> Reverting force-delete tests in [3]. To resolve such cases at some
> extend, Jon is going to add a white/black list of tests which can run
> on ceph job [4] depends on what all feature ceph implemented. But this
> does not resolve it completely due to many reason like
> 1. External use of Tempest become difficult where user needs to know
> what all tests to skip for which backend
> 2. Tempest tests become too specific to backend.
> 
> Now there are few options to resolve this:
> 1. Tempest should not tests such API/feature which are backend
> specific like mentioned by api-ref like[1].

Is the proposal here to test these features via the in-tree Cinder
tempest tests instead of from tempest itself, or just not test any
features in Cinder which have backend-specific differences?

> 2. Tempest test can be disabled/skip based on backend. - This is not
> good idea as it increase config options and overhead of setting those.

This option seems like the most straightforward way to get both good
test coverage and handle compatibility.

> 3. Tempest test can verify behavior with if else condition as per
> backend. This is bad idea and lose the test strength.
> 
> IMO options 1 is better options. More feedback are welcome.
> 
> ..1 
> https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
> ..2 https://bugs.launchpad.net/glance/+bug/1687538
> ..3 https://review.openstack.org/#/c/461625/
> ..4 http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html
> 
> -gmann
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Jordan Pittier]

On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann 
wrote:

> In Cinder, there are many features/APIs which are backend specific and
> will return 405 or 501 if same is not implemented on any backend [1].
> If such tests are implemented in Tempest, then it will break some gate
> where that backend job is voting. like ceph job in glance_store gate.
>
> There been many such cases recently where ceph jobs were broken due to
> such tests and recently it is for force-delete backup feature[2].
> Reverting force-delete tests in [3]. To resolve such cases at some
> extend, Jon is going to add a white/black list of tests which can run
> on ceph job [4] depends on what all feature ceph implemented. But this
> does not resolve it completely due to many reason like
> 1. External use of Tempest become difficult where user needs to know
> what all tests to skip for which backend
> 2. Tempest tests become too specific to backend.
>
> Now there are few options to resolve this:
> 1. Tempest should not tests such API/feature which are backend
> specific like mentioned by api-ref like[1].
>
So basically, if one of the 50 Cinder driver doesn't support a feature, we
should never test that feature ? What about the 49 other drivers ? If a
feature exists and can be tested in the Gate (with whatever default
config/driver is shipped) then I think we should test it.


> 2. Tempest test can be disabled/skip based on backend. - This is not
> good idea as it increase config options and overhead of setting those.
>
Using regex and blacklist, any 3rd party CI can skip any test based on the
test ID. Without introducing a config flag. See:
https://github.com/openstack-infra/project-config/blob/1cea31f402b6b047cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871


> 3. Tempest test can verify behavior with if else condition as per
> backend. This is bad idea and lose the test strength.
>
Yeah, that's bad.

>
> IMO options 1 is better options. More feedback are welcome.


> ..1 https://developer.openstack.org/api-ref/block-storage/v3/?
> expanded=force-delete-a-backup-detail#force-delete-a-backup
> ..2 https://bugs.launchpad.net/glance/+bug/1687538
> ..3 https://review.openstack.org/#/c/461625/
> ..4 http://lists.openstack.org/pipermail/openstack-dev/2017-
> April/115229.html
>
> -gmann
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Ghanshyam Mann]

In Cinder, there are many features/APIs which are backend specific and
will return 405 or 501 if same is not implemented on any backend [1].
If such tests are implemented in Tempest, then it will break some gate
where that backend job is voting. like ceph job in glance_store gate.

There been many such cases recently where ceph jobs were broken due to
such tests and recently it is for force-delete backup feature[2].
Reverting force-delete tests in [3]. To resolve such cases at some
extend, Jon is going to add a white/black list of tests which can run
on ceph job [4] depends on what all feature ceph implemented. But this
does not resolve it completely due to many reason like
1. External use of Tempest become difficult where user needs to know
what all tests to skip for which backend
2. Tempest tests become too specific to backend.

Now there are few options to resolve this:
1. Tempest should not tests such API/feature which are backend
specific like mentioned by api-ref like[1].
2. Tempest test can be disabled/skip based on backend. - This is not
good idea as it increase config options and overhead of setting those.
3. Tempest test can verify behavior with if else condition as per
backend. This is bad idea and lose the test strength.

IMO options 1 is better options. More feedback are welcome.

..1 
https://developer.openstack.org/api-ref/block-storage/v3/?expanded=force-delete-a-backup-detail#force-delete-a-backup
..2 https://bugs.launchpad.net/glance/+bug/1687538
..3 https://review.openstack.org/#/c/461625/
..4 http://lists.openstack.org/pipermail/openstack-dev/2017-April/115229.html

-gmann

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev