Re: [openstack-dev] [QA]Refactoring Scenarios manager.py

[Jordan Pittier]

On Wed, Mar 1, 2017 at 4:18 AM,  wrote:

> I think it a good solution, I already put +1 :)
>
>
> And, as to the scenario testcases, shall we:
>
> 1) remove test steps/checks already coverd in API test
>
Duplicate test steps/checks is not good and should be removed. It's not
related to the scenario refactoring effort, so please if you find
duplicated tests or test steps, we should remove them.

> 2) remove sequence test cases (such as test_server_sequence_suspend_resume),
> othersize scenario will get fatter and fatter
>
There's no definitive answer to that. We should just remember what a
scenario should be: test several openstack components, "real" world use
cases, "real" integration testing. Those should be our guideline for
scenarios. We should not buy into "I put it into the scenarios directory
because the helper methods were here and convenient" or "because I saw an
already existing scenario that look kind of the same".
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [qa][cinder][ceph] should Tempest tests the backend specific feature?

[Jordan Pittier]

On Tue, May 2, 2017 at 7:42 AM, Ghanshyam Mann 
wrote:

> In Cinder, there are many features/APIs which are backend specific and
> will return 405 or 501 if same is not implemented on any backend [1].
> If such tests are implemented in Tempest, then it will break some gate
> where that backend job is voting. like ceph job in glance_store gate.
>
> There been many such cases recently where ceph jobs were broken due to
> such tests and recently it is for force-delete backup feature[2].
> Reverting force-delete tests in [3]. To resolve such cases at some
> extend, Jon is going to add a white/black list of tests which can run
> on ceph job [4] depends on what all feature ceph implemented. But this
> does not resolve it completely due to many reason like
> 1. External use of Tempest become difficult where user needs to know
> what all tests to skip for which backend
> 2. Tempest tests become too specific to backend.
>
> Now there are few options to resolve this:
> 1. Tempest should not tests such API/feature which are backend
> specific like mentioned by api-ref like[1].
>
So basically, if one of the 50 Cinder driver doesn't support a feature, we
should never test that feature ? What about the 49 other drivers ? If a
feature exists and can be tested in the Gate (with whatever default
config/driver is shipped) then I think we should test it.


> 2. Tempest test can be disabled/skip based on backend. - This is not
> good idea as it increase config options and overhead of setting those.
>
Using regex and blacklist, any 3rd party CI can skip any test based on the
test ID. Without introducing a config flag. See:
https://github.com/openstack-infra/project-config/blob/1cea31f402b6b047cde203c12184b5392c90/jenkins/jobs/devstack-gate.yaml#L1871


> 3. Tempest test can verify behavior with if else condition as per
> backend. This is bad idea and lose the test strength.
>
Yeah, that's bad.

>
> IMO options 1 is better options. More feedback are welcome.


> ..1 https://developer.openstack.org/api-ref/block-storage/v3/?
> expanded=force-delete-a-backup-detail#force-delete-a-backup
> ..2 https://bugs.launchpad.net/glance/+bug/1687538
> ..3 https://review.openstack.org/#/c/461625/
> ..4 http://lists.openstack.org/pipermail/openstack-dev/2017-
> April/115229.html
>
> -gmann
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev