Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Sam P]

Hi Dims, Hi gmann,

Thanks guys.
 Slowly recovering from PTG jet lag

OK then.., I will start working on this.
I would also like to coordinate this work with LCOO [1] team,  where
we can get valuable feedback and contributions..
I will give you update on next QA IRC on 0900 UTC meeting, which is
9th of March?
(1700 UTC meeting is difficult for me, however I will definitely be
there if needed)

[1] https://wiki.openstack.org/wiki/LCOO
--- Regards,
Sampath



On Fri, Feb 24, 2017 at 8:57 PM, Ghanshyam Mann <ghanshyamm...@gmail.com> wrote:
> Yea, agree with dims.
>
> Sampath ,
>
> Thanks for taking over this, it is really great help. Please update the
> current spec with approaches you have. Timur help will be great if he show
> up sometime.
>
> Also we will add destructive testing as one of weekly meeting agenda and
> make sure you will get all help & required attention from QA team.
>
> -gmann
>
>
> On Fri, Feb 24, 2017 at 7:26 AM, Davanum Srinivas <dava...@gmail.com> wrote:
>>
>> Sampath,
>>
>> I am not sure if you will hear back from Timur soon as he may not be
>> working on this stuff anymore (in Mirantis). So i'd recommend picking
>> up the work if you don't hear from him soon.
>>
>> Thanks,
>> Dims
>>
>> On Thu, Feb 23, 2017 at 3:41 PM, Sam P <sam47pr...@gmail.com> wrote:
>> > Hi Timur,
>> >
>> >  The current status of this work is,
>> > 1) The QA user story for destructive testing of OpenStack cloud [1] is
>> > merged .
>> > 2) The spec for a new framework which will focus on HA/failover and
>> > destructive testing  [2] has no update since Nov 30 2016.
>> > 3) The commit for the new repository [3]  has abandoned due to no
>> > update after Nov 29 2016.
>> >
>> > Currently, I am working on 3rd party destructive/HA testing CI for
>> > Masakari[4] and very much interested in this work.
>> > I will keep working on [1] with PWG.
>> > Please let me know your plans for [2], and [3].
>> > If it is difficult for you to continue, I would love to continue your
>> > work on [2], and [3].
>> >
>> > [1] https://review.openstack.org/#/c/396142
>> > [2] https://review.openstack.org/#/c/399618
>> > [3] https://review.openstack.org/#/c/374667
>> > [4] https://wiki.openstack.org/wiki/Masakari
>> > --- Regards,
>> > Sampath
>> >
>> >
>> >
>> > On Mon, Nov 28, 2016 at 6:37 AM, Timur Nurlygayanov
>> > <tnurlygaya...@mirantis.com> wrote:
>> >> Hi team,
>> >>
>> >> here is a short update:
>> >>
>> >> 1) The QA user story for destructive testing of OpenStack cloud is on
>> >> review
>> >> [1].
>> >> 2) The spec for a new framework which will focus on HA/failover and
>> >> destructive testing is no review [2].
>> >> 3) The commit for the new repository is on review [3] as well.
>> >>
>> >> [1] https://review.openstack.org/#/c/396142
>> >> [2] https://review.openstack.org/#/c/399618
>> >> [3] https://review.openstack.org/#/c/374667
>> >>
>> >>
>> >> On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann
>> >> <ghanshyam.m...@nectechnologies.in> wrote:
>> >>>
>> >>> I like os-faults library which can provide the abstraction over
>> >>> different
>> >>> destructive actions like reboot/poweroff node etc.
>> >>>
>> >>>
>> >>>
>> >>> But not much clear about Stepler that what all tests it will contain.
>> >>> Tempest do have scenario tests which can hit the production env with
>> >>> significant way of testing scenario.
>> >>>
>> >>> It can cover the end user scenario also which involves the interaction
>> >>> of
>> >>> public OpenStack APIs and always in enhancement state by adding more
>> >>> and
>> >>> more scenario tests.
>> >>>
>> >>>
>> >>>
>> >>> Few query over Stepler as separate project:
>> >>>
>> >>> 1.   Is Stepler will cover only tests which hits the node level
>> >>> action(reboot, HA etc)?
>> >>>
>> >>> 2.   This should not mix the scenario tests which are in Tempest
>> >>> scope
>> >>> because that can make confusion for developers (where to add scenario
>> >>> tests)
>> >>> 

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Ghanshyam Mann]

Yea, agree with dims.

Sampath ,

Thanks for taking over this, it is really great help. Please update the
current spec with approaches you have. Timur help will be great if he show
up sometime.

Also we will add destructive testing as one of weekly meeting agenda and
make sure you will get all help & required attention from QA team.

-gmann

On Fri, Feb 24, 2017 at 7:26 AM, Davanum Srinivas <dava...@gmail.com> wrote:

> Sampath,
>
> I am not sure if you will hear back from Timur soon as he may not be
> working on this stuff anymore (in Mirantis). So i'd recommend picking
> up the work if you don't hear from him soon.
>
> Thanks,
> Dims
>
> On Thu, Feb 23, 2017 at 3:41 PM, Sam P <sam47pr...@gmail.com> wrote:
> > Hi Timur,
> >
> >  The current status of this work is,
> > 1) The QA user story for destructive testing of OpenStack cloud [1] is
> merged .
> > 2) The spec for a new framework which will focus on HA/failover and
> > destructive testing  [2] has no update since Nov 30 2016.
> > 3) The commit for the new repository [3]  has abandoned due to no
> > update after Nov 29 2016.
> >
> > Currently, I am working on 3rd party destructive/HA testing CI for
> > Masakari[4] and very much interested in this work.
> > I will keep working on [1] with PWG.
> > Please let me know your plans for [2], and [3].
> > If it is difficult for you to continue, I would love to continue your
> > work on [2], and [3].
> >
> > [1] https://review.openstack.org/#/c/396142
> > [2] https://review.openstack.org/#/c/399618
> > [3] https://review.openstack.org/#/c/374667
> > [4] https://wiki.openstack.org/wiki/Masakari
> > --- Regards,
> > Sampath
> >
> >
> >
> > On Mon, Nov 28, 2016 at 6:37 AM, Timur Nurlygayanov
> > <tnurlygaya...@mirantis.com> wrote:
> >> Hi team,
> >>
> >> here is a short update:
> >>
> >> 1) The QA user story for destructive testing of OpenStack cloud is on
> review
> >> [1].
> >> 2) The spec for a new framework which will focus on HA/failover and
> >> destructive testing is no review [2].
> >> 3) The commit for the new repository is on review [3] as well.
> >>
> >> [1] https://review.openstack.org/#/c/396142
> >> [2] https://review.openstack.org/#/c/399618
> >> [3] https://review.openstack.org/#/c/374667
> >>
> >>
> >> On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann
> >> <ghanshyam.m...@nectechnologies.in> wrote:
> >>>
> >>> I like os-faults library which can provide the abstraction over
> different
> >>> destructive actions like reboot/poweroff node etc.
> >>>
> >>>
> >>>
> >>> But not much clear about Stepler that what all tests it will contain.
> >>> Tempest do have scenario tests which can hit the production env with
> >>> significant way of testing scenario.
> >>>
> >>> It can cover the end user scenario also which involves the interaction
> of
> >>> public OpenStack APIs and always in enhancement state by adding more
> and
> >>> more scenario tests.
> >>>
> >>>
> >>>
> >>> Few query over Stepler as separate project:
> >>>
> >>> 1.   Is Stepler will cover only tests which hits the node level
> >>> action(reboot, HA etc)?
> >>>
> >>> 2.   This should not mix the scenario tests which are in Tempest
> scope
> >>> because that can make confusion for developers (where to add scenario
> tests)
> >>> as well as for tester(from where to run what scenario tests).
> >>>
> >>> 3.   How we make sure those tests run fine or at least run while
> >>> adding.
> >>>
> >>> a.       I think devstack enhancement for multi-node etc can do this as
> >>> mentioned by you also.
> >>>
> >>> b.  If so then why not adding those tests in Tempest only using
> >>> os-faults lib ?
> >>>
> >>>
> >>>
> >>> Overall I feel os-faults  as lib is really nice idea but tests scope
> can
> >>> go in Tempest under HW_scenario  (or something else) till it does not
> break
> >>> basic principle of Tempest.
> >>>
> >>>
> >>>
> >>> Thanks
> >>>
> >>> gmann
> >>>
> >>>
> >>>
> >>> From: Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com]
> >>> Sent: 

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Davanum Srinivas]

Sampath,

I am not sure if you will hear back from Timur soon as he may not be
working on this stuff anymore (in Mirantis). So i'd recommend picking
up the work if you don't hear from him soon.

Thanks,
Dims

On Thu, Feb 23, 2017 at 3:41 PM, Sam P <sam47pr...@gmail.com> wrote:
> Hi Timur,
>
>  The current status of this work is,
> 1) The QA user story for destructive testing of OpenStack cloud [1] is merged 
> .
> 2) The spec for a new framework which will focus on HA/failover and
> destructive testing  [2] has no update since Nov 30 2016.
> 3) The commit for the new repository [3]  has abandoned due to no
> update after Nov 29 2016.
>
> Currently, I am working on 3rd party destructive/HA testing CI for
> Masakari[4] and very much interested in this work.
> I will keep working on [1] with PWG.
> Please let me know your plans for [2], and [3].
> If it is difficult for you to continue, I would love to continue your
> work on [2], and [3].
>
> [1] https://review.openstack.org/#/c/396142
> [2] https://review.openstack.org/#/c/399618
> [3] https://review.openstack.org/#/c/374667
> [4] https://wiki.openstack.org/wiki/Masakari
> --- Regards,
> Sampath
>
>
>
> On Mon, Nov 28, 2016 at 6:37 AM, Timur Nurlygayanov
> <tnurlygaya...@mirantis.com> wrote:
>> Hi team,
>>
>> here is a short update:
>>
>> 1) The QA user story for destructive testing of OpenStack cloud is on review
>> [1].
>> 2) The spec for a new framework which will focus on HA/failover and
>> destructive testing is no review [2].
>> 3) The commit for the new repository is on review [3] as well.
>>
>> [1] https://review.openstack.org/#/c/396142
>> [2] https://review.openstack.org/#/c/399618
>> [3] https://review.openstack.org/#/c/374667
>>
>>
>> On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann
>> <ghanshyam.m...@nectechnologies.in> wrote:
>>>
>>> I like os-faults library which can provide the abstraction over different
>>> destructive actions like reboot/poweroff node etc.
>>>
>>>
>>>
>>> But not much clear about Stepler that what all tests it will contain.
>>> Tempest do have scenario tests which can hit the production env with
>>> significant way of testing scenario.
>>>
>>> It can cover the end user scenario also which involves the interaction of
>>> public OpenStack APIs and always in enhancement state by adding more and
>>> more scenario tests.
>>>
>>>
>>>
>>> Few query over Stepler as separate project:
>>>
>>> 1.   Is Stepler will cover only tests which hits the node level
>>> action(reboot, HA etc)?
>>>
>>> 2.   This should not mix the scenario tests which are in Tempest scope
>>> because that can make confusion for developers (where to add scenario tests)
>>> as well as for tester(from where to run what scenario tests).
>>>
>>> 3.   How we make sure those tests run fine or at least run while
>>> adding.
>>>
>>> a.   I think devstack enhancement for multi-node etc can do this as
>>> mentioned by you also.
>>>
>>> b.  If so then why not adding those tests in Tempest only using
>>> os-faults lib ?
>>>
>>>
>>>
>>> Overall I feel os-faults  as lib is really nice idea but tests scope can
>>> go in Tempest under HW_scenario  (or something else) till it does not break
>>> basic principle of Tempest.
>>>
>>>
>>>
>>> Thanks
>>>
>>> gmann
>>>
>>>
>>>
>>> From: Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com]
>>> Sent: 06 October 2016 20:09
>>> To: OpenStack Development Mailing List (not for usage questions)
>>> <openstack-dev@lists.openstack.org>
>>> Subject: Re: [openstack-dev] [QA] The end-user test suite for OpenStack
>>> clusters
>>>
>>>
>>>
>>> Ken, it is a good idea!
>>>
>>>
>>>
>>> The plan is to develop os-faults as a library which will be able
>>>
>>> to manage cluster nodes and OpenStack services on these nodes.
>>>
>>> It is a good idea to add some Tempest tests which will use
>>>
>>> os-faults library as well for some API tests.
>>>
>>>
>>>
>>> The Stepler framework [1] will use os-faults to perform all destructive
>>>
>>> actions in the clouds (the reboot of nodes, restart of OpenStack services,
>>>
>>> enable/disable network interfaces o

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Sam P]

Hi Timur,

 The current status of this work is,
1) The QA user story for destructive testing of OpenStack cloud [1] is merged .
2) The spec for a new framework which will focus on HA/failover and
destructive testing  [2] has no update since Nov 30 2016.
3) The commit for the new repository [3]  has abandoned due to no
update after Nov 29 2016.

Currently, I am working on 3rd party destructive/HA testing CI for
Masakari[4] and very much interested in this work.
I will keep working on [1] with PWG.
Please let me know your plans for [2], and [3].
If it is difficult for you to continue, I would love to continue your
work on [2], and [3].

[1] https://review.openstack.org/#/c/396142
[2] https://review.openstack.org/#/c/399618
[3] https://review.openstack.org/#/c/374667
[4] https://wiki.openstack.org/wiki/Masakari
--- Regards,
Sampath



On Mon, Nov 28, 2016 at 6:37 AM, Timur Nurlygayanov
<tnurlygaya...@mirantis.com> wrote:
> Hi team,
>
> here is a short update:
>
> 1) The QA user story for destructive testing of OpenStack cloud is on review
> [1].
> 2) The spec for a new framework which will focus on HA/failover and
> destructive testing is no review [2].
> 3) The commit for the new repository is on review [3] as well.
>
> [1] https://review.openstack.org/#/c/396142
> [2] https://review.openstack.org/#/c/399618
> [3] https://review.openstack.org/#/c/374667
>
>
> On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann
> <ghanshyam.m...@nectechnologies.in> wrote:
>>
>> I like os-faults library which can provide the abstraction over different
>> destructive actions like reboot/poweroff node etc.
>>
>>
>>
>> But not much clear about Stepler that what all tests it will contain.
>> Tempest do have scenario tests which can hit the production env with
>> significant way of testing scenario.
>>
>> It can cover the end user scenario also which involves the interaction of
>> public OpenStack APIs and always in enhancement state by adding more and
>> more scenario tests.
>>
>>
>>
>> Few query over Stepler as separate project:
>>
>> 1.   Is Stepler will cover only tests which hits the node level
>> action(reboot, HA etc)?
>>
>> 2.   This should not mix the scenario tests which are in Tempest scope
>> because that can make confusion for developers (where to add scenario tests)
>> as well as for tester(from where to run what scenario tests).
>>
>> 3.   How we make sure those tests run fine or at least run while
>> adding.
>>
>> a.   I think devstack enhancement for multi-node etc can do this as
>> mentioned by you also.
>>
>> b.  If so then why not adding those tests in Tempest only using
>> os-faults lib ?
>>
>>
>>
>> Overall I feel os-faults  as lib is really nice idea but tests scope can
>> go in Tempest under HW_scenario  (or something else) till it does not break
>> basic principle of Tempest.
>>
>>
>>
>> Thanks
>>
>> gmann
>>
>>
>>
>> From: Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com]
>> Sent: 06 October 2016 20:09
>> To: OpenStack Development Mailing List (not for usage questions)
>> <openstack-dev@lists.openstack.org>
>> Subject: Re: [openstack-dev] [QA] The end-user test suite for OpenStack
>> clusters
>>
>>
>>
>> Ken, it is a good idea!
>>
>>
>>
>> The plan is to develop os-faults as a library which will be able
>>
>> to manage cluster nodes and OpenStack services on these nodes.
>>
>> It is a good idea to add some Tempest tests which will use
>>
>> os-faults library as well for some API tests.
>>
>>
>>
>> The Stepler framework [1] will use os-faults to perform all destructive
>>
>> actions in the clouds (the reboot of nodes, restart of OpenStack services,
>>
>> enable/disable network interfaces or some firewall rules and etc).
>>
>>
>>
>> We need to get +1 from you in [1] to create the repository with
>>
>> advanced end-user scenario tests.
>>
>>
>>
>> Thank you!
>>
>>
>>
>> [1] https://review.openstack.org/#/c/374667/
>>
>>
>>
>> On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov <yloban...@mirantis.com>
>> wrote:
>>
>> Hi Ken,
>>
>>
>>
>> OS-Faults doesn't have any scenarios in the tree yet (the project is two
>> months old), but you can find some examples of the use in the
>> os-faults/examples directory.
>>
>>
>>
>> Regards,
>>
>> Yaroslav Lobankov.
>>
>>
&g

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Timur Nurlygayanov]

Hi team,

here is a short update:

1) The QA user story for destructive testing of OpenStack cloud is on
review [1].
2) The spec for a new framework which will focus on HA/failover and
destructive testing is no review [2].
3) The commit for the new repository is on review [3] as well.

[1] https://review.openstack.org/#/c/396142
[2] https://review.openstack.org/#/c/399618
[3] https://review.openstack.org/#/c/374667


On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann <
ghanshyam.m...@nectechnologies.in> wrote:

> I like os-faults library which can provide the abstraction over different
> destructive actions like reboot/poweroff node etc.
>
>
>
> But not much clear about Stepler that what all tests it will contain.
> Tempest do have scenario tests which can hit the production env with
> significant way of testing scenario.
>
> It can cover the end user scenario also which involves the interaction of
> public OpenStack APIs and always in enhancement state by adding more and
> more scenario tests.
>
>
>
> Few query over Stepler as separate project:
>
> 1.   Is Stepler will cover only tests which hits the node level
> action(reboot, HA etc)?
>
> 2.   This should not mix the scenario tests which are in Tempest
> scope because that can make confusion for developers (where to add scenario
> tests) as well as for tester(from where to run what scenario tests).
>
> 3.   How we make sure those tests run fine or at least run while
> adding.
>
> a.   I think devstack enhancement for multi-node etc can do this as
> mentioned by you also.
>
> b.  If so then why not adding those tests in Tempest only using
> os-faults lib ?
>
>
>
> Overall I feel os-faults  as lib is really nice idea but tests scope can
> go in Tempest under HW_scenario  (or something else) till it does not break
> basic principle of Tempest.
>
>
>
> Thanks
>
> gmann
>
>
>
> *From:* Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com]
> *Sent:* 06 October 2016 20:09
> *To:* OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>
> *Subject:* Re: [openstack-dev] [QA] The end-user test suite for OpenStack
> clusters
>
>
>
> Ken, it is a good idea!
>
>
>
> The plan is to develop os-faults as a library which will be able
>
> to manage cluster nodes and OpenStack services on these nodes.
>
> It is a good idea to add some Tempest tests which will use
>
> os-faults library as well for some API tests.
>
>
>
> The Stepler framework [1] will use os-faults to perform all destructive
>
> actions in the clouds (the reboot of nodes, restart of OpenStack services,
>
> enable/disable network interfaces or some firewall rules and etc).
>
>
>
> We need to get +1 from you in [1] to create the repository with
>
> advanced end-user scenario tests.
>
>
>
> Thank you!
>
>
>
> [1] https://review.openstack.org/#/c/374667/
>
>
>
> On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov <yloban...@mirantis.com>
> wrote:
>
> Hi Ken,
>
>
>
> OS-Faults doesn't have any scenarios in the tree yet (the project is two
> months old), but you can find some examples of the use in the
> os-faults/examples directory.
>
>
>
> Regards,
>
> Yaroslav Lobankov.
>
>
>
> On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi <ken1ohmi...@gmail.com>
> wrote:
>
> Hi Timur,
>
> Thanks for your explanation.
>
>
> 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov <tnurlygaya...@mirantis.com>:
> >
> >> I am guessing the above "restart nodes" is for verifying each
> >> OpenStack service restarts successfully, right?
> >
> > Yes, this is right. And we also will check that HA logic for these
> > services works correctly (for example, rescheduling of L3 Neutron
> > agents for networks).
> >
> >> But these service scripts are provided by distributors, and Devstack
> >> itself doesn't contain service scripts IIUC.
> >> So I'd like to know how to verify it on Devstack clouds.
> >
> > Yes, DevStack doesn't support many scenarios which are actual
> > and should be supported on the production clouds.
> > It will be not possible to run all advanced test scenarios for DevStack
> > clouds,
> > just because DevStack can't deploy OpenStack cloud with 3 controllers
> > now (so, probably it will be possible in the future).
> >
> > Of course, some advanced scenarios will support DevStack clouds,
> > for example, some test scenarios which are based on customer-found
> > issues from the real production clouds, like upload of the large ima

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Ghanshyam Mann]

I like os-faults library which can provide the abstraction over different 
destructive actions like reboot/poweroff node etc.

But not much clear about Stepler that what all tests it will contain. Tempest 
do have scenario tests which can hit the production env with significant way of 
testing scenario.
It can cover the end user scenario also which involves the interaction of 
public OpenStack APIs and always in enhancement state by adding more and more 
scenario tests.

Few query over Stepler as separate project:

1.   Is Stepler will cover only tests which hits the node level 
action(reboot, HA etc)?

2.   This should not mix the scenario tests which are in Tempest scope 
because that can make confusion for developers (where to add scenario tests) as 
well as for tester(from where to run what scenario tests).

3.   How we make sure those tests run fine or at least run while adding.

a.   I think devstack enhancement for multi-node etc can do this as 
mentioned by you also.

b.  If so then why not adding those tests in Tempest only using os-faults 
lib ?

Overall I feel os-faults  as lib is really nice idea but tests scope can go in 
Tempest under HW_scenario  (or something else) till it does not break basic 
principle of Tempest.

Thanks
gmann

From: Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com]
Sent: 06 October 2016 20:09
To: OpenStack Development Mailing List (not for usage questions) 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

Ken, it is a good idea!

The plan is to develop os-faults as a library which will be able
to manage cluster nodes and OpenStack services on these nodes.
It is a good idea to add some Tempest tests which will use
os-faults library as well for some API tests.

The Stepler framework [1] will use os-faults to perform all destructive
actions in the clouds (the reboot of nodes, restart of OpenStack services,
enable/disable network interfaces or some firewall rules and etc).

We need to get +1 from you in [1] to create the repository with
advanced end-user scenario tests.

Thank you!

[1] https://review.openstack.org/#/c/374667/

On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov 
<yloban...@mirantis.com<mailto:yloban...@mirantis.com>> wrote:
Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two months 
old), but you can find some examples of the use in the os-faults/examples 
directory.

Regards,
Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi 
<ken1ohmi...@gmail.com<mailto:ken1ohmi...@gmail.com>> wrote:
Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov 
<tnurlygaya...@mirantis.com<mailto:tnurlygaya...@mirantis.com>>:
>
>> I am guessing the above "restart nodes" is for verifying each
>> OpenStack service restarts successfully, right?
>
> Yes, this is right. And we also will check that HA logic for these
> services works correctly (for example, rescheduling of L3 Neutron
> agents for networks).
>
>> But these service scripts are provided by distributors, and Devstack
>> itself doesn't contain service scripts IIUC.
>> So I'd like to know how to verify it on Devstack clouds.
>
> Yes, DevStack doesn't support many scenarios which are actual
> and should be supported on the production clouds.
> It will be not possible to run all advanced test scenarios for DevStack
> clouds,
> just because DevStack can't deploy OpenStack cloud with 3 controllers
> now (so, probably it will be possible in the future).
>
> Of course, some advanced scenarios will support DevStack clouds,
> for example, some test scenarios which are based on customer-found
> issues from the real production clouds, like upload of the large images
> (100+ Gb)
> to Glance with Swift backend. Such cases are important for verification of
> pre-production environments, but not very important for CI gate jobs.
>
> It is also important to note that in these advanced cases we are targeting
> to check not only the logic of Python code, but also the correct
> configuration
> of all OpenStack components on some pre-production OpenStack clusters.
I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
opens

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Ken'ichi Ohmichi]

Hi Timur,

I discussed this on irc, and it seems hard to get feedback from many
people because now the scope and implementation of this os-faults are
separated into multiple pieces(like multiple mails, github code,
gerrit etc).
So how about proposing this as qa-spec?
If doing that, We can discuss this on clear document and it is clear
to get a consensus about adding this.

Thanks
Ken Ohmichi

---

.

2016-10-07 12:43 GMT-07:00 Ken'ichi Ohmichi :
> Hi Timur,
>
> 2016-10-06 4:08 GMT-07:00 Timur Nurlygayanov :
>> Ken, it is a good idea!
>>
>> The plan is to develop os-faults as a library which will be able
>> to manage cluster nodes and OpenStack services on these nodes.
>> It is a good idea to add some Tempest tests which will use
>> os-faults library as well for some API tests.
>
> Sorry for my misleading, that is not I wanted to say.
> I am thinking Tempest doesn't use os-faults as library.
> I just wanted to say some destructive tests which will use REST APIs
> can be implemented in Tempest instead of os-faults.
>
> For example, the compute service client contains disable_service which
> disables nova service but Tempest doesn't contain the corresponding
> test cases because Tempest tests need to run in parallel.
> https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/services_client.py#L54
>
> If some tests use this method, the other tests which run in parallel
> can be failed as unexpected on the gate.
> Such tests also are destructive and I am hoping these tests also will
> be able to run the gate by finding better way.
>
>> The Stepler framework [1] will use os-faults to perform all destructive
>> actions in the clouds (the reboot of nodes, restart of OpenStack services,
>> enable/disable network interfaces or some firewall rules and etc).
>>
>> We need to get +1 from you in [1] to create the repository with
>> advanced end-user scenario tests.
>
> OK, I'd like to summarize my thinking here for adding os-faults under
> QA project:
>
> Under QA project, the first target of most programs(tempest, devstack,
> etc) is for the gate testing.
> Of course, some of them are available on production clouds also as the
> design, but the first is for the gate.
> That means devstack clouds as the first target/purpose.
> At this time, this os-faults doesn't seem the gate as the first
> purpose based on current implementation.
> So I feel os-faults seems different from the existing programs.
>
> os-faults is very young and we will be able to extend it for devstack
> clouds also later, I hope.
> In addition, I heard this kind of tests(destructive, HA, etc) are
> requested from the other users.
> So this os-faults seems very valuable for users.
>
> Just in my opinion, I am find to add this os-faults under QA project.
> But before that, I'd like to get opinions from the other people of QA team.
>
> Thanks
> Ken Ohmichi
>
>
>
>
>
>
>
>
>
>
>
>
>> On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov 
>> wrote:
>>>
>>> Hi Ken,
>>>
>>> OS-Faults doesn't have any scenarios in the tree yet (the project is two
>>> months old), but you can find some examples of the use in the
>>> os-faults/examples directory.
>>>
>>> Regards,
>>> Yaroslav Lobankov.
>>>
>>> On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi 
>>> wrote:

 Hi Timur,

 Thanks for your explanation.

 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov
 :
 >
 >> I am guessing the above "restart nodes" is for verifying each
 >> OpenStack service restarts successfully, right?
 >
 > Yes, this is right. And we also will check that HA logic for these
 > services works correctly (for example, rescheduling of L3 Neutron
 > agents for networks).
 >
 >> But these service scripts are provided by distributors, and Devstack
 >> itself doesn't contain service scripts IIUC.
 >> So I'd like to know how to verify it on Devstack clouds.
 >
 > Yes, DevStack doesn't support many scenarios which are actual
 > and should be supported on the production clouds.
 > It will be not possible to run all advanced test scenarios for DevStack
 > clouds,
 > just because DevStack can't deploy OpenStack cloud with 3 controllers
 > now (so, probably it will be possible in the future).
 >
 > Of course, some advanced scenarios will support DevStack clouds,
 > for example, some test scenarios which are based on customer-found
 > issues from the real production clouds, like upload of the large images
 > (100+ Gb)
 > to Glance with Swift backend. Such cases are important for verification
 > of
 > pre-production environments, but not very important for CI gate jobs.
 >
 > It is also important to note that in these advanced cases we are
 > targeting
 > to check not only the logic of Python code, but also the correct
 > 

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Masayuki Igawa]

Hi,

My first impression is "It's interesting" :)

I actually don't read whole of this threads, but, I re-read the QA
mission statement[1].
I feel os-faults is a little bit far from the mission statement.
Because os-faults seems like focusing on operator testing.

But I think this can be under the QA project. Because tempest scope
has also operator testing.

[1] https://wiki.openstack.org/wiki/QA#Project_Team_Definition


On Sat, Oct 8, 2016 at 4:43 AM, Ken'ichi Ohmichi  wrote:
> Hi Timur,
>
> 2016-10-06 4:08 GMT-07:00 Timur Nurlygayanov :
>> Ken, it is a good idea!
>>
>> The plan is to develop os-faults as a library which will be able
>> to manage cluster nodes and OpenStack services on these nodes.
>> It is a good idea to add some Tempest tests which will use
>> os-faults library as well for some API tests.
>
> Sorry for my misleading, that is not I wanted to say.
> I am thinking Tempest doesn't use os-faults as library.
> I just wanted to say some destructive tests which will use REST APIs
> can be implemented in Tempest instead of os-faults.
>
> For example, the compute service client contains disable_service which
> disables nova service but Tempest doesn't contain the corresponding
> test cases because Tempest tests need to run in parallel.
> https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/services_client.py#L54
>
> If some tests use this method, the other tests which run in parallel
> can be failed as unexpected on the gate.
> Such tests also are destructive and I am hoping these tests also will
> be able to run the gate by finding better way.
>
>> The Stepler framework [1] will use os-faults to perform all destructive
>> actions in the clouds (the reboot of nodes, restart of OpenStack services,
>> enable/disable network interfaces or some firewall rules and etc).
>>
>> We need to get +1 from you in [1] to create the repository with
>> advanced end-user scenario tests.
>
> OK, I'd like to summarize my thinking here for adding os-faults under
> QA project:
>
> Under QA project, the first target of most programs(tempest, devstack,
> etc) is for the gate testing.
> Of course, some of them are available on production clouds also as the
> design, but the first is for the gate.
> That means devstack clouds as the first target/purpose.
> At this time, this os-faults doesn't seem the gate as the first
> purpose based on current implementation.
> So I feel os-faults seems different from the existing programs.
>
> os-faults is very young and we will be able to extend it for devstack
> clouds also later, I hope.
> In addition, I heard this kind of tests(destructive, HA, etc) are
> requested from the other users.
> So this os-faults seems very valuable for users.
>
> Just in my opinion, I am find to add this os-faults under QA project.
> But before that, I'd like to get opinions from the other people of QA team.
>
> Thanks
> Ken Ohmichi
>
>
>
>
>
>
>
>
>
>
>
>
>> On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov 
>> wrote:
>>>
>>> Hi Ken,
>>>
>>> OS-Faults doesn't have any scenarios in the tree yet (the project is two
>>> months old), but you can find some examples of the use in the
>>> os-faults/examples directory.
>>>
>>> Regards,
>>> Yaroslav Lobankov.
>>>
>>> On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi 
>>> wrote:

 Hi Timur,

 Thanks for your explanation.

 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov
 :
 >
 >> I am guessing the above "restart nodes" is for verifying each
 >> OpenStack service restarts successfully, right?
 >
 > Yes, this is right. And we also will check that HA logic for these
 > services works correctly (for example, rescheduling of L3 Neutron
 > agents for networks).
 >
 >> But these service scripts are provided by distributors, and Devstack
 >> itself doesn't contain service scripts IIUC.
 >> So I'd like to know how to verify it on Devstack clouds.
 >
 > Yes, DevStack doesn't support many scenarios which are actual
 > and should be supported on the production clouds.
 > It will be not possible to run all advanced test scenarios for DevStack
 > clouds,
 > just because DevStack can't deploy OpenStack cloud with 3 controllers
 > now (so, probably it will be possible in the future).
 >
 > Of course, some advanced scenarios will support DevStack clouds,
 > for example, some test scenarios which are based on customer-found
 > issues from the real production clouds, like upload of the large images
 > (100+ Gb)
 > to Glance with Swift backend. Such cases are important for verification
 > of
 > pre-production environments, but not very important for CI gate jobs.
 >
 > It is also important to note that in these advanced cases we are
 > targeting
 > to check not only the logic of Python 

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Ken'ichi Ohmichi]

Hi Timur,

2016-10-06 4:08 GMT-07:00 Timur Nurlygayanov :
> Ken, it is a good idea!
>
> The plan is to develop os-faults as a library which will be able
> to manage cluster nodes and OpenStack services on these nodes.
> It is a good idea to add some Tempest tests which will use
> os-faults library as well for some API tests.

Sorry for my misleading, that is not I wanted to say.
I am thinking Tempest doesn't use os-faults as library.
I just wanted to say some destructive tests which will use REST APIs
can be implemented in Tempest instead of os-faults.

For example, the compute service client contains disable_service which
disables nova service but Tempest doesn't contain the corresponding
test cases because Tempest tests need to run in parallel.
https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/services_client.py#L54

If some tests use this method, the other tests which run in parallel
can be failed as unexpected on the gate.
Such tests also are destructive and I am hoping these tests also will
be able to run the gate by finding better way.

> The Stepler framework [1] will use os-faults to perform all destructive
> actions in the clouds (the reboot of nodes, restart of OpenStack services,
> enable/disable network interfaces or some firewall rules and etc).
>
> We need to get +1 from you in [1] to create the repository with
> advanced end-user scenario tests.

OK, I'd like to summarize my thinking here for adding os-faults under
QA project:

Under QA project, the first target of most programs(tempest, devstack,
etc) is for the gate testing.
Of course, some of them are available on production clouds also as the
design, but the first is for the gate.
That means devstack clouds as the first target/purpose.
At this time, this os-faults doesn't seem the gate as the first
purpose based on current implementation.
So I feel os-faults seems different from the existing programs.

os-faults is very young and we will be able to extend it for devstack
clouds also later, I hope.
In addition, I heard this kind of tests(destructive, HA, etc) are
requested from the other users.
So this os-faults seems very valuable for users.

Just in my opinion, I am find to add this os-faults under QA project.
But before that, I'd like to get opinions from the other people of QA team.

Thanks
Ken Ohmichi












> On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov 
> wrote:
>>
>> Hi Ken,
>>
>> OS-Faults doesn't have any scenarios in the tree yet (the project is two
>> months old), but you can find some examples of the use in the
>> os-faults/examples directory.
>>
>> Regards,
>> Yaroslav Lobankov.
>>
>> On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi 
>> wrote:
>>>
>>> Hi Timur,
>>>
>>> Thanks for your explanation.
>>>
>>> 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov
>>> :
>>> >
>>> >> I am guessing the above "restart nodes" is for verifying each
>>> >> OpenStack service restarts successfully, right?
>>> >
>>> > Yes, this is right. And we also will check that HA logic for these
>>> > services works correctly (for example, rescheduling of L3 Neutron
>>> > agents for networks).
>>> >
>>> >> But these service scripts are provided by distributors, and Devstack
>>> >> itself doesn't contain service scripts IIUC.
>>> >> So I'd like to know how to verify it on Devstack clouds.
>>> >
>>> > Yes, DevStack doesn't support many scenarios which are actual
>>> > and should be supported on the production clouds.
>>> > It will be not possible to run all advanced test scenarios for DevStack
>>> > clouds,
>>> > just because DevStack can't deploy OpenStack cloud with 3 controllers
>>> > now (so, probably it will be possible in the future).
>>> >
>>> > Of course, some advanced scenarios will support DevStack clouds,
>>> > for example, some test scenarios which are based on customer-found
>>> > issues from the real production clouds, like upload of the large images
>>> > (100+ Gb)
>>> > to Glance with Swift backend. Such cases are important for verification
>>> > of
>>> > pre-production environments, but not very important for CI gate jobs.
>>> >
>>> > It is also important to note that in these advanced cases we are
>>> > targeting
>>> > to check not only the logic of Python code, but also the correct
>>> > configuration
>>> > of all OpenStack components on some pre-production OpenStack clusters.
>>>
>>> I guessed some part of os-faults can be moved to Tempest if os-faults
>>> contains API tests for enabling/disabling OpenStack services.
>>> Then, os-faults would be able to concentrate on more destructive tests
>>> like rebooting physical nodes, etc.
>>> However, I could not find any actual scenarios on current os-faults
>>> (https://github.com/openstack/os-faults).
>>> That seems to just contain some abstraction layers and unit tests. Can
>>> we see actual test scenarios of os-faults ?
>>> Maybe I missed something.
>>>
>>> 

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Timur Nurlygayanov]

Ken, it is a good idea!

The plan is to develop os-faults as a library which will be able
to manage cluster nodes and OpenStack services on these nodes.
It is a good idea to add some Tempest tests which will use
os-faults library as well for some API tests.

The Stepler framework [1] will use os-faults to perform all destructive
actions in the clouds (the reboot of nodes, restart of OpenStack services,
enable/disable network interfaces or some firewall rules and etc).

We need to get +1 from you in [1] to create the repository with
advanced end-user scenario tests.

Thank you!

[1] https://review.openstack.org/#/c/374667/

On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov 
wrote:

> Hi Ken,
>
> OS-Faults doesn't have any scenarios in the tree yet (the project is two
> months old), but you can find some examples of the use in the
> os-faults/examples directory.
>
> Regards,
> Yaroslav Lobankov.
>
> On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi 
> wrote:
>
>> Hi Timur,
>>
>> Thanks for your explanation.
>>
>> 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov > >:
>> >
>> >> I am guessing the above "restart nodes" is for verifying each
>> >> OpenStack service restarts successfully, right?
>> >
>> > Yes, this is right. And we also will check that HA logic for these
>> > services works correctly (for example, rescheduling of L3 Neutron
>> > agents for networks).
>> >
>> >> But these service scripts are provided by distributors, and Devstack
>> >> itself doesn't contain service scripts IIUC.
>> >> So I'd like to know how to verify it on Devstack clouds.
>> >
>> > Yes, DevStack doesn't support many scenarios which are actual
>> > and should be supported on the production clouds.
>> > It will be not possible to run all advanced test scenarios for DevStack
>> > clouds,
>> > just because DevStack can't deploy OpenStack cloud with 3 controllers
>> > now (so, probably it will be possible in the future).
>> >
>> > Of course, some advanced scenarios will support DevStack clouds,
>> > for example, some test scenarios which are based on customer-found
>> > issues from the real production clouds, like upload of the large images
>> > (100+ Gb)
>> > to Glance with Swift backend. Such cases are important for verification
>> of
>> > pre-production environments, but not very important for CI gate jobs.
>> >
>> > It is also important to note that in these advanced cases we are
>> targeting
>> > to check not only the logic of Python code, but also the correct
>> > configuration
>> > of all OpenStack components on some pre-production OpenStack clusters.
>>
>> I guessed some part of os-faults can be moved to Tempest if os-faults
>> contains API tests for enabling/disabling OpenStack services.
>> Then, os-faults would be able to concentrate on more destructive tests
>> like rebooting physical nodes, etc.
>> However, I could not find any actual scenarios on current os-faults
>> (https://github.com/openstack/os-faults).
>> That seems to just contain some abstraction layers and unit tests. Can
>> we see actual test scenarios of os-faults ?
>> Maybe I missed something.
>>
>> Thanks
>> Ken Ohmichi
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

Timur,
Senior QA Manager
OpenStack Projects
Mirantis Inc
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Yaroslav Lobankov]

Hi Ken,

OS-Faults doesn't have any scenarios in the tree yet (the project is two
months old), but you can find some examples of the use in the
os-faults/examples directory.

Regards,
Yaroslav Lobankov.

On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi 
wrote:

> Hi Timur,
>
> Thanks for your explanation.
>
> 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov :
> >
> >> I am guessing the above "restart nodes" is for verifying each
> >> OpenStack service restarts successfully, right?
> >
> > Yes, this is right. And we also will check that HA logic for these
> > services works correctly (for example, rescheduling of L3 Neutron
> > agents for networks).
> >
> >> But these service scripts are provided by distributors, and Devstack
> >> itself doesn't contain service scripts IIUC.
> >> So I'd like to know how to verify it on Devstack clouds.
> >
> > Yes, DevStack doesn't support many scenarios which are actual
> > and should be supported on the production clouds.
> > It will be not possible to run all advanced test scenarios for DevStack
> > clouds,
> > just because DevStack can't deploy OpenStack cloud with 3 controllers
> > now (so, probably it will be possible in the future).
> >
> > Of course, some advanced scenarios will support DevStack clouds,
> > for example, some test scenarios which are based on customer-found
> > issues from the real production clouds, like upload of the large images
> > (100+ Gb)
> > to Glance with Swift backend. Such cases are important for verification
> of
> > pre-production environments, but not very important for CI gate jobs.
> >
> > It is also important to note that in these advanced cases we are
> targeting
> > to check not only the logic of Python code, but also the correct
> > configuration
> > of all OpenStack components on some pre-production OpenStack clusters.
>
> I guessed some part of os-faults can be moved to Tempest if os-faults
> contains API tests for enabling/disabling OpenStack services.
> Then, os-faults would be able to concentrate on more destructive tests
> like rebooting physical nodes, etc.
> However, I could not find any actual scenarios on current os-faults
> (https://github.com/openstack/os-faults).
> That seems to just contain some abstraction layers and unit tests. Can
> we see actual test scenarios of os-faults ?
> Maybe I missed something.
>
> Thanks
> Ken Ohmichi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Ken'ichi Ohmichi]

Hi Timur,

Thanks for your explanation.

2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov :
>
>> I am guessing the above "restart nodes" is for verifying each
>> OpenStack service restarts successfully, right?
>
> Yes, this is right. And we also will check that HA logic for these
> services works correctly (for example, rescheduling of L3 Neutron
> agents for networks).
>
>> But these service scripts are provided by distributors, and Devstack
>> itself doesn't contain service scripts IIUC.
>> So I'd like to know how to verify it on Devstack clouds.
>
> Yes, DevStack doesn't support many scenarios which are actual
> and should be supported on the production clouds.
> It will be not possible to run all advanced test scenarios for DevStack
> clouds,
> just because DevStack can't deploy OpenStack cloud with 3 controllers
> now (so, probably it will be possible in the future).
>
> Of course, some advanced scenarios will support DevStack clouds,
> for example, some test scenarios which are based on customer-found
> issues from the real production clouds, like upload of the large images
> (100+ Gb)
> to Glance with Swift backend. Such cases are important for verification of
> pre-production environments, but not very important for CI gate jobs.
>
> It is also important to note that in these advanced cases we are targeting
> to check not only the logic of Python code, but also the correct
> configuration
> of all OpenStack components on some pre-production OpenStack clusters.

I guessed some part of os-faults can be moved to Tempest if os-faults
contains API tests for enabling/disabling OpenStack services.
Then, os-faults would be able to concentrate on more destructive tests
like rebooting physical nodes, etc.
However, I could not find any actual scenarios on current os-faults
(https://github.com/openstack/os-faults).
That seems to just contain some abstraction layers and unit tests. Can
we see actual test scenarios of os-faults ?
Maybe I missed something.

Thanks
Ken Ohmichi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Timur Nurlygayanov]

Hi Ken,

I am guessing the above "restart nodes" is for verifying each
> OpenStack service restarts successfully, right?

Yes, this is right. And we also will check that HA logic for these
services works correctly (for example, rescheduling of L3 Neutron
agents for networks).

But these service scripts are provided by distributors, and Devstack
> itself doesn't contain service scripts IIUC.
> So I'd like to know how to verify it on Devstack clouds.

Yes, DevStack doesn't support many scenarios which are actual
and should be supported on the production clouds.
It will be not possible to run all advanced test scenarios for DevStack
clouds,
just because DevStack can't deploy OpenStack cloud with 3 controllers
now (so, probably it will be possible in the future).

Of course, some advanced scenarios will support DevStack clouds,
for example, some test scenarios which are based on customer-found
issues from the real production clouds, like upload of the large images
(100+ Gb)
to Glance with Swift backend. Such cases are important for verification of
pre-production environments, but not very important for CI gate jobs.

It is also important to note that in these advanced cases we are targeting
to check not only the logic of Python code, but also the correct
configuration
of all OpenStack components on some pre-production OpenStack clusters.

Thank you!

On Wed, Sep 28, 2016 at 2:37 AM, Ken'ichi Ohmichi 
wrote:

> Hi Timur,
>
> Thanks for picking this up, that is interesting for me.
>
> 2016-09-22 5:58 GMT-07:00 Timur Nurlygayanov :
> >
> > we have an idea to create the test suite with destructive/HA and advanced
> > end-user scenarios for the OpenStack clusters. This test suite will
> contains
> > advanced scenario integration tests for OpenStack clusters to make sure
> that
> > the cluster is ready for the production.
> >
> > The test cases which we want to cover in this test suite:
> > 1) All simple and advanced destructive actions, like a reboot of the
> nodes,
> > restart OpenStack services and etc. (we can probably use of-faults
> library
> > [1], which we already use in Rally)
> > 2) All advanced test scenarios like a migration of the bunch of VMs
> between
> > nodes and booting of the VMs with large images (10+ Gb), send traffic
> > between VMs and in parallel restart Neutron l3 agents and etc.
> >
> > The key requirements:
> > 1) The framework should know the details of the deployments (how many
> nodes
> > we have, how to ssh to OpenStack nodes, how to restart the nodes and
> etc.).
> > This is why we don't want to add such "advanced" and HA-focused test
> > scenarios to Tempest.
>
> Yeah, this point is right. This "advanced" way is different from the
> design principle of Tempest[1].
> I am guessing the above "restart nodes" is for verifying each
> OpenStack service restarts successfully, right?
> For productions(or distributions), this verifying point seems
> important because service scripts need to restart OpenStack services
> automatically.
> But these service scripts are provided by distributors, and Devstack
> itself doesn't contain service scripts IIUC.
> So I'd like to know how to verify it on Devstack clouds.
>
> Thanks
> Ken Ohmichi
> ---
>
> [1]: https://github.com/openstack/tempest#design-principles
>
> > 2) We should be ready to run these tests for any clouds: DevStack clouds
> (we
> > can skip HA cases for DevStack), Fuel clouds, clouds which were deployed
> by
> > Ansible or Puppet tools.
> > 3) This framework should allow reproduce the issue in a repeatable
> manner,
> > this is why we can't just cover all the tests with Rally load tests +
> > destructive plugins (we are working on this right now too to have an
> ability
> > to test HA-related scenarios under the load).
> >
> > As we discussed on the OpenStack summit a year ago it is better to move
> such
> > test suite in a separate repository and this framework can became a part
> of
> > the QA (or at least BigTent) program in OpenStack.
> >
> > I've created the commit to OpenStack project-config repository:
> > https://review.openstack.org/#/c/374667/
> >
> > Could you please take a look?
> >
> > We understand that it will be hard to add such test suite to the gates
> for
> > every commit in OpenStack because we will need a lot of hardware. We
> don't
> > want to add these tests to the per-commit gates for now, it is ok to run
> > them just once a day, for example. And we definitely need to have such
> test
> > suite to validate our own pre-production clouds.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 

Timur,
Senior QA Manager
OpenStack Projects
Mirantis Inc
__

Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters

[Ken'ichi Ohmichi]

Hi Timur,

Thanks for picking this up, that is interesting for me.

2016-09-22 5:58 GMT-07:00 Timur Nurlygayanov :
>
> we have an idea to create the test suite with destructive/HA and advanced
> end-user scenarios for the OpenStack clusters. This test suite will contains
> advanced scenario integration tests for OpenStack clusters to make sure that
> the cluster is ready for the production.
>
> The test cases which we want to cover in this test suite:
> 1) All simple and advanced destructive actions, like a reboot of the nodes,
> restart OpenStack services and etc. (we can probably use of-faults library
> [1], which we already use in Rally)
> 2) All advanced test scenarios like a migration of the bunch of VMs between
> nodes and booting of the VMs with large images (10+ Gb), send traffic
> between VMs and in parallel restart Neutron l3 agents and etc.
>
> The key requirements:
> 1) The framework should know the details of the deployments (how many nodes
> we have, how to ssh to OpenStack nodes, how to restart the nodes and etc.).
> This is why we don't want to add such "advanced" and HA-focused test
> scenarios to Tempest.

Yeah, this point is right. This "advanced" way is different from the
design principle of Tempest[1].
I am guessing the above "restart nodes" is for verifying each
OpenStack service restarts successfully, right?
For productions(or distributions), this verifying point seems
important because service scripts need to restart OpenStack services
automatically.
But these service scripts are provided by distributors, and Devstack
itself doesn't contain service scripts IIUC.
So I'd like to know how to verify it on Devstack clouds.

Thanks
Ken Ohmichi
---

[1]: https://github.com/openstack/tempest#design-principles

> 2) We should be ready to run these tests for any clouds: DevStack clouds (we
> can skip HA cases for DevStack), Fuel clouds, clouds which were deployed by
> Ansible or Puppet tools.
> 3) This framework should allow reproduce the issue in a repeatable manner,
> this is why we can't just cover all the tests with Rally load tests +
> destructive plugins (we are working on this right now too to have an ability
> to test HA-related scenarios under the load).
>
> As we discussed on the OpenStack summit a year ago it is better to move such
> test suite in a separate repository and this framework can became a part of
> the QA (or at least BigTent) program in OpenStack.
>
> I've created the commit to OpenStack project-config repository:
> https://review.openstack.org/#/c/374667/
>
> Could you please take a look?
>
> We understand that it will be hard to add such test suite to the gates for
> every commit in OpenStack because we will need a lot of hardware. We don't
> want to add these tests to the per-commit gates for now, it is ok to run
> them just once a day, for example. And we definitely need to have such test
> suite to validate our own pre-production clouds.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev