Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi Dims, Hi gmann, Thanks guys. Slowly recovering from PTG jet lag OK then.., I will start working on this. I would also like to coordinate this work with LCOO [1] team, where we can get valuable feedback and contributions.. I will give you update on next QA IRC on 0900 UTC meeting, which is 9th of March? (1700 UTC meeting is difficult for me, however I will definitely be there if needed) [1] https://wiki.openstack.org/wiki/LCOO --- Regards, Sampath On Fri, Feb 24, 2017 at 8:57 PM, Ghanshyam Mann wrote: > Yea, agree with dims. > > Sampath , > > Thanks for taking over this, it is really great help. Please update the > current spec with approaches you have. Timur help will be great if he show > up sometime. > > Also we will add destructive testing as one of weekly meeting agenda and > make sure you will get all help & required attention from QA team. > > -gmann > > > On Fri, Feb 24, 2017 at 7:26 AM, Davanum Srinivas wrote: >> >> Sampath, >> >> I am not sure if you will hear back from Timur soon as he may not be >> working on this stuff anymore (in Mirantis). So i'd recommend picking >> up the work if you don't hear from him soon. >> >> Thanks, >> Dims >> >> On Thu, Feb 23, 2017 at 3:41 PM, Sam P wrote: >> > Hi Timur, >> > >> > The current status of this work is, >> > 1) The QA user story for destructive testing of OpenStack cloud [1] is >> > merged . >> > 2) The spec for a new framework which will focus on HA/failover and >> > destructive testing [2] has no update since Nov 30 2016. >> > 3) The commit for the new repository [3] has abandoned due to no >> > update after Nov 29 2016. >> > >> > Currently, I am working on 3rd party destructive/HA testing CI for >> > Masakari[4] and very much interested in this work. >> > I will keep working on [1] with PWG. >> > Please let me know your plans for [2], and [3]. >> > If it is difficult for you to continue, I would love to continue your >> > work on [2], and [3]. >> > >> > [1] https://review.openstack.org/#/c/396142 >> > [2] https://review.openstack.org/#/c/399618 >> > [3] https://review.openstack.org/#/c/374667 >> > [4] https://wiki.openstack.org/wiki/Masakari >> > --- Regards, >> > Sampath >> > >> > >> > >> > On Mon, Nov 28, 2016 at 6:37 AM, Timur Nurlygayanov >> > wrote: >> >> Hi team, >> >> >> >> here is a short update: >> >> >> >> 1) The QA user story for destructive testing of OpenStack cloud is on >> >> review >> >> [1]. >> >> 2) The spec for a new framework which will focus on HA/failover and >> >> destructive testing is no review [2]. >> >> 3) The commit for the new repository is on review [3] as well. >> >> >> >> [1] https://review.openstack.org/#/c/396142 >> >> [2] https://review.openstack.org/#/c/399618 >> >> [3] https://review.openstack.org/#/c/374667 >> >> >> >> >> >> On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann >> >> wrote: >> >>> >> >>> I like os-faults library which can provide the abstraction over >> >>> different >> >>> destructive actions like reboot/poweroff node etc. >> >>> >> >>> >> >>> >> >>> But not much clear about Stepler that what all tests it will contain. >> >>> Tempest do have scenario tests which can hit the production env with >> >>> significant way of testing scenario. >> >>> >> >>> It can cover the end user scenario also which involves the interaction >> >>> of >> >>> public OpenStack APIs and always in enhancement state by adding more >> >>> and >> >>> more scenario tests. >> >>> >> >>> >> >>> >> >>> Few query over Stepler as separate project: >> >>> >> >>> 1. Is Stepler will cover only tests which hits the node level >> >>> action(reboot, HA etc)? >> >>> >> >>> 2. This should not mix the scenario tests which are in Tempest >> >>> scope >> >>> because that can make confusion for developers (where to add scenario >> >>> tests) >> >>> as well as for tester(from where to run what scenario tests). >> >>> >> >>> 3. How we make sure those tests run fi
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Yea, agree with dims. Sampath , Thanks for taking over this, it is really great help. Please update the current spec with approaches you have. Timur help will be great if he show up sometime. Also we will add destructive testing as one of weekly meeting agenda and make sure you will get all help & required attention from QA team. -gmann On Fri, Feb 24, 2017 at 7:26 AM, Davanum Srinivas wrote: > Sampath, > > I am not sure if you will hear back from Timur soon as he may not be > working on this stuff anymore (in Mirantis). So i'd recommend picking > up the work if you don't hear from him soon. > > Thanks, > Dims > > On Thu, Feb 23, 2017 at 3:41 PM, Sam P wrote: > > Hi Timur, > > > > The current status of this work is, > > 1) The QA user story for destructive testing of OpenStack cloud [1] is > merged . > > 2) The spec for a new framework which will focus on HA/failover and > > destructive testing [2] has no update since Nov 30 2016. > > 3) The commit for the new repository [3] has abandoned due to no > > update after Nov 29 2016. > > > > Currently, I am working on 3rd party destructive/HA testing CI for > > Masakari[4] and very much interested in this work. > > I will keep working on [1] with PWG. > > Please let me know your plans for [2], and [3]. > > If it is difficult for you to continue, I would love to continue your > > work on [2], and [3]. > > > > [1] https://review.openstack.org/#/c/396142 > > [2] https://review.openstack.org/#/c/399618 > > [3] https://review.openstack.org/#/c/374667 > > [4] https://wiki.openstack.org/wiki/Masakari > > --- Regards, > > Sampath > > > > > > > > On Mon, Nov 28, 2016 at 6:37 AM, Timur Nurlygayanov > > wrote: > >> Hi team, > >> > >> here is a short update: > >> > >> 1) The QA user story for destructive testing of OpenStack cloud is on > review > >> [1]. > >> 2) The spec for a new framework which will focus on HA/failover and > >> destructive testing is no review [2]. > >> 3) The commit for the new repository is on review [3] as well. > >> > >> [1] https://review.openstack.org/#/c/396142 > >> [2] https://review.openstack.org/#/c/399618 > >> [3] https://review.openstack.org/#/c/374667 > >> > >> > >> On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann > >> wrote: > >>> > >>> I like os-faults library which can provide the abstraction over > different > >>> destructive actions like reboot/poweroff node etc. > >>> > >>> > >>> > >>> But not much clear about Stepler that what all tests it will contain. > >>> Tempest do have scenario tests which can hit the production env with > >>> significant way of testing scenario. > >>> > >>> It can cover the end user scenario also which involves the interaction > of > >>> public OpenStack APIs and always in enhancement state by adding more > and > >>> more scenario tests. > >>> > >>> > >>> > >>> Few query over Stepler as separate project: > >>> > >>> 1. Is Stepler will cover only tests which hits the node level > >>> action(reboot, HA etc)? > >>> > >>> 2. This should not mix the scenario tests which are in Tempest > scope > >>> because that can make confusion for developers (where to add scenario > tests) > >>> as well as for tester(from where to run what scenario tests). > >>> > >>> 3. How we make sure those tests run fine or at least run while > >>> adding. > >>> > >>> a. I think devstack enhancement for multi-node etc can do this as > >>> mentioned by you also. > >>> > >>> b. If so then why not adding those tests in Tempest only using > >>> os-faults lib ? > >>> > >>> > >>> > >>> Overall I feel os-faults as lib is really nice idea but tests scope > can > >>> go in Tempest under HW_scenario (or something else) till it does not > break > >>> basic principle of Tempest. > >>> > >>> > >>> > >>> Thanks > >>> > >>> gmann > >>> > >>> > >>> > >>> From: Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com] > >>> Sent: 06 October 2016 20:09 > >>> To: OpenStack Development Mailing List (not for usage questions) > >>>
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Sampath, I am not sure if you will hear back from Timur soon as he may not be working on this stuff anymore (in Mirantis). So i'd recommend picking up the work if you don't hear from him soon. Thanks, Dims On Thu, Feb 23, 2017 at 3:41 PM, Sam P wrote: > Hi Timur, > > The current status of this work is, > 1) The QA user story for destructive testing of OpenStack cloud [1] is merged > . > 2) The spec for a new framework which will focus on HA/failover and > destructive testing [2] has no update since Nov 30 2016. > 3) The commit for the new repository [3] has abandoned due to no > update after Nov 29 2016. > > Currently, I am working on 3rd party destructive/HA testing CI for > Masakari[4] and very much interested in this work. > I will keep working on [1] with PWG. > Please let me know your plans for [2], and [3]. > If it is difficult for you to continue, I would love to continue your > work on [2], and [3]. > > [1] https://review.openstack.org/#/c/396142 > [2] https://review.openstack.org/#/c/399618 > [3] https://review.openstack.org/#/c/374667 > [4] https://wiki.openstack.org/wiki/Masakari > --- Regards, > Sampath > > > > On Mon, Nov 28, 2016 at 6:37 AM, Timur Nurlygayanov > wrote: >> Hi team, >> >> here is a short update: >> >> 1) The QA user story for destructive testing of OpenStack cloud is on review >> [1]. >> 2) The spec for a new framework which will focus on HA/failover and >> destructive testing is no review [2]. >> 3) The commit for the new repository is on review [3] as well. >> >> [1] https://review.openstack.org/#/c/396142 >> [2] https://review.openstack.org/#/c/399618 >> [3] https://review.openstack.org/#/c/374667 >> >> >> On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann >> wrote: >>> >>> I like os-faults library which can provide the abstraction over different >>> destructive actions like reboot/poweroff node etc. >>> >>> >>> >>> But not much clear about Stepler that what all tests it will contain. >>> Tempest do have scenario tests which can hit the production env with >>> significant way of testing scenario. >>> >>> It can cover the end user scenario also which involves the interaction of >>> public OpenStack APIs and always in enhancement state by adding more and >>> more scenario tests. >>> >>> >>> >>> Few query over Stepler as separate project: >>> >>> 1. Is Stepler will cover only tests which hits the node level >>> action(reboot, HA etc)? >>> >>> 2. This should not mix the scenario tests which are in Tempest scope >>> because that can make confusion for developers (where to add scenario tests) >>> as well as for tester(from where to run what scenario tests). >>> >>> 3. How we make sure those tests run fine or at least run while >>> adding. >>> >>> a. I think devstack enhancement for multi-node etc can do this as >>> mentioned by you also. >>> >>> b. If so then why not adding those tests in Tempest only using >>> os-faults lib ? >>> >>> >>> >>> Overall I feel os-faults as lib is really nice idea but tests scope can >>> go in Tempest under HW_scenario (or something else) till it does not break >>> basic principle of Tempest. >>> >>> >>> >>> Thanks >>> >>> gmann >>> >>> >>> >>> From: Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com] >>> Sent: 06 October 2016 20:09 >>> To: OpenStack Development Mailing List (not for usage questions) >>> >>> Subject: Re: [openstack-dev] [QA] The end-user test suite for OpenStack >>> clusters >>> >>> >>> >>> Ken, it is a good idea! >>> >>> >>> >>> The plan is to develop os-faults as a library which will be able >>> >>> to manage cluster nodes and OpenStack services on these nodes. >>> >>> It is a good idea to add some Tempest tests which will use >>> >>> os-faults library as well for some API tests. >>> >>> >>> >>> The Stepler framework [1] will use os-faults to perform all destructive >>> >>> actions in the clouds (the reboot of nodes, restart of OpenStack services, >>> >>> enable/disable network interfaces or some firewall rules and etc). >>> >>> >>> >>> We need to get +1 from you in [1] to create the reposito
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi Timur, The current status of this work is, 1) The QA user story for destructive testing of OpenStack cloud [1] is merged . 2) The spec for a new framework which will focus on HA/failover and destructive testing [2] has no update since Nov 30 2016. 3) The commit for the new repository [3] has abandoned due to no update after Nov 29 2016. Currently, I am working on 3rd party destructive/HA testing CI for Masakari[4] and very much interested in this work. I will keep working on [1] with PWG. Please let me know your plans for [2], and [3]. If it is difficult for you to continue, I would love to continue your work on [2], and [3]. [1] https://review.openstack.org/#/c/396142 [2] https://review.openstack.org/#/c/399618 [3] https://review.openstack.org/#/c/374667 [4] https://wiki.openstack.org/wiki/Masakari --- Regards, Sampath On Mon, Nov 28, 2016 at 6:37 AM, Timur Nurlygayanov wrote: > Hi team, > > here is a short update: > > 1) The QA user story for destructive testing of OpenStack cloud is on review > [1]. > 2) The spec for a new framework which will focus on HA/failover and > destructive testing is no review [2]. > 3) The commit for the new repository is on review [3] as well. > > [1] https://review.openstack.org/#/c/396142 > [2] https://review.openstack.org/#/c/399618 > [3] https://review.openstack.org/#/c/374667 > > > On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann > wrote: >> >> I like os-faults library which can provide the abstraction over different >> destructive actions like reboot/poweroff node etc. >> >> >> >> But not much clear about Stepler that what all tests it will contain. >> Tempest do have scenario tests which can hit the production env with >> significant way of testing scenario. >> >> It can cover the end user scenario also which involves the interaction of >> public OpenStack APIs and always in enhancement state by adding more and >> more scenario tests. >> >> >> >> Few query over Stepler as separate project: >> >> 1. Is Stepler will cover only tests which hits the node level >> action(reboot, HA etc)? >> >> 2. This should not mix the scenario tests which are in Tempest scope >> because that can make confusion for developers (where to add scenario tests) >> as well as for tester(from where to run what scenario tests). >> >> 3. How we make sure those tests run fine or at least run while >> adding. >> >> a. I think devstack enhancement for multi-node etc can do this as >> mentioned by you also. >> >> b. If so then why not adding those tests in Tempest only using >> os-faults lib ? >> >> >> >> Overall I feel os-faults as lib is really nice idea but tests scope can >> go in Tempest under HW_scenario (or something else) till it does not break >> basic principle of Tempest. >> >> >> >> Thanks >> >> gmann >> >> >> >> From: Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com] >> Sent: 06 October 2016 20:09 >> To: OpenStack Development Mailing List (not for usage questions) >> >> Subject: Re: [openstack-dev] [QA] The end-user test suite for OpenStack >> clusters >> >> >> >> Ken, it is a good idea! >> >> >> >> The plan is to develop os-faults as a library which will be able >> >> to manage cluster nodes and OpenStack services on these nodes. >> >> It is a good idea to add some Tempest tests which will use >> >> os-faults library as well for some API tests. >> >> >> >> The Stepler framework [1] will use os-faults to perform all destructive >> >> actions in the clouds (the reboot of nodes, restart of OpenStack services, >> >> enable/disable network interfaces or some firewall rules and etc). >> >> >> >> We need to get +1 from you in [1] to create the repository with >> >> advanced end-user scenario tests. >> >> >> >> Thank you! >> >> >> >> [1] https://review.openstack.org/#/c/374667/ >> >> >> >> On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov >> wrote: >> >> Hi Ken, >> >> >> >> OS-Faults doesn't have any scenarios in the tree yet (the project is two >> months old), but you can find some examples of the use in the >> os-faults/examples directory. >> >> >> >> Regards, >> >> Yaroslav Lobankov. >> >> >> >> On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi >> wrote: >> >> Hi Timur, >> >> Thanks
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi team, here is a short update: 1) The QA user story for destructive testing of OpenStack cloud is on review [1]. 2) The spec for a new framework which will focus on HA/failover and destructive testing is no review [2]. 3) The commit for the new repository is on review [3] as well. [1] https://review.openstack.org/#/c/396142 [2] https://review.openstack.org/#/c/399618 [3] https://review.openstack.org/#/c/374667 On Fri, Oct 14, 2016 at 3:47 AM, Ghanshyam Mann < ghanshyam.m...@nectechnologies.in> wrote: > I like os-faults library which can provide the abstraction over different > destructive actions like reboot/poweroff node etc. > > > > But not much clear about Stepler that what all tests it will contain. > Tempest do have scenario tests which can hit the production env with > significant way of testing scenario. > > It can cover the end user scenario also which involves the interaction of > public OpenStack APIs and always in enhancement state by adding more and > more scenario tests. > > > > Few query over Stepler as separate project: > > 1. Is Stepler will cover only tests which hits the node level > action(reboot, HA etc)? > > 2. This should not mix the scenario tests which are in Tempest > scope because that can make confusion for developers (where to add scenario > tests) as well as for tester(from where to run what scenario tests). > > 3. How we make sure those tests run fine or at least run while > adding. > > a. I think devstack enhancement for multi-node etc can do this as > mentioned by you also. > > b. If so then why not adding those tests in Tempest only using > os-faults lib ? > > > > Overall I feel os-faults as lib is really nice idea but tests scope can > go in Tempest under HW_scenario (or something else) till it does not break > basic principle of Tempest. > > > > Thanks > > gmann > > > > *From:* Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com] > *Sent:* 06 October 2016 20:09 > *To:* OpenStack Development Mailing List (not for usage questions) < > openstack-dev@lists.openstack.org> > *Subject:* Re: [openstack-dev] [QA] The end-user test suite for OpenStack > clusters > > > > Ken, it is a good idea! > > > > The plan is to develop os-faults as a library which will be able > > to manage cluster nodes and OpenStack services on these nodes. > > It is a good idea to add some Tempest tests which will use > > os-faults library as well for some API tests. > > > > The Stepler framework [1] will use os-faults to perform all destructive > > actions in the clouds (the reboot of nodes, restart of OpenStack services, > > enable/disable network interfaces or some firewall rules and etc). > > > > We need to get +1 from you in [1] to create the repository with > > advanced end-user scenario tests. > > > > Thank you! > > > > [1] https://review.openstack.org/#/c/374667/ > > > > On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov > wrote: > > Hi Ken, > > > > OS-Faults doesn't have any scenarios in the tree yet (the project is two > months old), but you can find some examples of the use in the > os-faults/examples directory. > > > > Regards, > > Yaroslav Lobankov. > > > > On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi > wrote: > > Hi Timur, > > Thanks for your explanation. > > > 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov : > > > >> I am guessing the above "restart nodes" is for verifying each > >> OpenStack service restarts successfully, right? > > > > Yes, this is right. And we also will check that HA logic for these > > services works correctly (for example, rescheduling of L3 Neutron > > agents for networks). > > > >> But these service scripts are provided by distributors, and Devstack > >> itself doesn't contain service scripts IIUC. > >> So I'd like to know how to verify it on Devstack clouds. > > > > Yes, DevStack doesn't support many scenarios which are actual > > and should be supported on the production clouds. > > It will be not possible to run all advanced test scenarios for DevStack > > clouds, > > just because DevStack can't deploy OpenStack cloud with 3 controllers > > now (so, probably it will be possible in the future). > > > > Of course, some advanced scenarios will support DevStack clouds, > > for example, some test scenarios which are based on customer-found > > issues from the real production clouds, like upload of the large images > > (100+ Gb) > > to Glance with Swift backend. S
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
I like os-faults library which can provide the abstraction over different destructive actions like reboot/poweroff node etc. But not much clear about Stepler that what all tests it will contain. Tempest do have scenario tests which can hit the production env with significant way of testing scenario. It can cover the end user scenario also which involves the interaction of public OpenStack APIs and always in enhancement state by adding more and more scenario tests. Few query over Stepler as separate project: 1. Is Stepler will cover only tests which hits the node level action(reboot, HA etc)? 2. This should not mix the scenario tests which are in Tempest scope because that can make confusion for developers (where to add scenario tests) as well as for tester(from where to run what scenario tests). 3. How we make sure those tests run fine or at least run while adding. a. I think devstack enhancement for multi-node etc can do this as mentioned by you also. b. If so then why not adding those tests in Tempest only using os-faults lib ? Overall I feel os-faults as lib is really nice idea but tests scope can go in Tempest under HW_scenario (or something else) till it does not break basic principle of Tempest. Thanks gmann From: Timur Nurlygayanov [mailto:tnurlygaya...@mirantis.com] Sent: 06 October 2016 20:09 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters Ken, it is a good idea! The plan is to develop os-faults as a library which will be able to manage cluster nodes and OpenStack services on these nodes. It is a good idea to add some Tempest tests which will use os-faults library as well for some API tests. The Stepler framework [1] will use os-faults to perform all destructive actions in the clouds (the reboot of nodes, restart of OpenStack services, enable/disable network interfaces or some firewall rules and etc). We need to get +1 from you in [1] to create the repository with advanced end-user scenario tests. Thank you! [1] https://review.openstack.org/#/c/374667/ On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov mailto:yloban...@mirantis.com>> wrote: Hi Ken, OS-Faults doesn't have any scenarios in the tree yet (the project is two months old), but you can find some examples of the use in the os-faults/examples directory. Regards, Yaroslav Lobankov. On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi mailto:ken1ohmi...@gmail.com>> wrote: Hi Timur, Thanks for your explanation. 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov mailto:tnurlygaya...@mirantis.com>>: > >> I am guessing the above "restart nodes" is for verifying each >> OpenStack service restarts successfully, right? > > Yes, this is right. And we also will check that HA logic for these > services works correctly (for example, rescheduling of L3 Neutron > agents for networks). > >> But these service scripts are provided by distributors, and Devstack >> itself doesn't contain service scripts IIUC. >> So I'd like to know how to verify it on Devstack clouds. > > Yes, DevStack doesn't support many scenarios which are actual > and should be supported on the production clouds. > It will be not possible to run all advanced test scenarios for DevStack > clouds, > just because DevStack can't deploy OpenStack cloud with 3 controllers > now (so, probably it will be possible in the future). > > Of course, some advanced scenarios will support DevStack clouds, > for example, some test scenarios which are based on customer-found > issues from the real production clouds, like upload of the large images > (100+ Gb) > to Glance with Swift backend. Such cases are important for verification of > pre-production environments, but not very important for CI gate jobs. > > It is also important to note that in these advanced cases we are targeting > to check not only the logic of Python code, but also the correct > configuration > of all OpenStack components on some pre-production OpenStack clusters. I guessed some part of os-faults can be moved to Tempest if os-faults contains API tests for enabling/disabling OpenStack services. Then, os-faults would be able to concentrate on more destructive tests like rebooting physical nodes, etc. However, I could not find any actual scenarios on current os-faults (https://github.com/openstack/os-faults). That seems to just contain some abstraction layers and unit tests. Can we see actual test scenarios of os-faults ? Maybe I missed something. Thanks Ken Ohmichi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openst
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi Timur, I discussed this on irc, and it seems hard to get feedback from many people because now the scope and implementation of this os-faults are separated into multiple pieces(like multiple mails, github code, gerrit etc). So how about proposing this as qa-spec? If doing that, We can discuss this on clear document and it is clear to get a consensus about adding this. Thanks Ken Ohmichi --- . 2016-10-07 12:43 GMT-07:00 Ken'ichi Ohmichi : > Hi Timur, > > 2016-10-06 4:08 GMT-07:00 Timur Nurlygayanov : >> Ken, it is a good idea! >> >> The plan is to develop os-faults as a library which will be able >> to manage cluster nodes and OpenStack services on these nodes. >> It is a good idea to add some Tempest tests which will use >> os-faults library as well for some API tests. > > Sorry for my misleading, that is not I wanted to say. > I am thinking Tempest doesn't use os-faults as library. > I just wanted to say some destructive tests which will use REST APIs > can be implemented in Tempest instead of os-faults. > > For example, the compute service client contains disable_service which > disables nova service but Tempest doesn't contain the corresponding > test cases because Tempest tests need to run in parallel. > https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/services_client.py#L54 > > If some tests use this method, the other tests which run in parallel > can be failed as unexpected on the gate. > Such tests also are destructive and I am hoping these tests also will > be able to run the gate by finding better way. > >> The Stepler framework [1] will use os-faults to perform all destructive >> actions in the clouds (the reboot of nodes, restart of OpenStack services, >> enable/disable network interfaces or some firewall rules and etc). >> >> We need to get +1 from you in [1] to create the repository with >> advanced end-user scenario tests. > > OK, I'd like to summarize my thinking here for adding os-faults under > QA project: > > Under QA project, the first target of most programs(tempest, devstack, > etc) is for the gate testing. > Of course, some of them are available on production clouds also as the > design, but the first is for the gate. > That means devstack clouds as the first target/purpose. > At this time, this os-faults doesn't seem the gate as the first > purpose based on current implementation. > So I feel os-faults seems different from the existing programs. > > os-faults is very young and we will be able to extend it for devstack > clouds also later, I hope. > In addition, I heard this kind of tests(destructive, HA, etc) are > requested from the other users. > So this os-faults seems very valuable for users. > > Just in my opinion, I am find to add this os-faults under QA project. > But before that, I'd like to get opinions from the other people of QA team. > > Thanks > Ken Ohmichi > > > > > > > > > > > > >> On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov >> wrote: >>> >>> Hi Ken, >>> >>> OS-Faults doesn't have any scenarios in the tree yet (the project is two >>> months old), but you can find some examples of the use in the >>> os-faults/examples directory. >>> >>> Regards, >>> Yaroslav Lobankov. >>> >>> On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi >>> wrote: Hi Timur, Thanks for your explanation. 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov : > >> I am guessing the above "restart nodes" is for verifying each >> OpenStack service restarts successfully, right? > > Yes, this is right. And we also will check that HA logic for these > services works correctly (for example, rescheduling of L3 Neutron > agents for networks). > >> But these service scripts are provided by distributors, and Devstack >> itself doesn't contain service scripts IIUC. >> So I'd like to know how to verify it on Devstack clouds. > > Yes, DevStack doesn't support many scenarios which are actual > and should be supported on the production clouds. > It will be not possible to run all advanced test scenarios for DevStack > clouds, > just because DevStack can't deploy OpenStack cloud with 3 controllers > now (so, probably it will be possible in the future). > > Of course, some advanced scenarios will support DevStack clouds, > for example, some test scenarios which are based on customer-found > issues from the real production clouds, like upload of the large images > (100+ Gb) > to Glance with Swift backend. Such cases are important for verification > of > pre-production environments, but not very important for CI gate jobs. > > It is also important to note that in these advanced cases we are > targeting > to check not only the logic of Python code, but also the correct > configuration > of all OpenStack components on some pre-production OpenStack clusters. I guessed some part of os-fa
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi, My first impression is "It's interesting" :) I actually don't read whole of this threads, but, I re-read the QA mission statement[1]. I feel os-faults is a little bit far from the mission statement. Because os-faults seems like focusing on operator testing. But I think this can be under the QA project. Because tempest scope has also operator testing. [1] https://wiki.openstack.org/wiki/QA#Project_Team_Definition On Sat, Oct 8, 2016 at 4:43 AM, Ken'ichi Ohmichi wrote: > Hi Timur, > > 2016-10-06 4:08 GMT-07:00 Timur Nurlygayanov : >> Ken, it is a good idea! >> >> The plan is to develop os-faults as a library which will be able >> to manage cluster nodes and OpenStack services on these nodes. >> It is a good idea to add some Tempest tests which will use >> os-faults library as well for some API tests. > > Sorry for my misleading, that is not I wanted to say. > I am thinking Tempest doesn't use os-faults as library. > I just wanted to say some destructive tests which will use REST APIs > can be implemented in Tempest instead of os-faults. > > For example, the compute service client contains disable_service which > disables nova service but Tempest doesn't contain the corresponding > test cases because Tempest tests need to run in parallel. > https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/services_client.py#L54 > > If some tests use this method, the other tests which run in parallel > can be failed as unexpected on the gate. > Such tests also are destructive and I am hoping these tests also will > be able to run the gate by finding better way. > >> The Stepler framework [1] will use os-faults to perform all destructive >> actions in the clouds (the reboot of nodes, restart of OpenStack services, >> enable/disable network interfaces or some firewall rules and etc). >> >> We need to get +1 from you in [1] to create the repository with >> advanced end-user scenario tests. > > OK, I'd like to summarize my thinking here for adding os-faults under > QA project: > > Under QA project, the first target of most programs(tempest, devstack, > etc) is for the gate testing. > Of course, some of them are available on production clouds also as the > design, but the first is for the gate. > That means devstack clouds as the first target/purpose. > At this time, this os-faults doesn't seem the gate as the first > purpose based on current implementation. > So I feel os-faults seems different from the existing programs. > > os-faults is very young and we will be able to extend it for devstack > clouds also later, I hope. > In addition, I heard this kind of tests(destructive, HA, etc) are > requested from the other users. > So this os-faults seems very valuable for users. > > Just in my opinion, I am find to add this os-faults under QA project. > But before that, I'd like to get opinions from the other people of QA team. > > Thanks > Ken Ohmichi > > > > > > > > > > > > >> On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov >> wrote: >>> >>> Hi Ken, >>> >>> OS-Faults doesn't have any scenarios in the tree yet (the project is two >>> months old), but you can find some examples of the use in the >>> os-faults/examples directory. >>> >>> Regards, >>> Yaroslav Lobankov. >>> >>> On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi >>> wrote: Hi Timur, Thanks for your explanation. 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov : > >> I am guessing the above "restart nodes" is for verifying each >> OpenStack service restarts successfully, right? > > Yes, this is right. And we also will check that HA logic for these > services works correctly (for example, rescheduling of L3 Neutron > agents for networks). > >> But these service scripts are provided by distributors, and Devstack >> itself doesn't contain service scripts IIUC. >> So I'd like to know how to verify it on Devstack clouds. > > Yes, DevStack doesn't support many scenarios which are actual > and should be supported on the production clouds. > It will be not possible to run all advanced test scenarios for DevStack > clouds, > just because DevStack can't deploy OpenStack cloud with 3 controllers > now (so, probably it will be possible in the future). > > Of course, some advanced scenarios will support DevStack clouds, > for example, some test scenarios which are based on customer-found > issues from the real production clouds, like upload of the large images > (100+ Gb) > to Glance with Swift backend. Such cases are important for verification > of > pre-production environments, but not very important for CI gate jobs. > > It is also important to note that in these advanced cases we are > targeting > to check not only the logic of Python code, but also the correct > configuration > of all OpenStack components on some pre-production OpenStack clusters.
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi Timur, 2016-10-06 4:08 GMT-07:00 Timur Nurlygayanov : > Ken, it is a good idea! > > The plan is to develop os-faults as a library which will be able > to manage cluster nodes and OpenStack services on these nodes. > It is a good idea to add some Tempest tests which will use > os-faults library as well for some API tests. Sorry for my misleading, that is not I wanted to say. I am thinking Tempest doesn't use os-faults as library. I just wanted to say some destructive tests which will use REST APIs can be implemented in Tempest instead of os-faults. For example, the compute service client contains disable_service which disables nova service but Tempest doesn't contain the corresponding test cases because Tempest tests need to run in parallel. https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/services_client.py#L54 If some tests use this method, the other tests which run in parallel can be failed as unexpected on the gate. Such tests also are destructive and I am hoping these tests also will be able to run the gate by finding better way. > The Stepler framework [1] will use os-faults to perform all destructive > actions in the clouds (the reboot of nodes, restart of OpenStack services, > enable/disable network interfaces or some firewall rules and etc). > > We need to get +1 from you in [1] to create the repository with > advanced end-user scenario tests. OK, I'd like to summarize my thinking here for adding os-faults under QA project: Under QA project, the first target of most programs(tempest, devstack, etc) is for the gate testing. Of course, some of them are available on production clouds also as the design, but the first is for the gate. That means devstack clouds as the first target/purpose. At this time, this os-faults doesn't seem the gate as the first purpose based on current implementation. So I feel os-faults seems different from the existing programs. os-faults is very young and we will be able to extend it for devstack clouds also later, I hope. In addition, I heard this kind of tests(destructive, HA, etc) are requested from the other users. So this os-faults seems very valuable for users. Just in my opinion, I am find to add this os-faults under QA project. But before that, I'd like to get opinions from the other people of QA team. Thanks Ken Ohmichi > On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov > wrote: >> >> Hi Ken, >> >> OS-Faults doesn't have any scenarios in the tree yet (the project is two >> months old), but you can find some examples of the use in the >> os-faults/examples directory. >> >> Regards, >> Yaroslav Lobankov. >> >> On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi >> wrote: >>> >>> Hi Timur, >>> >>> Thanks for your explanation. >>> >>> 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov >>> : >>> > >>> >> I am guessing the above "restart nodes" is for verifying each >>> >> OpenStack service restarts successfully, right? >>> > >>> > Yes, this is right. And we also will check that HA logic for these >>> > services works correctly (for example, rescheduling of L3 Neutron >>> > agents for networks). >>> > >>> >> But these service scripts are provided by distributors, and Devstack >>> >> itself doesn't contain service scripts IIUC. >>> >> So I'd like to know how to verify it on Devstack clouds. >>> > >>> > Yes, DevStack doesn't support many scenarios which are actual >>> > and should be supported on the production clouds. >>> > It will be not possible to run all advanced test scenarios for DevStack >>> > clouds, >>> > just because DevStack can't deploy OpenStack cloud with 3 controllers >>> > now (so, probably it will be possible in the future). >>> > >>> > Of course, some advanced scenarios will support DevStack clouds, >>> > for example, some test scenarios which are based on customer-found >>> > issues from the real production clouds, like upload of the large images >>> > (100+ Gb) >>> > to Glance with Swift backend. Such cases are important for verification >>> > of >>> > pre-production environments, but not very important for CI gate jobs. >>> > >>> > It is also important to note that in these advanced cases we are >>> > targeting >>> > to check not only the logic of Python code, but also the correct >>> > configuration >>> > of all OpenStack components on some pre-production OpenStack clusters. >>> >>> I guessed some part of os-faults can be moved to Tempest if os-faults >>> contains API tests for enabling/disabling OpenStack services. >>> Then, os-faults would be able to concentrate on more destructive tests >>> like rebooting physical nodes, etc. >>> However, I could not find any actual scenarios on current os-faults >>> (https://github.com/openstack/os-faults). >>> That seems to just contain some abstraction layers and unit tests. Can >>> we see actual test scenarios of os-faults ? >>> Maybe I missed something. >>> >>> Thanks >>> Ken Ohmichi >>> >>> >>>
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Ken, it is a good idea! The plan is to develop os-faults as a library which will be able to manage cluster nodes and OpenStack services on these nodes. It is a good idea to add some Tempest tests which will use os-faults library as well for some API tests. The Stepler framework [1] will use os-faults to perform all destructive actions in the clouds (the reboot of nodes, restart of OpenStack services, enable/disable network interfaces or some firewall rules and etc). We need to get +1 from you in [1] to create the repository with advanced end-user scenario tests. Thank you! [1] https://review.openstack.org/#/c/374667/ On Tue, Oct 4, 2016 at 8:53 PM, Yaroslav Lobankov wrote: > Hi Ken, > > OS-Faults doesn't have any scenarios in the tree yet (the project is two > months old), but you can find some examples of the use in the > os-faults/examples directory. > > Regards, > Yaroslav Lobankov. > > On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi > wrote: > >> Hi Timur, >> >> Thanks for your explanation. >> >> 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov > >: >> > >> >> I am guessing the above "restart nodes" is for verifying each >> >> OpenStack service restarts successfully, right? >> > >> > Yes, this is right. And we also will check that HA logic for these >> > services works correctly (for example, rescheduling of L3 Neutron >> > agents for networks). >> > >> >> But these service scripts are provided by distributors, and Devstack >> >> itself doesn't contain service scripts IIUC. >> >> So I'd like to know how to verify it on Devstack clouds. >> > >> > Yes, DevStack doesn't support many scenarios which are actual >> > and should be supported on the production clouds. >> > It will be not possible to run all advanced test scenarios for DevStack >> > clouds, >> > just because DevStack can't deploy OpenStack cloud with 3 controllers >> > now (so, probably it will be possible in the future). >> > >> > Of course, some advanced scenarios will support DevStack clouds, >> > for example, some test scenarios which are based on customer-found >> > issues from the real production clouds, like upload of the large images >> > (100+ Gb) >> > to Glance with Swift backend. Such cases are important for verification >> of >> > pre-production environments, but not very important for CI gate jobs. >> > >> > It is also important to note that in these advanced cases we are >> targeting >> > to check not only the logic of Python code, but also the correct >> > configuration >> > of all OpenStack components on some pre-production OpenStack clusters. >> >> I guessed some part of os-faults can be moved to Tempest if os-faults >> contains API tests for enabling/disabling OpenStack services. >> Then, os-faults would be able to concentrate on more destructive tests >> like rebooting physical nodes, etc. >> However, I could not find any actual scenarios on current os-faults >> (https://github.com/openstack/os-faults). >> That seems to just contain some abstraction layers and unit tests. Can >> we see actual test scenarios of os-faults ? >> Maybe I missed something. >> >> Thanks >> Ken Ohmichi >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Timur, Senior QA Manager OpenStack Projects Mirantis Inc __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi Ken, OS-Faults doesn't have any scenarios in the tree yet (the project is two months old), but you can find some examples of the use in the os-faults/examples directory. Regards, Yaroslav Lobankov. On Tue, Oct 4, 2016 at 8:02 PM, Ken'ichi Ohmichi wrote: > Hi Timur, > > Thanks for your explanation. > > 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov : > > > >> I am guessing the above "restart nodes" is for verifying each > >> OpenStack service restarts successfully, right? > > > > Yes, this is right. And we also will check that HA logic for these > > services works correctly (for example, rescheduling of L3 Neutron > > agents for networks). > > > >> But these service scripts are provided by distributors, and Devstack > >> itself doesn't contain service scripts IIUC. > >> So I'd like to know how to verify it on Devstack clouds. > > > > Yes, DevStack doesn't support many scenarios which are actual > > and should be supported on the production clouds. > > It will be not possible to run all advanced test scenarios for DevStack > > clouds, > > just because DevStack can't deploy OpenStack cloud with 3 controllers > > now (so, probably it will be possible in the future). > > > > Of course, some advanced scenarios will support DevStack clouds, > > for example, some test scenarios which are based on customer-found > > issues from the real production clouds, like upload of the large images > > (100+ Gb) > > to Glance with Swift backend. Such cases are important for verification > of > > pre-production environments, but not very important for CI gate jobs. > > > > It is also important to note that in these advanced cases we are > targeting > > to check not only the logic of Python code, but also the correct > > configuration > > of all OpenStack components on some pre-production OpenStack clusters. > > I guessed some part of os-faults can be moved to Tempest if os-faults > contains API tests for enabling/disabling OpenStack services. > Then, os-faults would be able to concentrate on more destructive tests > like rebooting physical nodes, etc. > However, I could not find any actual scenarios on current os-faults > (https://github.com/openstack/os-faults). > That seems to just contain some abstraction layers and unit tests. Can > we see actual test scenarios of os-faults ? > Maybe I missed something. > > Thanks > Ken Ohmichi > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi Timur, Thanks for your explanation. 2016-09-29 6:22 GMT-07:00 Timur Nurlygayanov : > >> I am guessing the above "restart nodes" is for verifying each >> OpenStack service restarts successfully, right? > > Yes, this is right. And we also will check that HA logic for these > services works correctly (for example, rescheduling of L3 Neutron > agents for networks). > >> But these service scripts are provided by distributors, and Devstack >> itself doesn't contain service scripts IIUC. >> So I'd like to know how to verify it on Devstack clouds. > > Yes, DevStack doesn't support many scenarios which are actual > and should be supported on the production clouds. > It will be not possible to run all advanced test scenarios for DevStack > clouds, > just because DevStack can't deploy OpenStack cloud with 3 controllers > now (so, probably it will be possible in the future). > > Of course, some advanced scenarios will support DevStack clouds, > for example, some test scenarios which are based on customer-found > issues from the real production clouds, like upload of the large images > (100+ Gb) > to Glance with Swift backend. Such cases are important for verification of > pre-production environments, but not very important for CI gate jobs. > > It is also important to note that in these advanced cases we are targeting > to check not only the logic of Python code, but also the correct > configuration > of all OpenStack components on some pre-production OpenStack clusters. I guessed some part of os-faults can be moved to Tempest if os-faults contains API tests for enabling/disabling OpenStack services. Then, os-faults would be able to concentrate on more destructive tests like rebooting physical nodes, etc. However, I could not find any actual scenarios on current os-faults (https://github.com/openstack/os-faults). That seems to just contain some abstraction layers and unit tests. Can we see actual test scenarios of os-faults ? Maybe I missed something. Thanks Ken Ohmichi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi Ken, I am guessing the above "restart nodes" is for verifying each > OpenStack service restarts successfully, right? Yes, this is right. And we also will check that HA logic for these services works correctly (for example, rescheduling of L3 Neutron agents for networks). But these service scripts are provided by distributors, and Devstack > itself doesn't contain service scripts IIUC. > So I'd like to know how to verify it on Devstack clouds. Yes, DevStack doesn't support many scenarios which are actual and should be supported on the production clouds. It will be not possible to run all advanced test scenarios for DevStack clouds, just because DevStack can't deploy OpenStack cloud with 3 controllers now (so, probably it will be possible in the future). Of course, some advanced scenarios will support DevStack clouds, for example, some test scenarios which are based on customer-found issues from the real production clouds, like upload of the large images (100+ Gb) to Glance with Swift backend. Such cases are important for verification of pre-production environments, but not very important for CI gate jobs. It is also important to note that in these advanced cases we are targeting to check not only the logic of Python code, but also the correct configuration of all OpenStack components on some pre-production OpenStack clusters. Thank you! On Wed, Sep 28, 2016 at 2:37 AM, Ken'ichi Ohmichi wrote: > Hi Timur, > > Thanks for picking this up, that is interesting for me. > > 2016-09-22 5:58 GMT-07:00 Timur Nurlygayanov : > > > > we have an idea to create the test suite with destructive/HA and advanced > > end-user scenarios for the OpenStack clusters. This test suite will > contains > > advanced scenario integration tests for OpenStack clusters to make sure > that > > the cluster is ready for the production. > > > > The test cases which we want to cover in this test suite: > > 1) All simple and advanced destructive actions, like a reboot of the > nodes, > > restart OpenStack services and etc. (we can probably use of-faults > library > > [1], which we already use in Rally) > > 2) All advanced test scenarios like a migration of the bunch of VMs > between > > nodes and booting of the VMs with large images (10+ Gb), send traffic > > between VMs and in parallel restart Neutron l3 agents and etc. > > > > The key requirements: > > 1) The framework should know the details of the deployments (how many > nodes > > we have, how to ssh to OpenStack nodes, how to restart the nodes and > etc.). > > This is why we don't want to add such "advanced" and HA-focused test > > scenarios to Tempest. > > Yeah, this point is right. This "advanced" way is different from the > design principle of Tempest[1]. > I am guessing the above "restart nodes" is for verifying each > OpenStack service restarts successfully, right? > For productions(or distributions), this verifying point seems > important because service scripts need to restart OpenStack services > automatically. > But these service scripts are provided by distributors, and Devstack > itself doesn't contain service scripts IIUC. > So I'd like to know how to verify it on Devstack clouds. > > Thanks > Ken Ohmichi > --- > > [1]: https://github.com/openstack/tempest#design-principles > > > 2) We should be ready to run these tests for any clouds: DevStack clouds > (we > > can skip HA cases for DevStack), Fuel clouds, clouds which were deployed > by > > Ansible or Puppet tools. > > 3) This framework should allow reproduce the issue in a repeatable > manner, > > this is why we can't just cover all the tests with Rally load tests + > > destructive plugins (we are working on this right now too to have an > ability > > to test HA-related scenarios under the load). > > > > As we discussed on the OpenStack summit a year ago it is better to move > such > > test suite in a separate repository and this framework can became a part > of > > the QA (or at least BigTent) program in OpenStack. > > > > I've created the commit to OpenStack project-config repository: > > https://review.openstack.org/#/c/374667/ > > > > Could you please take a look? > > > > We understand that it will be hard to add such test suite to the gates > for > > every commit in OpenStack because we will need a lot of hardware. We > don't > > want to add these tests to the per-commit gates for now, it is ok to run > > them just once a day, for example. And we definitely need to have such > test > > suite to validate our own pre-production clouds. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Timur, Senior QA Manager OpenStack Projects Mirantis Inc __ OpenStack Development Mailing List (not for usage quest
Re: [openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi Timur, Thanks for picking this up, that is interesting for me. 2016-09-22 5:58 GMT-07:00 Timur Nurlygayanov : > > we have an idea to create the test suite with destructive/HA and advanced > end-user scenarios for the OpenStack clusters. This test suite will contains > advanced scenario integration tests for OpenStack clusters to make sure that > the cluster is ready for the production. > > The test cases which we want to cover in this test suite: > 1) All simple and advanced destructive actions, like a reboot of the nodes, > restart OpenStack services and etc. (we can probably use of-faults library > [1], which we already use in Rally) > 2) All advanced test scenarios like a migration of the bunch of VMs between > nodes and booting of the VMs with large images (10+ Gb), send traffic > between VMs and in parallel restart Neutron l3 agents and etc. > > The key requirements: > 1) The framework should know the details of the deployments (how many nodes > we have, how to ssh to OpenStack nodes, how to restart the nodes and etc.). > This is why we don't want to add such "advanced" and HA-focused test > scenarios to Tempest. Yeah, this point is right. This "advanced" way is different from the design principle of Tempest[1]. I am guessing the above "restart nodes" is for verifying each OpenStack service restarts successfully, right? For productions(or distributions), this verifying point seems important because service scripts need to restart OpenStack services automatically. But these service scripts are provided by distributors, and Devstack itself doesn't contain service scripts IIUC. So I'd like to know how to verify it on Devstack clouds. Thanks Ken Ohmichi --- [1]: https://github.com/openstack/tempest#design-principles > 2) We should be ready to run these tests for any clouds: DevStack clouds (we > can skip HA cases for DevStack), Fuel clouds, clouds which were deployed by > Ansible or Puppet tools. > 3) This framework should allow reproduce the issue in a repeatable manner, > this is why we can't just cover all the tests with Rally load tests + > destructive plugins (we are working on this right now too to have an ability > to test HA-related scenarios under the load). > > As we discussed on the OpenStack summit a year ago it is better to move such > test suite in a separate repository and this framework can became a part of > the QA (or at least BigTent) program in OpenStack. > > I've created the commit to OpenStack project-config repository: > https://review.openstack.org/#/c/374667/ > > Could you please take a look? > > We understand that it will be hard to add such test suite to the gates for > every commit in OpenStack because we will need a lot of hardware. We don't > want to add these tests to the per-commit gates for now, it is ok to run > them just once a day, for example. And we definitely need to have such test > suite to validate our own pre-production clouds. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [QA] The end-user test suite for OpenStack clusters
Hi team, we have an idea to create the test suite with destructive/HA and advanced end-user scenarios for the OpenStack clusters. This test suite will contains advanced scenario integration tests for OpenStack clusters to make sure that the cluster is ready for the production. The test cases which we want to cover in this test suite: 1) All simple and advanced destructive actions, like a reboot of the nodes, restart OpenStack services and etc. (we can probably use of-faults library [1], which we already use in Rally) 2) All advanced test scenarios like a migration of the bunch of VMs between nodes and booting of the VMs with large images (10+ Gb), send traffic between VMs and in parallel restart Neutron l3 agents and etc. The key requirements: 1) The framework should know the details of the deployments (how many nodes we have, how to ssh to OpenStack nodes, how to restart the nodes and etc.). This is why we don't want to add such "advanced" and HA-focused test scenarios to Tempest. 2) We should be ready to run these tests for any clouds: DevStack clouds (we can skip HA cases for DevStack), Fuel clouds, clouds which were deployed by Ansible or Puppet tools. 3) This framework should allow reproduce the issue in a repeatable manner, this is why we can't just cover all the tests with Rally load tests + destructive plugins (we are working on this right now too to have an ability to test HA-related scenarios under the load). As we discussed on the OpenStack summit a year ago it is better to move such test suite in a separate repository and this framework can became a part of the QA (or at least BigTent) program in OpenStack. I've created the commit to OpenStack project-config repository: https://review.openstack.org/#/c/374667/ Could you please take a look? We understand that it will be hard to add such test suite to the gates for every commit in OpenStack because we will need a lot of hardware. We don't want to add these tests to the per-commit gates for now, it is ok to run them just once a day, for example. And we definitely need to have such test suite to validate our own pre-production clouds. Thank you! [1] https://github.com/openstack/os-faults -- Timur, QA Manager OpenStack Projects Mirantis Inc __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev