Hi Ghanshyam and Ildiko,
Firstly, sorry for the late response. I have just finished reading from the links you provided. It is very excited for me knowing that tempest could consider stress testing as part of OpenStack QA. In OPNFV, we start to plan stress testing from last release. It would be more excited for me if we could cooperate on this kind of testing. I have added a summary of our resiliency/stress/robustness/long duration testing activities in https://etherpad.openstack.org/p/qa-queens-ptg (as agreed in OPNFV testing group) and hoped it could form some foundation for our further discussion. I also put the summary below, please feel free to contact me if there is any doubt/comment/question. In OPNFV Danube (OS Newton) release, OPNFV testing working group (Testperf) decides to extend the OPNFV testing scope by introducing basic stress test cases. There are 5 test cases discussed. Different from the destructive test (HA) that OpenStack QA team have considered, these test cases are basically designed from load stress perspective instead of redundancy perspective. In Danube, Bottlenecks project (OPNFV system limit testing project) and Yardstick project (OPNFV performance test project for NFVI and VNF) jointly implemented 2 of them (concurrently create/destroy VM pairs and do ping, system throughput limit) while Bottlenecks acts as the load manager calling yardstick to execute each test iteration. These tests are designed to test for breaking points and provide level of confidence of the system to users. More details could be found at a) https://wiki.opnfv.org/display/bottlenecks/Stress+Testing+over+OPNFV+Platform b) https://wiki.opnfv.org/download/attachments/2926539/Testing%20over%20Long%20Duration%20POD.pptx?version=2&modificationDate=1502943821000&api=v2 Stress test cases for OPNFV Euphrates (OS Ocata) release can be seen as extension/enhancement of those in D release. These test cases are under degugging in Bottlenecks/Yardstick repo (Bottlenecks as load manager while Yardstick execute each test iteration): a) VNF scale out/up tests (also plan to measure storage usage simultaneously): https://wiki.opnfv.org/pages/viewpage.action?pageId=12390101 b) Life-cycle event with throughputs (measure NFVI to support concurrent network usage from different VM pairs): https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Baseline+Stress+Test+Case+for+Bottlenecks+E+Release In OPNFV E release, we also plan to do long duration test over OS Ocata. A separate CI pipe testing OPNFV XCI (OSA) is proposed to accomplish the job. We have applied specific pod for the testing. Detailed proposal could be found at: a) https://wiki.opnfv.org/display/testing/Euphrates+Testing+needs b) https://wiki.opnfv.org/download/attachments/2926539/testing%20evolution%20v1_4.pptx?version=1&modificationDate=1503937629000&api=v2 c) https://wiki.opnfv.org/download/attachments/2926539/Testing%20over%20Long%20Duration%20POD.pptx?version=2&modificationDate=1502943821000&api=v2 The long duration test is supposed to be started when OPNFV E release is published. Bottlenecks plans to add a monitoring module for these tests: https://wiki.opnfv.org/display/DEV/Intern+Project%3A+Monitoring+Stress+Testing+for+Bottlenecks+E+Release ps. @Testperf please add if there is anything missing Waiting for your reply, Gabriel -----Original Message----- From: Ghanshyam Mann [mailto:ghanshyamm...@gmail.com] Sent: Tuesday, August 29, 2017 10:12 AM To: Ildiko Vancsa Cc: RICHOMME Morgan IMT/OLN; Andrea Frittoli; test...@lists.opnfv.org; Yuyang (Gabriel); Pierre Lynch Subject: Re: [OPNFV] [OpenStack] resilency/stress/long duration/robustness testing synchro Thanks ildiko for adding in ML. From OpenStack QA perspective, we have some scope limitation of adding the tests in Tempest repo but it can be done as separate repo similar to other similar kind of testing (RBAC testing). Good things is that we do have common testing framework and plugin mechanism to implement more broader testing as separate QA/other community projects. I am pasting few quick link to get overview of OpenStack QA projects : Current Scope of Tempest and Plugin framework. - https://docs.openstack.org/tempest/latest/field_guide/index.html - https://docs.openstack.org/tempest/latest/plugin.html Next plan: 1. Tacker Tempest Pugin: I am planing to add Tacker Tempest plugin framework in PTG and further my team will start adding the suitable Scenario tests. We are doing NFV PoC in NEC with manually testing. Our next goal is to automate that testing which can be useful in open way. 2. Stress Testing: Tempest used to have stress tests but it is being moved out of Tempest due to scope limitation. Currently it is in private repo but this can be added as QA projects. I remember someone asked about this in Boston but never followed up. Further it can be extended to include NFV related stress testing also. Link https://github.com/ghanshyammann/tempest_stress 3. Destructive Testing: There is proposal for doing the destructive testing. We are planning to discuss the same in PTG. This is more about HA and other destructive action on system. There are some NFV testing scope also in this. Link https://review.openstack.org/#/c/443504/ PTG QA etherpad: https://etherpad.openstack.org/p/qa-queens-ptg Further, let's meet at PTG and get more clarity about requirement and how OpenStack QA can help on those. -gmann On Tue, Aug 29, 2017 at 1:57 AM, Ildiko Vancsa <ild...@openstack.org<mailto:ild...@openstack.org>> wrote: > Hi Morgan, > > Sounds good, also thank you for the pointers. I agree that collaboration in > this area would be highly beneficial for all parties. > > We are planning to have a co-located workshop [1] between ETSI NFV and > OpenStack on Tuesday at the PTG venue in Denver. Pierre (Lynch) reached out > to me in the meantime about resiliency and long duration testing as agenda > item for possible future collaboration there. Ghanshyam and Andrea are > planning to attend that session as well. > > That session could be a good starting point for gathering the relevant people > and then discuss the topics further during the week on the QA sessions and > after the events in the area onward. > > If you know about people who plan to join please encourage them to add their > names to the etherpad(s) so we know who’s interested in the topic and who to > look for/contact. > > Please let us know if you have any questions or comments. > > Thanks and Best Regards, > Ildikó > > [1] > https://etherpad.openstack.org/p/etsi-nfv-openstack-gathering-denver > > >> On 2017. Aug 28., at 18:29, >> <morgan.richo...@orange.com<mailto:morgan.richo...@orange.com>> >> <morgan.richo...@orange.com<mailto:morgan.richo...@orange.com>> wrote: >> >> Hi, >> >> we dicussed a little bit this topic during our last weekly meeting >> [1] >> >> some OPNFV/ETSI NFV actors will be in Denver so it should be the right place >> to start cross discussions. >> >> I shared your link [2], it seems that we are sharing lots of >> concerns :) (too many tests, keep the time control of the gate, >> test/scenario promotion/ test weight....) we could clearly benefit >> from your experience >> >> Gabriel (bottlenecks project) created first capability testing suites >> on our Danube (~ Newton) it leads to our action to ask for resources >> and get a dedicated pod for long duration/resiliency testing see [2] >> [3] >> >> Unfortunatly I would not be able to join the PTG this time, but I am >> sure some OPNFV contributors will >> >> I will add a session on the review of your etherpad before the PTG >> >> /Morgan >> >> >> [1]: >> http://ircbot.wl.linuxfoundation.org/meetings/opnfv-testperf/2017/opn >> fv-testperf.2017-08-24-14.59.html >> [2]: >> https://wiki.opnfv.org/display/meetings/Test+Working+Group+Weekly+Mee >> ting?preview=/2926539/12389654/testing%20evolution%20v1_4.pptx >> [3] >> https://wiki.opnfv.org/display/meetings/Test+Working+Group+Weekly+Mee >> ting?preview=/2926539/12388723/Threshold%20for%20Concurrent%20VM%20cr >> eation.pptx >> >> On 24/08/2017 11:29, Ildiko Vancsa wrote: >>> Hi Morgan, >>> >>> Thank you for reaching out. >>> >>> I added Ghanshyam and Andrea to the thread, they are members of our QA team >>> and can give you more insights on how testing works on our end and what the >>> plans are to extend the scope further. >>> >>> @ Ghanshyam, Andrea: Could you help with a short overview and links to some >>> materials that describes where we are now and what the plans are? >>> >>> >>> Also the QA team is meeting at the upcoming PTG [1][2] in Denver in >>> September, which would be a great opportunity to discuss topics like >>> robustness testing and collaboration between the two communities on >>> testing. It would be great if you could participate in those discussions >>> while it would also be a good place to synchronize. >>> >>> Are you or anyone from the OPNFV testing groups planning to come to the PTG? >>> >>> Thanks and Best Regards, >>> Ildikó >>> >>> [1] https://www.openstack.org/ptg/ >>> [2] https://etherpad.openstack.org/p/qa-queens-ptg >>> >>> >>>> On 2017. Aug 22., at 15:25, >>>> morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> wrote: >>>> >>>> Hi Ildiko, >>>> >>>> the OPNFV testing group is trying to setup resiliency/long >>>> duration/robustness/stress tests in OPNFV. >>>> https://wiki.opnfv.org/download/attachments/2926539/Testing%20over%<https://wiki.opnfv.org/download/attachments/2926539/Testing%20over%25> >>>> 20Long%20Duration%20POD.pptx?version=2&modificationDate=15029438210 >>>> 00&api=v2 >>>> >>>> Due to OPNFV constraints (late availability of the stable release, >>>> dependencies to the installers), we assume that the best option would >>>> consist in running tests on Openstack installed through OSA (XCI) we would >>>> consider the OpenStack version targeted for the OPNFV solution, share time >>>> slot between the different test projects and report feedback upstream. >>>> >>>> During the presentation to the TSC then OPNFV Infra working group, >>>> somebody suggests to synchro with OpenStack on this topic. >>>> Would it be possible to organize a synchro point and see what is already >>>> done in OpenStack? >>>> The testing group would be interesting to get an overview of the test >>>> projects / test cases addressing these issues in OpenStack community. >>>> >>>> Regards >>>> >>>> Morgan >>>> >>>> ___________________________________________________________________ >>>> ______________________________________________________ >>>> >>>> Ce message et ses pieces jointes peuvent contenir des informations >>>> confidentielles ou privilegiees et ne doivent donc pas etre >>>> diffuses, exploites ou copies sans autorisation. Si vous avez recu >>>> ce message par erreur, veuillez le signaler a l'expediteur et le detruire >>>> ainsi que les pieces jointes. Les messages electroniques etant >>>> susceptibles d'alteration, Orange decline toute responsabilite si ce >>>> message a ete altere, deforme ou falsifie. Merci. >>>> >>>> This message and its attachments may contain confidential or >>>> privileged information that may be protected by law; they should not be >>>> distributed, used or copied without authorisation. >>>> If you have received this email in error, please notify the sender and >>>> delete this message and its attachments. >>>> As emails may be altered, Orange is not liable for messages that have been >>>> modified, changed or falsified. >>>> Thank you. >>>> >> >> -- >> Morgan Richomme >> Orange/ IMT/ OLN/ CNC/ NCA/ SINA >> >> Network architect for innovative services Future of the Network >> community member Open source Orange community manager >> >> >> tel. +33 (0) 296 072 106 >> mob. +33 (0) 637 753 326 >> morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> >> >> >> _____________________________________________________________________ >> ____________________________________________________ >> >> Ce message et ses pieces jointes peuvent contenir des informations >> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, >> exploites ou copies sans autorisation. Si vous avez recu ce message >> par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les >> pieces jointes. Les messages electroniques etant susceptibles d'alteration, >> Orange decline toute responsabilite si ce message a ete altere, deforme ou >> falsifie. Merci. >> >> This message and its attachments may contain confidential or >> privileged information that may be protected by law; they should not be >> distributed, used or copied without authorisation. >> If you have received this email in error, please notify the sender and >> delete this message and its attachments. >> As emails may be altered, Orange is not liable for messages that have been >> modified, changed or falsified. >> Thank you. >> >
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
