Hi, I just want to stress again that CI is a software development practice, not something that is just used for release testing purposes. The CI doesn't start and end with daily runs so what we are doing with what we call "CI" is fundamentally broken.
As a community, it is past time we take a step back and look at what we've been doing. One of the things that require improvements is how we take our decisions. I suppose I'm not terribly wrong if I say our decisions, especially the release related ones, are mainly driven by pushing more stuff out and on time rather than being on time with quality. If we look at our past releases, we have successfully been lowering the quality by removing testing and limiting the time it takes for testing community to troubleshoot/work on issues with developers. I think 2nd release had the highest quality - we at least took benchmarking results into account and gave more time to our testing community. Last release, we were almost going to release something that's not even tested. Due to this, we push more and more stuff out with lower quality. This results in long cycle times, lack of resources and queued executions because we want to ensure that that many number of things can be deployed X number of times rather than looking at if few of those things can be deployed, providing the intended functionality and can stay up for Y number of days under stress, giving people confidence in them so they can go ahead with their feature integration. Because of this, the integration happens pretty late in release cycle, not leaving enough time for people to test things properly. Apart from that, everything we do changes all the time. Infra, test frameworks, test cases, installers, features, upstream components, etc. We need to take things under control and limit the number of things that change between 2 runs so we can compare apples with apples to see what went wrong when something turns red and fix it properly, relieving some pressure from the people and the infrastructure by avoiding rerunning/retroubleshooting improperly fixed things so we can prevent waste of time and resources. Some of you know that we have been trying to improve what we are doing but we haven't been successful due to different reasons. Some of the examples are commit gating, promotion, trust indicator, not putting non-working stuff on baremetal, having separate pipelines for test projects, better handling of our resources, etc. We will obviously continue trying these and even more as long as we get support from the community. So I personally support this initiative fully and I even think that we should cut the number of baremetal deployments in half and dedicate the rest of the time to non-functional testing. This might require us to cut the number of features we release, bring stuff on to baremetal when/if they meet certain criteria, or be clever when it comes to how we use our limited resources (human and infra) and construct/utilize our CI. /Fatih From: <opnfv-tech-discuss-boun...@lists.opnfv.org> on behalf of Wenjing Chu <chu.wenj...@gmail.com> Date: Monday, 26 June 2017 at 21:36 To: "Beierl, Mark" <mark.bei...@dell.com> Cc: "test...@lists.opnfv.org" <test...@lists.opnfv.org>, "opnfv-tech-discuss@lists.opnfv.org" <opnfv-tech-discuss@lists.opnfv.org> Subject: Re: [opnfv-tech-discuss] [OPNFV] Draft support for presentation to TSC +1 on the growing importance of resilience and stress testing. I'd like to offer support of your effort from the CVP perspectives: One is that resiliency testing is one of the prime candidates for CVP in E release in my mind. The TC3, for instance, is a great example. And the second point, Dovetail is facing similar testing challenges. We would also like to see the creation of another testing pipeline that can be focused on stable systems and generic scenarios, somewhat removed from the CI used for day-to-day release testing. Wenjing On Fri, Jun 23, 2017 at 8:08 AM, Beierl, Mark <mark.bei...@dell.com<mailto:mark.bei...@dell.com>> wrote: I have added 2 slides, which may be deleted if not adding enough to the message: Slide #3 - Heavy Reading readout from summit shows Performance and Stress testing to be growing in OPNFV importance to respondents. This is a screen shot of Roz's talk. If someone can find the original slide deck, it would be better. Slide #8 - After "Towards a test strategy". Shows what it could look like to have cross-project stresses running at same time. Example shows combined storage and tenant networks with VSPERF and StorPerf sharing traffic vs. dedicated networks and full throughput available for both. Regards, Mark Mark Beierl SW System Sr Principal Engineer Dell EMC | Office of the CTO mobile +1 613 314 8106<tel:(613)%20314-8106> mark.bei...@dell.com > On Jun 23, 2017, at 06:38, Yuyang (Gabriel) > <gabriel.yuy...@huawei.com<mailto:gabriel.yuy...@huawei.com>> wrote: > > Hi Morgan, > > Thanks for the information! Please find my version of the slides deck > attached. > In addition, I have installed Libre office. Feel awesome!! > > @All Feel free to do any change needed. > > > Best, > Gabriel > > -----Original Message----- > From: morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> > [mailto:morgan.richo...@orange.com] > Sent: Friday, June 23, 2017 3:36 PM > To: Yuyang (Gabriel); Brattain, Ross B; Gaoliang (kubi); Jose Lausuch; > Cooper, Trevor; MORTON, ALFRED C (AL); > mark.bei...@emc.com<mailto:mark.bei...@emc.com>; Fatih Degirmenci; Donald > Hunter (donaldh); Yujun Zhang > Cc: test...@lists.opnfv.org<mailto:test...@lists.opnfv.org>; > opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> > Subject: Re: [OPNFV] Draft support for presentation to TSC > > the pptx version > note you can open odp with Libre office, an free software alternative to > Microsoft office > > OK to add the list of stress tests under planning > > /Morgan > > On 23/06/2017 09:19, Yuyang (Gabriel) wrote: >> Hi Morgan, >> >> Sure, I can present the beginning for the stress test in Danube. >> One quick question, do we need to add a brief list for the stress test cases >> under planning? >> >> By the way, do you have a pptx version of your slides? So that we can modify >> the slides cooperatively. The odp version looks dark and a mess in my office >> 2013. >> >> Thanks, >> Gabriel >> >> >> -----Original Message----- >> From: morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> >> [mailto:morgan.richo...@orange.com] >> Sent: Friday, June 23, 2017 2:39 PM >> To: Yuyang (Gabriel); Brattain, Ross B; Gaoliang (kubi); Jose Lausuch; >> Cooper, Trevor; MORTON, ALFRED C (AL); >> mark.bei...@emc.com<mailto:mark.bei...@emc.com>; Fatih >> Degirmenci; Donald Hunter (donaldh); Yujun Zhang >> Cc: test...@lists.opnfv.org<mailto:test...@lists.opnfv.org>; >> opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> >> Subject: [OPNFV] Draft support for presentation to TSC >> >> Hi >> >> as discussed yesterday during the weekly meeting, I prepared a slide deck >> for the discussion with the TSC (I tried to summarize our discussion and >> hope it reflects the wiki pages we created on the topic). >> >> Feel free to comment/complete/criticize/modify >> @Gabriel: would you be OK to present the beginning? (stress tests >> created for Danube + wiki page and email thread) >> >> I already booked a slot for the meeting next week >> https://wiki.opnfv.org/display/meetings/TSC >> the agenda looks already pretty busy not sure we will have time, we >> will see >> >> /Morgan >> >> >> >> ______________________________________________________________________ >> ___________________________________________________ >> >> Ce message et ses pieces jointes peuvent contenir des informations >> confidentielles ou privilegiees et ne doivent donc pas etre diffuses, >> exploites ou copies sans autorisation. Si vous avez recu ce message par >> erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les >> pieces jointes. Les messages electroniques etant susceptibles d'alteration, >> Orange decline toute responsabilite si ce message a ete altere, deforme ou >> falsifie. Merci. >> >> This message and its attachments may contain confidential or privileged >> information that may be protected by law; they should not be distributed, >> used or copied without authorisation. >> If you have received this email in error, please notify the sender and >> delete this message and its attachments. >> As emails may be altered, Orange is not liable for messages that have been >> modified, changed or falsified. >> Thank you. >> > > -- > Morgan Richomme > Orange/ IMT/ OLN/ CNC/ NCA/ SINA > > Network architect for innovative services Future of the Network community > member Open source Orange community manager > > > tel. +33 (0) 296 072 106 > mob. +33 (0) 637 753 326 > morgan.richo...@orange.com<mailto:morgan.richo...@orange.com> > > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc pas etre diffuses, > exploites ou copies sans autorisation. Si vous avez recu ce message par > erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les > pieces jointes. Les messages electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; they should not be distributed, > used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > > <testing evolution v1_1.pptx> _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
