Hi, On 24.03.2021 19:35, [email protected] wrote: > *TL; DR:* > ------------ > xtesting dockers (tests) and xtesting-onap (test launchers) should have been > frozen before RC0..so next time we have to think to that.. > Integration shall guarantee a stable testing baseline to OOM to gate > confortably.. > > Full Text > ----------- > > As indicated by Sylvain, the tests in gate are used as a merge criteria > > so to avoid any turbulence, I should have frozen > - the xtesting dockers i.e the tests: no new test - I merged dcaemod > yesterday - it was false negative, the test was OK but was reported failed > for few hours (it was not declared in DB, I declared it this morning) > - xtesting-onap: I made an error this afternoon. I merged a MR dealing with > weekly rule for tern test on weekly which broke the testing CI part - I just > saw the approval..and saw the CI issue few minutes too late, I reverted it > quickly but we missed 2 cycles so ~ 4h. Regarding the numbers of patches to > be checked before RC0 and the gating resources, I should not have even tried. > (I oom_redeployed the 2 faulty gates). > > it means also that we have to sync on the chronology of the integration of > the tests in CI. > I assume that we should guarantee a stable test baseline to OOM between > before the RC0.
Well in perfect world probably yes but we all know that tests also needs to be fixed and updated when teams are introducing new containers > > - Old tests (version N-1) used to verify that there is no regression are not > a real problem, they continue to work or need adaptations but we can detect > it. > - New tests (version N) can deal with > -- new features (needs the ONAP patch merges before being valid - we had a > misalignment with policy, the tests were updated but the patch not merged in > OOM for some weeks, the policy healthcheck was always FAIL) That's obviously a chicken and egg problem that can be solved in many ways but either of them would require some changes in the gating One of them that I can imagine is to keep the reference to xtesting docker version in then OOM repo. To updates xtesting dockers you need oom commit, as you need oom commit every change of those dockers would be gated. Additionally everyone who introduces or modifies a feature and needs to user newer version of xtesting docker may update it in the same commit as the functional change. Not sure if it's the best idea but still it's at least some idea... What do you think? > -- old features that were not covered so far by existing tests (usually not a > problem as it can be integrated in master eraly during the dev process) > > that is why I try to share early in time the CI evolution some weeks before > the RC0 => Honolulu CI evolution shared on the 17th of February: > https://protect2.fireeye.com/v1/url?k=c0ae9ad2-9f35a35d-c0af119d-0cc47a31307c-9d5162ca6ac8b403&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fwiki.onap.org%2Fdownload%2Fattachments%2F6593670%2Fhonolulu_ci_evolution_08022021.pdf%3Fversion%3D1%26modificationDate%3D1612772982000%26api%3Dv2 > > but it is not easy to plan for new tests when the feature will be available > > Regarding the tests for Honolulu > - dcaemod (indicated by KK during the February meeting) is fully integrated > in CI > - tern is under integration > - pnf-macro under test (no new ONAP feature, covering SO macro flow + > multicloud + simulator management from onaptests) > - basic-clamp under test, we are not far but the lack of stability on master > prevented to finalize the test > - CDS regression tests: as discussed I wonder if it would not make sense to > add in in CSIT test first > - stability tests: need also a stable weekly master. For Honolulu we will > probably not integrate it in CI (problematic to launch long duration tests > from CI currently under investigation with the tern test) > > and very recently due to the SO stuck requests, Michal and myself started > working on basic_vm_macro and basic_cnf_macro (old feature/new tests). > Note this behavior has been observed in pnf-registrate which is using the > macro mode > > conclusions Personally I still believe that main fault is on us as OOM committers. We've been really to gentle with the review and allow patches to be merged trying to persuade ourselfs that we know the root cause of the issue, not taking into account that this one issue that we know may hide tons of other issues. > I would recommend for next releases > - to freeze the tests/CI Test launcher at RC0-n weeks..n to be defined I'd say that the deadline here would be same as end of coding for projects. > - to integrate new tests based on new features only on master when the branch > of the new version is available (RC0) Well I'm not convinced here. I'd allow those to be added at any time but maybe in the beginning we should have a separate category for them like "testing" or sth so that they are there but we just don't take them into account while merging patches until they are move to other category > e.g. if we would consider such case today integrate in Master when Honolulu > is available (we can always run the tests manually on the staging/daily and > prepare the MR for the CI integration) > - to allow integration of new tests (old features) before the frozen period > then after the RC0 As above. Personally I'd prefer to have some kind of control over xtesting dockers so that we are not surprised nor affected by their changes. Currently oom version is fully independent from them. I believe it woudl be great to have even a simple file like xtesting_version in the OOM repo that would be used to indicate which version of tests should be used. This would allow your team to work smoothly and integrate tests whenever they want and then just update that file whenever we need or you are ready to introduce new tests. Apart from what you mentioned I believe that main issue that we need to tackle is the lack of prompt bug fixing in projects. Some of the issues that we've been fixing recently had jira tickets that were created in November and still not fixed. Even I personallyhave a patch for OOM that was created half a year ago and I cannot merge it because of bug that is in VID that is unfixed for more than half of the year. And it's same for many tickets. We (OOM & integration team) observe multiple issues and fill in jiras but often there is no interest from the project to fix those for a very long time. There is a lot of work being done to create new code but not many people care to make sure that what they already have works fine. I believe that with such attitude our quality and perception of ONAP in general really hurts. > > what is your view? > > ________________________________________ > De : [email protected] [[email protected]] de la part de Sylvain > Desbureaux via lists.onap.org [[email protected]] > Envoyé : mercredi 24 mars 2021 18:34 > À : Krzysztof Opasiak > Cc : [email protected]; Lefevre, Catherine; Seshu m; TIMONEY, DAN; > [email protected]; 徐冉; HAHN III, JIM; Fiachra Corcoran; > [email protected]; onap-tsc > Objet : Re: [onap-tsc] [IMPORTANT] OOM status update for RC0 > > Also note that after the break of OOM master branch these last weeks and in > order to not break it again, we (the OOM committers) have decided to merge a > patch only if the code is OK and if the following rules on gate result is met: > * changed component must have its pods running > * if the patch changes one of « core » ONAP component (AAI, DMAAP, SDC, SDNC, > SO and VID): we mandate 100% healthchecks and 100% e2e tests to pass > * for other components, we allow one healthcheck and one e2e test to fail > > As the predictability of an ONAP deployment is not perfect, that a gate takes > ~200min, that we have only 4 system gating right now (azure systems tends to > fails fast these last days so I’ve decided to remove them for now), try to > push your changes as soon as possible if you want to have these changes in > Honolulu release. > > Best regards, > Sylvain > >> Le 24 mars 2021 à 18:17, Krzysztof Opasiak <[email protected]> a écrit : >> >> Team! >> >> *TL; DR:* >> >> OOM patch submitters, please fix your patches/respond to our comments >> ASAP. Less than 24h has left till RC0 and functional changes not merged >> before RC0 will be postponed to Istanbul. >> >> This is call to all OOM patch submitter but especially to projects: SO, >> SDNC, VFC, UUI, Policy, DMAAP. >> >> *Full text:* >> >> Heads-up, there is less than 24 hours left until proposed RC0 date so >> let me give you the status update for RC0. Please treat it as a reminder >> for projects who would like to have their functional changes to be >> included in the Honolulu release. >> >> We finally managed to process all 3 categories (bugfix, beforeM3 and >> afterM3) that we had previously. All patches that are still in our >> pipeline have been divided into 2 categories using tags: >> >> honoluluCandidates - for patches that could be merged before RC0 >> https://protect2.fireeye.com/v1/url?k=aac407ec-f55f3e63-aac58ca3-0cc47a31307c-65c404bf990abdbb&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fq%2Fhashtag%3A%2522honoluluCandidate%2522%2Bstatus%3Aopen >> >> istanbul - for patches that we want to delay till Istanbul (effectively >> until we branch out honolulu). >> https://protect2.fireeye.com/v1/url?k=af181b56-f08322d9-af199019-0cc47a31307c-be63ae1602577644&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fq%2Fhashtag%3A%2522istanbul%2522%2Bstatus%3Aopen >> >> So for Honolulu we currently have 18 candidates to be included. >> >> Out of those, below patches are waiting for the OOM team to review and gate: >> https://protect2.fireeye.com/v1/url?k=851b55f2-da806c7d-851adebd-0cc47a31307c-a5a0dc446760407a&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F119125 >> https://protect2.fireeye.com/v1/url?k=a106a209-fe9d9b86-a1072946-0cc47a31307c-e4e0bfa26d3b9c40&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F117808 >> https://protect2.fireeye.com/v1/url?k=4227b6cb-1dbc8f44-42263d84-0cc47a31307c-7eb6d184886ce440&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F118925 >> https://protect2.fireeye.com/v1/url?k=ce689f9f-91f3a610-ce6914d0-0cc47a31307c-c859dc10838d6638&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F119488 >> https://protect2.fireeye.com/v1/url?k=9f76553a-c0ed6cb5-9f77de75-0cc47a31307c-9b6d7341429155ac&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F118248 >> https://protect2.fireeye.com/v1/url?k=d8d58a96-874eb319-d8d401d9-0cc47a31307c-4bfef9f8f6c1ffa4&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F117395 >> >> It seems to be save to say that till tomorrow TSC meeting we should be >> able to handle most of them unless some unexpected failures occur and we >> will need submitters to fix their patches. >> >> Apart from that, below patches are waiting for authors to fix >> them/respond to our review: >> >> https://protect2.fireeye.com/v1/url?k=5614a20b-098f9b84-56152944-0cc47a31307c-c956bf78870f6ec4&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F113414 >> https://protect2.fireeye.com/v1/url?k=f797e4c9-a80cdd46-f7966f86-0cc47a31307c-b1eff49466feacd6&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F114380 >> https://protect2.fireeye.com/v1/url?k=255f003d-7ac439b2-255e8b72-0cc47a31307c-b17701096825dafa&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F118995 >> https://protect2.fireeye.com/v1/url?k=29a077d2-763b4e5d-29a1fc9d-0cc47a31307c-8445c680c11d7b80&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F118331 >> https://protect2.fireeye.com/v1/url?k=fc7a39e0-a3e1006f-fc7bb2af-0cc47a31307c-65173633068403fb&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F118284 >> https://protect2.fireeye.com/v1/url?k=5b97df07-040ce688-5b965448-0cc47a31307c-17ae0004c6f15e1b&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F118510 >> https://protect2.fireeye.com/v1/url?k=72d7958f-2d4cac00-72d61ec0-0cc47a31307c-b13f3c935af8261e&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F119102 >> https://protect2.fireeye.com/v1/url?k=49c4360b-165f0f84-49c5bd44-0cc47a31307c-1cf541dbdc349082&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F119012 >> https://protect2.fireeye.com/v1/url?k=0b721965-54e920ea-0b73922a-0cc47a31307c-749f2f327b8e1a12&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F119124 >> https://protect2.fireeye.com/v1/url?k=960cbc46-c99785c9-960d3709-0cc47a31307c-ec05e63241714dbb&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F118602 >> https://protect2.fireeye.com/v1/url?k=97ca4422-c8517dad-97cbcf6d-0cc47a31307c-37947a81f2524bec&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fgerrit.onap.org%2Fr%2Fc%2Foom%2F%2B%2F119309 >> >> This means that everything is in your hands now. You need to fix your >> review ASAP in order to give us time to gate and merge your patch before >> reaching RC0 as according to new release cadence all OOM reviews that >> contain functional changes and cannot be merged before RC0 are going to >> be delayed until Istanbul (we'll just merge it to master as soon as you >> make it work). >> >> Special heads up for: >> SO >> SDNC >> VFC >> UUI >> Policy >> DMAAP >> >> Major Honolulu container updates for those projects are still in the >> review and waiting to be fixed. Especially for DMAAP which still >> contains the hardcoded cert and if we don't get a working version of >> your review soon then we'll have a certificate issue in master within >> few days. >> >> Best regards, >> -- >> Krzysztof Opasiak >> Samsung R&D Institute Poland >> Samsung Electronics > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > > > > > > > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > -- Krzysztof Opasiak Samsung R&D Institute Poland Samsung Electronics -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#23057): https://lists.onap.org/g/onap-discuss/message/23057 Mute This Topic: https://lists.onap.org/mt/81584261/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
