Hi Alec, Several upstream projects such as tempest should run vs master. But for instance, we have to disable RefStack when switching to OpenStack Queens as it’s currently linked to testr (what about master?). I’m not sure we can introduce all new testcases in master and then we may develop in stable/gambia too. (It’s not about git operations)
From the time being, most of our testcases will fail if Snaps doesn’t support master, Healthcheck included. Most of first operations simply leverage on Snaps (user or flavor creations). Xtesting (Framework) can easily support all 3 branches. That’s out of my concerns as it’s fully cover by unit tests and we don’t need functional gating for it. Since F, Functest-core only contains all the testcase modules (OpenStack) and then all related python dependencies. Yes following both tracks simply forces a Functest and CI/CD refactoring. Cédric De : [email protected] [mailto:[email protected]] De la part de Alec Hothan (ahothan) Envoyé : lundi 23 avril 2018 17:34 À : [email protected]; David McBride; Aaron Smith; Fatih Degirmenci Cc : [email protected] Objet : Re: [opnfv-tech-discuss] [barometer] Hi Cedric, Inline… From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Sunday, April 22, 2018 at 11:17 AM To: "Alec Hothan (ahothan)" <[email protected]<mailto:[email protected]>>, David McBride <[email protected]<mailto:[email protected]>>, Aaron Smith <[email protected]<mailto:[email protected]>>, Fatih Degirmenci <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [opnfv-tech-discuss] [barometer] Hello Alec, I fully agree with you. Yes Functest team would have to work on 3 branches (and it's not only about cherry-picks): - stable/fraser (bugfixes) - stable/gambia (dev) - master (dev) Just to be clear when you say “Functest team would have to work on 3 branches”, does it only include functest core code? Or is it functest core + all the test projects that run in Functest? Since Fuctest-core pretty much focuses on orchestrating test tools and retrieving results from them or providing APIs for test tools to report results back to functest core. I would think that openstack dependencies used by functest core itself would be minimal and relatively stable. Isn’t that the case? As for managing multiple branches, if it becomes unavoidable, it is clearly more work but it is not the end of the world and pretty straightforward with git if managed properly. It may be possible only if - Functest team accepts the additionnal work, - OPNFV implements a new CI/CD job design for Test Frameworks (we can't wait for weeks if we have to maintain 3 stable branches) This definitely will need to be addressed by finding a dedicated pod for validating test projects so that new versions can be promoted to gate the 2 tracks. - we fully redefine our objectives (removing dependencies and splitting our repos would become the first steps but it seems much more relevant to verify operating VIM), It is more scalable to have each test project team manage compatibility across the release window (which could be not much work for most) than trying to have functest team do it for every test project that uses functest-core. If we manage to have that regression pod for testing projects in place, then it should not be too hard to verify how each test project fares against a limited set of openstack releases (only those that opnfv needs to support in both tracks). I said that Functest is forced to follow both tracks else the new OPNFV models can hardly be successful (it may work if OpenStack doesn't introduce big changes but who knows at the beginning of th new OpenStack release?). The "master" model is built from a falsy upward compatibility as we have already discussed during your release meetings. We shouldn't verify scenarios following VIM master by older VIM clients (Gambia) even if it could work by exceptions. Could I please get the latest results of Functest vs XCI? I'm still thinking about all technical impacts in Functest and OPNFV Features testcases: - requirements synchronization (master could be equal to VIM master or not depending on the OPNFV Features) - dependencies (at least we would have to rewrite our Healthcheck testcases due to the dependency to snaps) - duplicated work (everything has to be cherry-picked or we do skip OpenStack Queen) - possible testcase exclusions (our vnf testcases depend on middlewares such as orchestrators which couldn't support master) - etc. And we can't support 2 stable branches with the current daily job design: - no control loop - wait for days and weeks the results (it will be worse as stable scenarios will run in weekly job) If I must answer right now, we will ony support the traditionnal track. (And we won't hack Functest to verify a nested master installer (i.e. to bypass timeout issues) or to solve possible API version incompatibilities) Lots of key challenges have already been listed and from the time being only Healtcheck is selected as release criteria and as verify step in the new models. https://wiki.opnfv.org/display/SWREL/User+stories+for+OPNFV+release+artifacts http://testresults.opnfv.org/functest/gambiachallenges/ https://gerrit.opnfv.org/gerrit/#/c/55951/ Tim (Rozet) and Tim (Irnich) have already proposed their helps regarding Functional Gating which is a good start for tracking both. http://testresults.opnfv.org/functest/gates/ My concern about CI/CD optimizations (stable scenarios will run once a week) is opened whatever the tracks supported. We need to see with Fatih how new versions of a test project get pushed to the 2 CI Flows corresponding to the 2 tracks. I think the test community should aim for a workflow where a test project would mainly develop and test itself using the dedicated test pod, then provide stable tags to the CD and/or traditional track (that can run a slightly different release of openstack) for consumption by installers and feature projects. Thanks Alec _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
