Morgan, Jenkins jobs for daily installation and regression tests are off the jenkins/grafana site that Gary runs.
http://onapci.org/jenkins/ http://onapci.org/grafana/ http://onapci.org/grafana/d/8cGRqBOmz/daily-summary?orgId=1 We could start the discussion even before ONS. I would encourage us to look at incorporating testing into the testsuite project rather than a new project but I may be mis-understanding what you mean by xtesting. Installation of ONAP is generally in the demo or integration repos (demo is more for the test vnfs going forward) . Testing is in testsuite and has end to end tests as well as domain focused tests for onboarding, instantiation and closed loop As you indicated we break up the problem into 3 domains: Demo project -> test vnfs and miscellaneous scripts needed to run demo’s at events. This is also where we have the scripts to build images from opensource tools (e.g. Intel’s vCPE VNFs) Integraiton project -> resources/artifacts supporting the instllation of ONAP via OOM and HEAT – source for environment specific files to drive jenkins CI etec. Testsuite project -> automated testing of ONAP in its role of onboard, instantiate and control loop using the ONAP instance deployed by Integration and VNFs in the demo project . Testsuite includes mostly robot framework tests but also python based tests/interfaces are allowed/encouraged where they make sense. Brian From: [email protected] <[email protected]> On Behalf Of Morgan Richomme Sent: Thursday, August 23, 2018 8:34 AM To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; PLATANIA, MARCO <[email protected]>; WAN, KEVIN <[email protected]>; FREEMAN, BRIAN D <[email protected]> Cc: DEBEAU Eric IMT/OLN <[email protected]>; [email protected] Subject: [onap-discuss] [ONAP] E2E VNF functional testing Hi, a topic was dedicated to vCPE automation during Integration meeting today. I think that E2E VNF testing automation is key to stabilize ONAP For me automation means the ability to - check packaging - onboard the model - instantiate the VNF - check that resources are propely created on the target infrastructure - perform functional tests (e.g. SIP tests on a vIMS VNF) - clean the resources - report the results Automation shall be replicable on any ONAP platform without big effort except configuration file changes It means decoupling use cases/ONAP installation (no hardcoded values in ONAP installation, no lab manual specific configuration) Proprietary VNFs due to licensing models are very hard to integrate in CI/CD chains (which shall not prevent ambitious use cases to integrate such VNFs) It is clearly challenging As far as I know there are currently several initiatives aiming to provide ONAP VNF E2E automatic testing 1) integration project with the different cases including vCPE (integration team) https://git.onap.org/testsuite/tree/robot<https://urldefense.proofpoint.com/v2/url?u=https-3A__git.onap.org_testsuite_tree_robot&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Gg4gmC80V5yeUcO8S6LA5ldv_A3lRsY4VWdqEBUu4yM&s=6ouN07lzctI0n1yC3En7jBLpebIxZLOjluTqClZ3n1M&e=> https://git.onap.org/demo/tree/<https://urldefense.proofpoint.com/v2/url?u=https-3A__git.onap.org_demo_tree_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Gg4gmC80V5yeUcO8S6LA5ldv_A3lRsY4VWdqEBUu4yM&s=EHox59SMJPEmO77AhWuvesEjFruG7s7hK7dsLezTzNk&e=> https://git.onap.org/integration/tree/test<https://urldefense.proofpoint.com/v2/url?u=https-3A__git.onap.org_integration_tree_test&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Gg4gmC80V5yeUcO8S6LA5ldv_A3lRsY4VWdqEBUu4yM&s=pRnYTGcTKyr5b4bpAxYSRSCumL2AXej6ayLBCdl4GTA&e=> Using Robot you can already onboard and instantiate several VNFs (vLB, vFW, vVG, vCPE): https://git.onap.org/testsuite/tree/robot/testsuites/model-distribution.robot<https://urldefense.proofpoint.com/v2/url?u=https-3A__git.onap.org_testsuite_tree_robot_testsuites_model-2Ddistribution.robot&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Gg4gmC80V5yeUcO8S6LA5ldv_A3lRsY4VWdqEBUu4yM&s=SHWJyQF6GaJpH4gedp2HQbwMi3WUdn6evwD4FkzlwaY&e=> Some tests were integrated in CI and even use for some robustnes tests (I did not find the jenkins url corresponding to daily runs of there testcases) 2) VNFSDK/VPP as part of VNF certification program (contact C.Donley) information shared during LFN C&C meeting - the idea is to work on a verification program for VNF as the one initiated on the infrastructure by OPNFV VNFs have been announced for September only packaging check are planned for Casablanca as a first step but onboarding/lifecycle tests/functional tests are mentioned in the roadmap dovetail framework is mentioned as the tooling to launch the tests 3) Amdocs demo shared during last ONS summit (contact: Moshe Hoadley) It is also linked to 2) as a poc of ONAP Life cycle testing through dovetail https://wiki.onap.org/display/DW/VNFTEST+integration+with+DOVETAIL?preview=/28377754/28377756/onap-opnfv-collaboration-demo-21_march_2018.mp4<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_VNFTEST-2Bintegration-2Bwith-2BDOVETAIL-3Fpreview-3D_28377754_28377756_onap-2Dopnfv-2Dcollaboration-2Ddemo-2D21-5Fmarch-5F2018.mp4&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Gg4gmC80V5yeUcO8S6LA5ldv_A3lRsY4VWdqEBUu4yM&s=O1qgD_BqOhiAtXggQhgsq9c2FNxsJyGaoS28Btzo_7k&e=> I did not find de code associated to the demo 4) OTF - Open Test Framework (contact: Kevin Wan) https://wiki.onap.org/display/DW/OTF+-+Open+Test+Framework<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_OTF-2B-2D-2BOpen-2BTest-2BFramework&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Gg4gmC80V5yeUcO8S6LA5ldv_A3lRsY4VWdqEBUu4yM&s=fJTYlWIxgXNx7S0SsS-iPGUSKlmXtRzpToDoX8ipuMQ&e=> project in progress / still no official repo 5) Light python framework shared during last ONS (contact: Morgan Richomme) This is a python framework allowing to onboard and instantiate VNFs. It can be used with Clearwater vIMS, vMRF (proprietary), vAAA (freeradius), ONAP vFW The code is available here: https://gitlab.com/Orange-OpenSource/onap-tests<https://urldefense.proofpoint.com/v2/url?u=https-3A__gitlab.com_Orange-2DOpenSource_onap-2Dtests&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Gg4gmC80V5yeUcO8S6LA5ldv_A3lRsY4VWdqEBUu4yM&s=wTY2-Nv7tXdCwGBn-doqNJhTTlwDfmpLqEUkiaMoouY&e=> The plan is to integrate it in xtesting project (as robot healthcheck tests have been integrated) xtesting (https://xtesting.readthedocs.io/en/latest/<https://urldefense.proofpoint.com/v2/url?u=https-3A__xtesting.readthedocs.io_en_latest_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Gg4gmC80V5yeUcO8S6LA5ldv_A3lRsY4VWdqEBUu4yM&s=kRV--4bLNtdxiPflucvmoPkW9pGzniBgwUTfJQGmagI&e=>) is a light framework to harmonize the way of launching the tests (launch, get results, report results), supporting Robot, python, bash, junit based tests. it is used in OPNFV and easy to integrate in CI chains (we use it internally from our gitlab pipelines). 6) Postman collection can be used for automation 7) ... As we can see there are several solutions adressing the same goals It probably would make sense to see if possible synergies are possible Would it make sense to plan a f2f meeting during next ONS summit? /Morgan _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12039): https://lists.onap.org/g/onap-discuss/message/12039 Mute This Topic: https://lists.onap.org/mt/24929469/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
