Hi Chris thanks for the reply comments, answers and questions in the mail
/Morgan Le jeudi 23 août 2018 à 20:46 +0000, Christopher Donley (Chris) a écrit : Most of your wishlist items are in the VNFSDK roadmap. ok does it mean that - Testsuite deals with tests of ONAP (which may include canonical VMs) as a solution - VNFSDK deals with ONAP compatible VNF tests. VNFSDK name was a bit misleading for me as the goal is not to provide a toolkit to develop but to validate and integrate in CI/CD chains. However the description of the project is clear :)"build an ecosystem for ONAP compatible VNFs by developing tools for vendor CI/CD toolchains and developing validation and testing tools" Does it mean that you plan to develop VNF, leverage OPNFV VNF project, develop tests in addition of testing tools? #3 on your list is already part of VNFSDK (dovetail-integration repo). ok I had a quick loop, it looks like python + yaml templating but covers the same areas than robot + python extention script from testsuite it is already dockerized an inherited stuff from dovetail I need to test it :) Regarding #2, we have been focusing on packaging tests, but recently updated the functional tests (functest repo) in Beijing. Lifecycle tests have been dormant since OPEN-O, but we can resurrect as soon as we have resources. One prerequisite is a small test version of ONAP (ONAP-lite?). Ideally, we should be able to quickly instantiate it in a cloud environment, run our tests, and tear it down. We should talk about your Python framework. regarding packaging, I created a docker integrating the demo and the vvp/validation repos within a xtesting docker this 45 Mo docker (https://hub.docker.com/r/morganrol/xtesting-onap-vnfpkgcheck/) checks the different heat template available in the demo repos (traces attached) regarding the Python Framework Our first idea was to create python client for the components (like in Openstack); we used theses components to onboard then instantiate some simple opensource VNFs Our goal was to include daily instantiation of simple VNFs (vIMS, vAAA - much simpler than the demo VNF including proprietary VNFs but integrable into CI/CD chains (open source VNFs)) We can sync during a VNFSDK / Integration / ad-hoc meeting if you want We're cleaning up our test engine this release to make it easier to run Java, Python, or shell tests. xtesting includes natively the possibility to run bash, python, junit or robot tests. it also runs VNFs through a python vnf abstraction developed in OPNFV and used with open-o, cloudify, juju, openbaton, heat (currently heat/cloudify+clearwater IMS, cloudify+vyos vrouter and juju+vEPC previously we had also open-o/openbaton + clearwater IMS and openbaton = openIMS) Prior to now, we had a number of test engines coming in from different sources, using different languages. I expect by the end of Casablanca to have a stable test framework so we can focus more on the tests in Dublin. I would say that Robotframework is a very stable test framework :) The python frameworks are also probably stable... From Amsterdam to Beijing, the only unstability I can see is due to API & data model changes in the components Somehow the diversity of the test frameworks reflects the stability of the SUT and the community testing vitality. It is not surprising to see several initiatives. It is a richness for a community and it becomes a real added value when synergies occur. I will be at ONS, and happy to meet while we're there. Let's plan a slot to discuss these different aspects. Chris From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, August 23, 2018 at 5:33 AM To: Yunxia Chen <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, Chris Donley <[email protected]<mailto:[email protected]>>, Gary Wu <[email protected]<mailto:[email protected]>>, "Yang Xu (Yang, Fixed Network)" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Cc: DEBEAU Eric IMT/OLN <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [ONAP] E2E VNF functional testing Hi, a topic was dedicated to vCPE automation during Integration meeting today. I think that E2E VNF testing automation is key to stabilize ONAP For me automation means the ability to - check packaging - onboard the model - instantiate the VNF - check that resources are propely created on the target infrastructure - perform functional tests (e.g. SIP tests on a vIMS VNF) - clean the resources - report the results Automation shall be replicable on any ONAP platform without big effort except configuration file changes It means decoupling use cases/ONAP installation (no hardcoded values in ONAP installation, no lab manual specific configuration) Proprietary VNFs due to licensing models are very hard to integrate in CI/CD chains (which shall not prevent ambitious use cases to integrate such VNFs) It is clearly challenging As far as I know there are currently several initiatives aiming to provide ONAP VNF E2E automatic testing 1) integration project with the different cases including vCPE (integration team) https://git.onap.org/testsuite/tree/robot https://git.onap.org/demo/tree/ https://git.onap.org/integration/tree/test Using Robot you can already onboard and instantiate several VNFs (vLB, vFW, vVG, vCPE): https://git.onap.org/testsuite/tree/robot/testsuites/model-distribution.robot Some tests were integrated in CI and even use for some robustnes tests (I did not find the jenkins url corresponding to daily runs of there testcases) 2) VNFSDK/VPP as part of VNF certification program (contact C.Donley) information shared during LFN C&C meeting - the idea is to work on a verification program for VNF as the one initiated on the infrastructure by OPNFV VNFs have been announced for September only packaging check are planned for Casablanca as a first step but onboarding/lifecycle tests/functional tests are mentioned in the roadmap dovetail framework is mentioned as the tooling to launch the tests 3) Amdocs demo shared during last ONS summit (contact: Moshe Hoadley) It is also linked to 2) as a poc of ONAP Life cycle testing through dovetail https://wiki.onap.org/display/DW/VNFTEST+integration+with+DOVETAIL?preview=/28377754/28377756/onap-opnfv-collaboration-demo-21_march_2018.mp4 I did not find de code associated to the demo 4) OTF - Open Test Framework (contact: Kevin Wan) https://wiki.onap.org/display/DW/OTF+-+Open+Test+Framework project in progress / still no official repo 5) Light python framework shared during last ONS (contact: Morgan Richomme) This is a python framework allowing to onboard and instantiate VNFs. It can be used with Clearwater vIMS, vMRF (proprietary), vAAA (freeradius), ONAP vFW The code is available here: https://gitlab.com/Orange-OpenSource/onap-tests The plan is to integrate it in xtesting project (as robot healthcheck tests have been integrated) xtesting (https://xtesting.readthedocs.io/en/latest/) is a light framework to harmonize the way of launching the tests (launch, get results, report results), supporting Robot, python, bash, junit based tests. it is used in OPNFV and easy to integrate in CI chains (we use it internally from our gitlab pipelines). 6) Postman collection can be used for automation 7) ... As we can see there are several solutions adressing the same goals It probably would make sense to see if possible synergies are possible Would it make sense to plan a f2f meeting during next ONS summit? /Morgan _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12054): https://lists.onap.org/g/onap-discuss/message/12054 Mute This Topic: https://lists.onap.org/mt/24929469/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
 sudo docker run xtesting-onap-vnfpkgchk 2018-08-23 16:09:55,897 - xtesting.ci.run_tests - INFO - Deployment description: +--------------------------------------+----------------------------------------------------------+ | ENV VAR | VALUE | +--------------------------------------+----------------------------------------------------------+ | BUILD_TAG | | | ENERGY_RECORDER_API_URL | http://energy.opnfv.fr/resources | | ENERGY_RECORDER_API_PASSWORD | | | CI_LOOP | daily | | TEST_DB_URL | http://testresults.opnfv.org/test/api/v1/results | | INSTALLER_TYPE | unknown | | DEPLOY_SCENARIO | os-nosdn-nofeature-noha | | ENERGY_RECORDER_API_USER | | | NODE_NAME | | +--------------------------------------+----------------------------------------------------------+ 2018-08-23 16:09:55,900 - xtesting.ci.run_tests - INFO - TESTS TO BE EXECUTED: +----------------------+---------------+---------------------+-------------------------------------+ | TIERS | ORDER | DESCRIPTION | TESTCASES | +----------------------+---------------+---------------------+-------------------------------------+ | onap_vnf_pkg | 1 | | fw cpe lb fwcl lbms vg onap | +----------------------+---------------+---------------------+-------------------------------------+ 2018-08-23 16:09:55,921 - xtesting.ci.run_tests - INFO - Loading test case 'fw'... 2018-08-23 16:09:55,969 - xtesting.ci.run_tests - INFO - Running test case 'fw'... 2018-08-23 16:10:04,355 - xtesting.core.feature - INFO - Test result is stored in '/var/lib/xtesting/results/fw.log' 2018-08-23 16:10:04,356 - xtesting.core.feature - ERROR - Execute command: pytest /src/valid_script/ice_validator/tests --cache-clear --runxfail --template-dir=/src/demo/heat/vFW failed 2018-08-23 16:10:04,356 - xtesting.ci.run_tests - INFO - Test result: +-------------------+-----------------+------------------+----------------+ | TEST CASE | PROJECT | DURATION | RESULT | +-------------------+-----------------+------------------+----------------+ | fw | demo | 00:08 | FAIL | +-------------------+-----------------+------------------+----------------+ 2018-08-23 16:10:04,358 - xtesting.ci.run_tests - ERROR - The test case 'fw' failed. 2018-08-23 16:10:04,378 - xtesting.ci.run_tests - INFO - Loading test case 'cpe'... 2018-08-23 16:10:04,396 - xtesting.ci.run_tests - INFO - Running test case 'cpe'... 2018-08-23 16:10:05,313 - xtesting.core.feature - INFO - Test result is stored in '/var/lib/xtesting/results/cpe.log' 2018-08-23 16:10:05,314 - xtesting.core.feature - ERROR - Execute command: pytest /src/valid_script/ice_validator/tests --cache-clear --runxfail --template-dir=/src/demo/heat/vCPE failed 2018-08-23 16:10:05,314 - xtesting.ci.run_tests - INFO - Test result: +-------------------+-----------------+------------------+----------------+ | TEST CASE | PROJECT | DURATION | RESULT | +-------------------+-----------------+------------------+----------------+ | cpe | demo | 00:01 | FAIL | +-------------------+-----------------+------------------+----------------+ 2018-08-23 16:10:05,316 - xtesting.ci.run_tests - ERROR - The test case 'cpe' failed. 2018-08-23 16:10:05,334 - xtesting.ci.run_tests - INFO - Loading test case 'lb'... 2018-08-23 16:10:05,351 - xtesting.ci.run_tests - INFO - Running test case 'lb'... 2018-08-23 16:10:16,064 - xtesting.core.feature - INFO - Test result is stored in '/var/lib/xtesting/results/lb.log' 2018-08-23 16:10:16,064 - xtesting.core.feature - ERROR - Execute command: pytest /src/valid_script/ice_validator/tests --cache-clear --runxfail --template-dir=/src/demo/heat/vLB failed 2018-08-23 16:10:16,065 - xtesting.ci.run_tests - INFO - Test result: +-------------------+-----------------+------------------+----------------+ | TEST CASE | PROJECT | DURATION | RESULT | +-------------------+-----------------+------------------+----------------+ | lb | demo | 00:11 | FAIL | +-------------------+-----------------+------------------+----------------+ 2018-08-23 16:10:16,067 - xtesting.ci.run_tests - ERROR - The test case 'lb' failed. 2018-08-23 16:10:16,084 - xtesting.ci.run_tests - INFO - Loading test case 'fwcl'... 2018-08-23 16:10:16,102 - xtesting.ci.run_tests - INFO - Running test case 'fwcl'... 2018-08-23 16:10:17,016 - xtesting.core.feature - INFO - Test result is stored in '/var/lib/xtesting/results/fwcl.log' 2018-08-23 16:10:17,017 - xtesting.core.feature - ERROR - Execute command: pytest /src/valid_script/ice_validator/tests --cache-clear --runxfail --template-dir=/src/demo/heat/vFWCL failed 2018-08-23 16:10:17,017 - xtesting.ci.run_tests - INFO - Test result: +-------------------+-----------------+------------------+----------------+ | TEST CASE | PROJECT | DURATION | RESULT | +-------------------+-----------------+------------------+----------------+ | fwcl | demo | 00:01 | FAIL | +-------------------+-----------------+------------------+----------------+ 2018-08-23 16:10:17,019 - xtesting.ci.run_tests - ERROR - The test case 'fwcl' failed. 2018-08-23 16:10:17,036 - xtesting.ci.run_tests - INFO - Loading test case 'lbms'... 2018-08-23 16:10:17,053 - xtesting.ci.run_tests - INFO - Running test case 'lbms'... 2018-08-23 16:10:28,082 - xtesting.core.feature - INFO - Test result is stored in '/var/lib/xtesting/results/lbms.log' 2018-08-23 16:10:28,083 - xtesting.core.feature - ERROR - Execute command: pytest /src/valid_script/ice_validator/tests --cache-clear --runxfail --template-dir=/src/demo/heat/vLBMS failed 2018-08-23 16:10:28,083 - xtesting.ci.run_tests - INFO - Test result: +-------------------+-----------------+------------------+----------------+ | TEST CASE | PROJECT | DURATION | RESULT | +-------------------+-----------------+------------------+----------------+ | lbms | demo | 00:11 | FAIL | +-------------------+-----------------+------------------+----------------+ 2018-08-23 16:10:28,085 - xtesting.ci.run_tests - ERROR - The test case 'lbms' failed. 2018-08-23 16:10:28,101 - xtesting.ci.run_tests - INFO - Loading test case 'vg'... 2018-08-23 16:10:28,119 - xtesting.ci.run_tests - INFO - Running test case 'vg'... 2018-08-23 16:10:35,937 - xtesting.core.feature - INFO - Test result is stored in '/var/lib/xtesting/results/vg.log' 2018-08-23 16:10:35,938 - xtesting.core.feature - ERROR - Execute command: pytest /src/valid_script/ice_validator/tests --cache-clear --runxfail --template-dir=/src/demo/heat/vG failed 2018-08-23 16:10:35,938 - xtesting.ci.run_tests - INFO - Test result: +-------------------+-----------------+------------------+----------------+ | TEST CASE | PROJECT | DURATION | RESULT | +-------------------+-----------------+------------------+----------------+ | vg | demo | 00:08 | FAIL | +-------------------+-----------------+------------------+----------------+ 2018-08-23 16:10:35,940 - xtesting.ci.run_tests - ERROR - The test case 'vg' failed. 2018-08-23 16:10:35,957 - xtesting.ci.run_tests - INFO - Loading test case 'onap'... 2018-08-23 16:10:35,975 - xtesting.ci.run_tests - INFO - Running test case 'onap'... 2018-08-23 16:10:50,334 - xtesting.core.feature - INFO - Test result is stored in '/var/lib/xtesting/results/onap.log' 2018-08-23 16:10:50,334 - xtesting.core.feature - ERROR - Execute command: pytest /src/valid_script/ice_validator/tests --cache-clear --runxfail --template-dir=/src/demo/heat/ONAP failed 2018-08-23 16:10:50,335 - xtesting.ci.run_tests - INFO - Test result: +-------------------+-----------------+------------------+----------------+ | TEST CASE | PROJECT | DURATION | RESULT | +-------------------+-----------------+------------------+----------------+ | onap | demo | 00:14 | FAIL | +-------------------+-----------------+------------------+----------------+ 2018-08-23 16:10:50,337 - xtesting.ci.run_tests - ERROR - The test case 'onap' failed. 2018-08-23 16:10:50,337 - xtesting.ci.run_tests - INFO - Xtesting report: +-------------------+-----------------+----------------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +-------------------+-----------------+----------------------+------------------+----------------+ | fw | demo | onap_vnf_pkg | 00:08 | FAIL | | cpe | demo | onap_vnf_pkg | 00:01 | FAIL | | lb | demo | onap_vnf_pkg | 00:11 | FAIL | | fwcl | demo | onap_vnf_pkg | 00:01 | FAIL | | lbms | demo | onap_vnf_pkg | 00:11 | FAIL | | vg | demo | onap_vnf_pkg | 00:08 | FAIL | | onap | demo | onap_vnf_pkg | 00:14 | FAIL | +-------------------+-----------------+----------------------+------------------+----------------+ 2018-08-23 16:10:50,341 - xtesting.ci.run_tests - INFO - Execution exit value: Result.EX_ERROR
