Hi, the Orange ONAP Openlab reinstallation is completed. We reinstalled it in Beijing version and will upgrade in Casablanca in December.
I re-initially pushed all the SSH keys on Tuesday (our automation procedures are now quiete efficient) but if all the healthcheck tests were OK, it was not possible to deploy a VNFs.. This problem was detected using our xtesting-onap-vnf use cases. The cause was identified: the behavior of our keystone wrapper (as our OpenStack is based on Queens, keystone v2 is no more supported) changed. It was still trying to deal with Keystone v2. This was fixed, then the SDNC listener had to be restarted before running successfully the tests. To be sure we reinstalled the ONAP beijing last night with the fix and re-tried the healthcheck then the xtesting onap tests with success this morning (CET). You should now be able to connect to the jumphost. Lessons learned: - the platform (ONAP Beijing installed with OOM) has been relatively stable since September (some module restarts needed time to time but no deadlock) - we moved from 2.0.0 to Beijing tag and got the GUI for the AA&I (always got a 503 so far) - the critical power outage (long duration time outage with finally power interuption during a week where most of our small team were in PTO) triggered severe damages on our discs ... and by consequence on our Ceph cluster. We cumulated different problems when re-powering up the system (Cassandra and Ceph are not the best friends...) but were able after several operations on discs to recover the cluster. However most of the ONAP DB were corrupted. That is why it was decided to redeploy ONAP (we did not reinstall the whole infrastructure, only rancher + ONAP). Enjoy testing Regards Morgan NB: traces healthcheck 2018-11-15 09:42:59,470 - xtesting.ci.run_tests - INFO - Xtesting report: +---------------------------+------------------+--------------+------------------+----------------+ | TEST CASE | PROJECT | TIER | DURATION | RESULT | +---------------------------+------------------+--------------+------------------+----------------+ | robot_healthcheck | functest | onap | 00:04 | PASS | | robot_api | functest | onap | 00:04 | PASS | | robot_dcae | functest | onap | 00:00 | PASS | | robot_multicloud | functest | onap | 00:00 | PASS | | robot_3rdparty | functest | onap | 00:01 | PASS | +---------------------------+------------------+--------------+------------------+----------------+ automation basic vm (onboarding + instantiation) 2018-11-15 09:59:43,826 - onap_tests.scenario.e2e - INFO - VNF ubuntu16: test initialized >>> test.execute() 2018-11-15 09:59:51,898 - onap_tests.scenario.e2e - INFO - Start onboarding 2018-11-15 09:59:52,978 - onap_tests.scenario.onboard - INFO - Create vendor 2018-11-15 09:59:54,074 - onap_tests.scenario.onboard - INFO - Check and update vendor 2018-11-15 09:59:54,304 - onap_tests.scenario.onboard - INFO - vendor submit 2018-11-15 09:59:54,633 - onap_tests.scenario.onboard - INFO - Check and update vendor 2018-11-15 09:59:54,855 - onap_tests.scenario.onboard - INFO - Debut creation VSP ubuntu16_VSP 2018-11-15 09:59:55,216 - onap_tests.scenario.onboard - INFO - Check and update vendor info 2018-11-15 09:59:55,216 - onap_tests.scenario.onboard - INFO - Create VSP ubuntu16_VSP 2018-11-15 09:59:56,446 - onap_tests.scenario.onboard - INFO - Check and update VSP 2018-11-15 09:59:56,663 - onap_tests.scenario.onboard - INFO - upload Heat files to VSP ubuntu16_VSP 2018-11-15 09:59:56,965 - onap_tests.scenario.onboard - INFO - Check and update VSP 2018-11-15 09:59:57,177 - onap_tests.scenario.onboard - INFO - validate Heat files 2018-11-15 09:59:59,917 - onap_tests.scenario.onboard - INFO - Check and update VSP 2018-11-15 10:00:00,131 - onap_tests.scenario.onboard - INFO - vsp commit 2018-11-15 10:00:01,356 - onap_tests.scenario.onboard - INFO - Check and update VSP 2018-11-15 10:00:01,601 - onap_tests.scenario.onboard - INFO - vsp submit 2018-11-15 10:00:03,690 - onap_tests.scenario.onboard - INFO - Check and update VSP 2018-11-15 10:00:03,913 - onap_tests.scenario.onboard - INFO - Create CSAR package 2018-11-15 10:00:04,513 - onap_tests.scenario.onboard - INFO - Check and update VSP 2018-11-15 10:00:06,141 - onap_tests.scenario.onboard - INFO - Check and update VSP info 2018-11-15 10:00:06,141 - onap_tests.scenario.onboard - INFO - Create VF ubuntu16_VF 2018-11-15 10:00:06,141 - onap_tests.scenario.onboard - INFO - VF using VSP name ubuntu16_VSP 2018-11-15 10:00:06,141 - onap_tests.scenario.onboard - INFO - VF using VSP id b83764c5ac8b492cb40c954753f90d09 2018-11-15 10:00:06,141 - onap_tests.scenario.onboard - INFO - VF using VSP csar_uuid b83764c5ac8b492cb40c954753f90d09 2018-11-15 10:00:11,877 - onap_tests.scenario.onboard - INFO - Check and update VF ubuntu16_VF 2018-11-15 10:00:12,649 - onap_tests.scenario.onboard - INFO - vf checkin ubuntu16_VF 2018-11-15 10:00:14,611 - onap_tests.scenario.onboard - INFO - Check and update VF ubuntu16_VF 2018-11-15 10:00:15,419 - onap_tests.scenario.onboard - INFO - vf submit 2018-11-15 10:00:17,907 - onap_tests.scenario.onboard - INFO - Check and update VF 2018-11-15 10:00:18,727 - onap_tests.scenario.onboard - INFO - vf Start Certification 2018-11-15 10:00:20,601 - onap_tests.scenario.onboard - INFO - Check and update VF 2018-11-15 10:00:21,389 - onap_tests.scenario.onboard - INFO - vf certify 2018-11-15 10:00:24,300 - onap_tests.scenario.onboard - INFO - Check and update VF 2018-11-15 10:00:25,166 - onap_tests.scenario.onboard - INFO - vf unique_id update 2018-11-15 10:00:26,022 - onap_tests.scenario.onboard - INFO - Check and load VF info 2018-11-15 10:00:26,992 - onap_tests.scenario.onboard - INFO - Create Service 2018-11-15 10:00:29,448 - onap_tests.scenario.onboard - INFO - Check and update Service 2018-11-15 10:00:30,014 - onap_tests.scenario.onboard - INFO - Add VF ubuntu16_VF to service 2018-11-15 10:00:32,522 - onap_tests.scenario.onboard - INFO - service checkin 2018-11-15 10:00:34,313 - onap_tests.scenario.onboard - INFO - Check and update service 2018-11-15 10:00:34,781 - onap_tests.scenario.onboard - INFO - service submit 2018-11-15 10:00:38,564 - onap_tests.scenario.onboard - INFO - Check and update service 2018-11-15 10:00:38,978 - onap_tests.scenario.onboard - INFO - service Start Certification 2018-11-15 10:00:40,814 - onap_tests.scenario.onboard - INFO - Check and update service 2018-11-15 10:00:41,321 - onap_tests.scenario.onboard - INFO - service certify 2018-11-15 10:00:43,615 - onap_tests.scenario.onboard - INFO - Check and update service 2018-11-15 10:00:44,262 - onap_tests.scenario.onboard - INFO - service unique_id update 2018-11-15 10:00:44,450 - onap_tests.scenario.onboard - INFO - service Start Certification 2018-11-15 10:00:46,564 - onap_tests.scenario.onboard - INFO - Check and update service 2018-11-15 10:00:47,048 - onap_tests.scenario.onboard - INFO - service Distribution 2018-11-15 10:00:49,731 - onap_tests.scenario.onboard - INFO - Check and update service 2018-11-15 10:02:50,715 - onap_tests.scenario.onboard - INFO - Service distributed 2018-11-15 10:02:52,376 - onap_tests.scenario.onboard - INFO - declare_service_in_aai 2018-11-15 10:02:52,848 - onap_tests.scenario.onboard - INFO - resource_version = 1542276172598 2018-11-15 10:02:52,976 - onap_tests.scenario.onboard - INFO - declare_customer = generic 2018-11-15 10:02:53,379 - onap_tests.scenario.onboard - INFO - declare_cloud_region 2018-11-15 10:02:53,925 - onap_tests.scenario.onboard - INFO - add_tenant_to_cloud 2018-11-15 10:02:54,714 - onap_tests.scenario.e2e - INFO - Onboarding OK 2018-11-15 10:02:54,715 - onap_tests.scenario.e2e - INFO - Wait for distribution 2018-11-15 10:02:54,715 - onap_tests.scenario.e2e - INFO - Start instantiation 2018-11-15 10:02:54,715 - onap_tests.scenario.e2e - INFO - NBI option: False 2018-11-15 10:02:55,481 - onap_tests.scenario.instantiate - INFO - get VNF ubuntu16_VF 0 info 2018-11-15 10:02:57,415 - onap_tests.scenario.instantiate - INFO - Start the instantiation of the VNF 2018-11-15 10:02:57,416 - onap_tests.scenario.instantiate - INFO - 1) Create Service instance in SO 2018-11-15 10:03:19,713 - onap_tests.scenario.instantiate - INFO - 2) Create VNF instance ubuntu16_VF 0 in SO 2018-11-15 10:03:40,637 - onap_tests.scenario.instantiate - INFO - 3) Preload VNF ubuntu16_VF 0 in SDNC 2018-11-15 10:03:50,980 - onap_tests.scenario.instantiate - INFO - 4) Create MODULE ubuntu16_VF 0 instance in SO 2018-11-15 10:05:11,686 - onap_tests.scenario.instantiate - INFO - SO module vf(s) created Le jeudi 08 novembre 2018 à 09:08 +0100, RICHOMME Morgan TGI/OLN a écrit : Hi, as some of you already remarked, we are facing criticial issues on the openlab at the moment. the root cause seems to be the unsollicited power breakdown from last week, which triggered critical problems on our Ceph backend used by most of the DB (and there are a lots in OpenStack and ONAP). We were able to re-establish the system beginning of the week but new problems occurs as soon we fix some. We will reinstall the whole POD after a check of the physical disk. Sorry for the incovenience Morgan _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13761): https://lists.onap.org/g/onap-discuss/message/13761 Mute This Topic: https://lists.onap.org/mt/28145664/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
