Re: [opnfv-tech-discuss] [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

[morgan.richomme]

Hi,

I will modify our CI chains in order to alternate check with fraser / master 
and keep on providing feedback.
We may change the config to make the dashboard publically available
After one week of perfect runs (last week), we started getting some errors.. 
api_check / refstack / tempest / rally / patrole (not always the same tests)
but all the errors were linked to volume management
Lots of old test volumes got stuck and it was not possible to delete them.
All the problems were due to cinder operations launched on 1 of the 3 
controllers (HA mode), it explains the random effect (sometimes patrole was OK 
when cinder operations were done on controler 1 or 2 and fails when done on 
controler 3)
It was impossible to delete them properly. There was probably a loss of 
synchronization on this controler.
The decision was taken to reboot this controller

After the reboot, fraser tests are back to nominal this morning.

++--+-+--++
| TEST CASE  | PROJECT  | TIER| 
DURATION | RESULT |
++--+-+--++
|  connection_check  | functest | healthcheck |  
00:07   |  PASS  |
| api_check  | functest | healthcheck |  
09:53   |  PASS  |
| snaps_health_check | functest | healthcheck |  
00:53   |  PASS  |


+--+--+---+--++
|  TEST CASE   | PROJECT  |  TIER | 
DURATION | RESULT |
+--+--+---+--++
|  vping_ssh   | functest | smoke |  00:56  
 |  PASS  |
|vping_userdata| functest | smoke |  00:41  
 |  PASS  |
| tempest_smoke_serial | functest | smoke |  18:43  
 |  PASS  |
| rally_sanity | functest | smoke |  27:11  
 |  PASS  |
|   refstack_defcore   | functest | smoke |  08:11  
 |  PASS  |
|   patrole| functest | smoke |  04:57  
 |  PASS  |
| snaps_smoke  | functest | smoke |  58:07  
 |  PASS  |
| odl  | functest | smoke |  00:00  
 |  SKIP  |
| odl_netvirt  | functest | smoke |  00:00  
 |  SKIP  |
|neutron_trunk | functest | smoke |  00:00  
 |  SKIP  |
+--+--+---+--++
vnf tests in progress

it is interesting and shows clearly the need for long duration tests :)

OK for xtesting update.
I am currently trying to finalize some VNF testing that I planned to integrated 
in an xtesting/onap-vnf, I will share with the integration team when it will be 
more mature
Meanwhile our xtesting/onap-robot is run daily on our Open Lab

/Morgan

Le jeudi 12 avril 2018 à 14:16 +0200, OLLIVIER Cédric IMT/OLN a écrit :
Hello Morgan,

Thank you very much for your help.
All tests done via ONAP OpenLab has been very helpful throughout the F-release 
by providing the quickest feedbacks.
We are checking if Functest testcases have to complete Snaps’ garbage 
collectors for floating ips.
That would explain this remaining resources.

Yes,  we will target live testing for the next release.
http://testresults.opnfv.org/functest/gambiachallenges/

About the reuse of OPNFV in ONAP, Xtesting has been synchronized to Queens this 
morning (opnfv/xtesting:latest).
Onap Healthcheck container could inherit from opnfv/xtesting:fraser if it needs 
requirements synchronized with OpenStack Pike.

Functest (master) could switch to Queens as well in order to get first 
feedbacks before next MS0.
https://gerrit.opnfv.org/gerrit/#/c/3/

Cédric

De : RICHOMME Morgan IMT/OLN
Envoyé : mardi 10 avril 2018 09:52
À : dmcbr...@linuxfoundation.org; opnfv-tech-discuss@lists.opnfv.org; OLLIVIER 
Cédric IMT/OLN
Cc : helen.c...@huawei.com; DEBEAU Eric IMT/OLN; gabriel.yuy...@huawei.com
Objet : [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

Hi

for information, we switched to the fraser version for our daily test on our 
ONAP OpenLab, a communit lab for ONAP users.
Even if the lab is used by ONAP contributors for ONAP testing, we are reusing 
Functest test suites to be sure that the infrastructure supporting ONAP is 
still properly working.

We were using latest tag so far and decided to go to fraser to provide some 

Re: [opnfv-tech-discuss] [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

[cedric.ollivier]

Hello Morgan,

Would it be possible to switch to OPNFV Functest latest containers (several 
runs) ?
We have updated all OPNFV Functest test cases for Gambia Release (N+1) and we 
need a reference (and stable) platform to get a real overview.

From the time being, only Compass is currently providing interesting results 
(most installers are only running OPNFV Functest Fraser).
It seems that we are getting the same results between OPNFV Functest Fraser and 
Gambia (e.g. highlighting same bugs about Compass).
https://build.opnfv.org/ci/job/functest-compass-baremetal-daily-master/92/console
 (os-nosdn-nofeature-ha)
https://build.opnfv.org/ci/view/functest/job/functest-compass-baremetal-daily-master/88/console
 (os-odl_l3-nofeature-ha)

It should be noted that refstack_defcore is temporarily disabled.
It simply doesn’t meet OpenStack Queens requirements (refstack-client is 
hardcoded to testr).

We could expect a full success vs Orange ONAP Openlab which is already fully 
verified by OPNFV Functest Fraser.
https://wiki.opnfv.org/pages/viewpage.action?pageId=13211751

Even if OpenStack Pike has been deployed, it should be fine thanks to backward 
compatibility enforced by OpenStack.

Thank you in advance for your help,
Cédric

De : OLLIVIER Cédric IMT/OLN
Envoyé : jeudi 12 avril 2018 14:16
À : RICHOMME Morgan IMT/OLN; dmcbr...@linuxfoundation.org; 
opnfv-tech-discuss@lists.opnfv.org
Cc : helen.c...@huawei.com; DEBEAU Eric IMT/OLN; gabriel.yuy...@huawei.com
Objet : RE: [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

Hello Morgan,

Thank you very much for your help.
All tests done via ONAP OpenLab has been very helpful throughout the F-release 
by providing the quickest feedbacks.

We are checking if Functest testcases have to complete Snaps’ garbage 
collectors for floating ips.
That would explain this remaining resources.

Yes,  we will target live testing for the next release.
http://testresults.opnfv.org/functest/gambiachallenges/

About the reuse of OPNFV in ONAP, Xtesting has been synchronized to Queens this 
morning (opnfv/xtesting:latest).
Onap Healthcheck container could inherit from opnfv/xtesting:fraser if it needs 
requirements synchronized with OpenStack Pike.

Functest (master) could switch to Queens as well in order to get first 
feedbacks before next MS0.
https://gerrit.opnfv.org/gerrit/#/c/3/

Cédric

De : RICHOMME Morgan IMT/OLN
Envoyé : mardi 10 avril 2018 09:52
À : dmcbr...@linuxfoundation.org; 
opnfv-tech-discuss@lists.opnfv.org; 
OLLIVIER Cédric IMT/OLN
Cc : helen.c...@huawei.com; DEBEAU Eric IMT/OLN; 
gabriel.yuy...@huawei.com
Objet : [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

Hi

for information, we switched to the fraser version for our daily test on our 
ONAP OpenLab, a communit lab for ONAP users.
Even if the lab is used by ONAP contributors for ONAP testing, we are reusing 
Functest test suites to be sure that the infrastructure supporting ONAP is 
still properly working.

We were using latest tag so far and decided to go to fraser to provide some 
feedback for the OPNFV forthcoming release.
The infrastructure has been deployed using XCI/bifrost scripts + Kolla 
installer (currently under integration in XCI) on bare metal.
The scenario corresponds to a basic os-nosdn-nofeature-ha scenario.

the results are perfect :)

++--+-+--++
| TEST CASE  | PROJECT  | TIER| 
DURATION | RESULT |
++--+-+--++
|  connection_check  | functest | healthcheck |  
00:07   |  PASS  |
| api_check  | functest | healthcheck |  
08:26   |  PASS  |
| snaps_health_check | functest | healthcheck |  
00:49   |  PASS  |
++--+-+--++

+--+--+---+--++
|  TEST CASE   | PROJECT  |  TIER | 
DURATION | RESULT |
+--+--+---+--++
|  vping_ssh   | functest | smoke |  00:42  
 |  PASS  |
|vping_userdata| functest | smoke |  00:35  
 |  PASS  |
| tempest_smoke_serial | functest | smoke |  12:23  
 |  PASS  |
| rally_sanity | functest | smoke |  20:46  
 |  PASS  |
|   refstack_defcore  

Re: [opnfv-tech-discuss] [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

[cedric.ollivier]

Hello Morgan,

Thank you very much for your help.
All tests done via ONAP OpenLab has been very helpful throughout the F-release 
by providing the quickest feedbacks.
We are checking if Functest testcases have to complete Snaps’ garbage 
collectors for floating ips.
That would explain this remaining resources.

Yes,  we will target live testing for the next release.
http://testresults.opnfv.org/functest/gambiachallenges/

About the reuse of OPNFV in ONAP, Xtesting has been synchronized to Queens this 
morning (opnfv/xtesting:latest).
Onap Healthcheck container could inherit from opnfv/xtesting:fraser if it needs 
requirements synchronized with OpenStack Pike.

Functest (master) could switch to Queens as well in order to get first 
feedbacks before next MS0.
https://gerrit.opnfv.org/gerrit/#/c/3/

Cédric

De : RICHOMME Morgan IMT/OLN
Envoyé : mardi 10 avril 2018 09:52
À : dmcbr...@linuxfoundation.org; opnfv-tech-discuss@lists.opnfv.org; OLLIVIER 
Cédric IMT/OLN
Cc : helen.c...@huawei.com; DEBEAU Eric IMT/OLN; gabriel.yuy...@huawei.com
Objet : [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

Hi

for information, we switched to the fraser version for our daily test on our 
ONAP OpenLab, a communit lab for ONAP users.
Even if the lab is used by ONAP contributors for ONAP testing, we are reusing 
Functest test suites to be sure that the infrastructure supporting ONAP is 
still properly working.

We were using latest tag so far and decided to go to fraser to provide some 
feedback for the OPNFV forthcoming release.
The infrastructure has been deployed using XCI/bifrost scripts + Kolla 
installer (currently under integration in XCI) on bare metal.
The scenario corresponds to a basic os-nosdn-nofeature-ha scenario.

the results are perfect :)

++--+-+--++
| TEST CASE  | PROJECT  | TIER| 
DURATION | RESULT |
++--+-+--++
|  connection_check  | functest | healthcheck |  
00:07   |  PASS  |
| api_check  | functest | healthcheck |  
08:26   |  PASS  |
| snaps_health_check | functest | healthcheck |  
00:49   |  PASS  |
++--+-+--++

+--+--+---+--++
|  TEST CASE   | PROJECT  |  TIER | 
DURATION | RESULT |
+--+--+---+--++
|  vping_ssh   | functest | smoke |  00:42  
 |  PASS  |
|vping_userdata| functest | smoke |  00:35  
 |  PASS  |
| tempest_smoke_serial | functest | smoke |  12:23  
 |  PASS  |
| rally_sanity | functest | smoke |  20:46  
 |  PASS  |
|   refstack_defcore   | functest | smoke |  06:53  
 |  PASS  |
|   patrole| functest | smoke |  04:29  
 |  PASS  |
| snaps_smoke  | functest | smoke |  50:18  
 |  PASS  |
| odl  | functest | smoke |  00:00  
 |  SKIP  |
| odl_netvirt  | functest | smoke |  00:00  
 |  SKIP  |
|neutron_trunk | functest | smoke |  00:00  
 |  SKIP  |
+--+--+---+--++

+--+--+--+--++
|  TEST CASE   | PROJECT  | TIER | DURATION |   
  RESULT |
+--+--+--+--++
| cloudify_ims | functest | vnf  |  27:40   |   
   PASS  |
| vyos_vrouter | functest | vnf  |  18:23   |   
   PASS  |
|   juju_epc   | functest | vnf  |  51:17   |   
   PASS  |
+--+--+--+--++

Thanks to the functest and XCI team!
Tests will be run daily so we would report any issue if we are facing some but 
it looks pretty stable.
Unlike the traditionnal approach, we are running daily tests on a up 
infrastructure (we do not reinstall it systematically before launching the 
tests)
so it is connected 

Re: [opnfv-tech-discuss] [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

[Yuyang (Gabriel)]

Looking pretty well!
Thanks Functest and XCI team!

Look forward to seeing the migration to long duration test!

Thanks,
Gabriel

From: morgan.richo...@orange.com [mailto:morgan.richo...@orange.com]
Sent: Tuesday, April 10, 2018 3:52 PM
To: dmcbr...@linuxfoundation.org; opnfv-tech-discuss@lists.opnfv.org; OLLIVIER 
Cédric IMT/OLN 
Cc: Yunxia Chen ; DEBEAU Eric IMT/OLN 
; Yuyang (Gabriel) 
Subject: [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

Hi

for information, we switched to the fraser version for our daily test on our 
ONAP OpenLab, a communit lab for ONAP users.
Even if the lab is used by ONAP contributors for ONAP testing, we are reusing 
Functest test suites to be sure that the infrastructure supporting ONAP is 
still properly working.

We were using latest tag so far and decided to go to fraser to provide some 
feedback for the OPNFV forthcoming release.
The infrastructure has been deployed using XCI/bifrost scripts + Kolla 
installer (currently under integration in XCI) on bare metal.
The scenario corresponds to a basic os-nosdn-nofeature-ha scenario.

the results are perfect :)

++--+-+--++
| TEST CASE  | PROJECT  | TIER| 
DURATION | RESULT |
++--+-+--++
|  connection_check  | functest | healthcheck |  
00:07   |  PASS  |
| api_check  | functest | healthcheck |  
08:26   |  PASS  |
| snaps_health_check | functest | healthcheck |  
00:49   |  PASS  |
++--+-+--++

+--+--+---+--++
|  TEST CASE   | PROJECT  |  TIER | 
DURATION | RESULT |
+--+--+---+--++
|  vping_ssh   | functest | smoke |  00:42  
 |  PASS  |
|vping_userdata| functest | smoke |  00:35  
 |  PASS  |
| tempest_smoke_serial | functest | smoke |  12:23  
 |  PASS  |
| rally_sanity | functest | smoke |  20:46  
 |  PASS  |
|   refstack_defcore   | functest | smoke |  06:53  
 |  PASS  |
|   patrole| functest | smoke |  04:29  
 |  PASS  |
| snaps_smoke  | functest | smoke |  50:18  
 |  PASS  |
| odl  | functest | smoke |  00:00  
 |  SKIP  |
| odl_netvirt  | functest | smoke |  00:00  
 |  SKIP  |
|neutron_trunk | functest | smoke |  00:00  
 |  SKIP  |
+--+--+---+--++

+--+--+--+--++
|  TEST CASE   | PROJECT  | TIER | DURATION |   
  RESULT |
+--+--+--+--++
| cloudify_ims | functest | vnf  |  27:40   |   
   PASS  |
| vyos_vrouter | functest | vnf  |  18:23   |   
   PASS  |
|   juju_epc   | functest | vnf  |  51:17   |   
   PASS  |
+--+--+--+--++

Thanks to the functest and XCI team!
Tests will be run daily so we would report any issue if we are facing some but 
it looks pretty stable.
Unlike the traditionnal approach, we are running daily tests on a up 
infrastructure (we do not reinstall it systematically before launching the 
tests)
so it is connected somehow to the long duration test acctivities coordinated by 
Gabriel

The only issues we faced so far with latest version were due to the cleaning of 
the resources (shortages of floatting IPs)
This issue has been identified and will be addressed as far as I understood in 
Gambia release.
we run now a cleaning script after the tests to free some resources

As a conclusion, the Orange Onap OpenLab validates the Fraser scenario 
os-nosdn-nofeature-ha for XCI/Kolla with functest healthcheck/smoke/vnf 
(duration 3h and 20 minutes)
It could be interesting for 

[opnfv-tech-discuss] [OPNFV] [Fraser] Fraser Functest run on Orange ONAP Open lab

[morgan.richomme]

Hi

for information, we switched to the fraser version for our daily test on our 
ONAP OpenLab, a communit lab for ONAP users.
Even if the lab is used by ONAP contributors for ONAP testing, we are reusing 
Functest test suites to be sure that the infrastructure supporting ONAP is 
still properly working.

We were using latest tag so far and decided to go to fraser to provide some 
feedback for the OPNFV forthcoming release.
The infrastructure has been deployed using XCI/bifrost scripts + Kolla 
installer (currently under integration in XCI) on bare metal.
The scenario corresponds to a basic os-nosdn-nofeature-ha scenario.

the results are perfect :)

++--+-+--++
| TEST CASE  | PROJECT  | TIER| 
DURATION | RESULT |
++--+-+--++
|  connection_check  | functest | healthcheck |  
00:07   |  PASS  |
| api_check  | functest | healthcheck |  
08:26   |  PASS  |
| snaps_health_check | functest | healthcheck |  
00:49   |  PASS  |
++--+-+--++

+--+--+---+--++
|  TEST CASE   | PROJECT  |  TIER | 
DURATION | RESULT |
+--+--+---+--++
|  vping_ssh   | functest | smoke |  00:42  
 |  PASS  |
|vping_userdata| functest | smoke |  00:35  
 |  PASS  |
| tempest_smoke_serial | functest | smoke |  12:23  
 |  PASS  |
| rally_sanity | functest | smoke |  20:46  
 |  PASS  |
|   refstack_defcore   | functest | smoke |  06:53  
 |  PASS  |
|   patrole| functest | smoke |  04:29  
 |  PASS  |
| snaps_smoke  | functest | smoke |  50:18  
 |  PASS  |
| odl  | functest | smoke |  00:00  
 |  SKIP  |
| odl_netvirt  | functest | smoke |  00:00  
 |  SKIP  |
|neutron_trunk | functest | smoke |  00:00  
 |  SKIP  |
+--+--+---+--++

+--+--+--+--++
|  TEST CASE   | PROJECT  | TIER | DURATION |   
  RESULT |
+--+--+--+--++
| cloudify_ims | functest | vnf  |  27:40   |   
   PASS  |
| vyos_vrouter | functest | vnf  |  18:23   |   
   PASS  |
|   juju_epc   | functest | vnf  |  51:17   |   
   PASS  |
+--+--+--+--++

Thanks to the functest and XCI team!
Tests will be run daily so we would report any issue if we are facing some but 
it looks pretty stable.
Unlike the traditionnal approach, we are running daily tests on a up 
infrastructure (we do not reinstall it systematically before launching the 
tests)
so it is connected somehow to the long duration test acctivities coordinated by 
Gabriel

The only issues we faced so far with latest version were due to the cleaning of 
the resources (shortages of floatting IPs)
This issue has been identified and will be addressed as far as I understood in 
Gambia release.
we run now a cleaning script after the tests to free some resources

As a conclusion, the Orange Onap OpenLab validates the Fraser scenario 
os-nosdn-nofeature-ha for XCI/Kolla with functest healthcheck/smoke/vnf 
(duration 3h and 20 minutes)
It could be interesting for us to add Yardstick daily tests...it is in the todo 
list...

Morgan


_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange