Hi Michael
thanks for the feedback.
Our Casablanca chain is based on casablanca tag (not 3.0.1-ONAP)
would you recommend to move to 3.0.1-ONAP?
does the casablanca tag not correspond to the last stable version?
I am still not fully clear with the different versions.
Using this tag, on our daily casablanca chain, we do not have error on pomba
anymore (last night CI run).
onap-dmaap-dbc-pg-0 1/1
Running 0 10h
onap-dmaap-dbc-pg-1 0/1
Running 103 10h
onap-dmaap-dbc-pgpool-546dd7b4db-dbf9z 1/1
Running 0 10h
onap-dmaap-dbc-pgpool-546dd7b4db-vlflq 1/1
Running 0 10h
onap-dmaap-dmaap-bus-controller-789f469845-nf64v 1/1
Running 0 10h
onap-dmaap-dmaap-dr-db-676df89b46-q4hw8 1/1
Running 0 10h
onap-dmaap-dmaap-dr-node-74bbf5fdc9-gvh5f 0/1
Running 157 10h
onap-dmaap-dmaap-dr-prov-6978bb9b44-pjlpc 1/1
Running 0 10h
onap-dmaap-message-router-6bbd6b7544-sg45k 1/1
Running 0 10h
onap-dmaap-message-router-kafka-f4c9d4898-vqswv 1/1
Running 0 10h
onap-dmaap-message-router-zookeeper-76b96bdfc5-x5gj9 1/1
Running 0 10h
onap-sdnc-sdnc-dmaap-listener-6d566f5cb8-942zx 1/1
Running 0 10h
onap-pomba-pomba-aaictxbuilder-6d84cdff87-btc6z 2/2
Running 0 9h
onap-pomba-pomba-contextaggregator-95476848d-c9jh7 1/1
Running 0 9h
onap-pomba-pomba-data-router-6794b78d59-hjkl4 1/1
Running 0 9h
onap-pomba-pomba-elasticsearch-5f9f6cdbcf-5p89t 1/1
Running 0 9h
onap-pomba-pomba-kibana-c74b96f66-d9vqs 1/1
Running 0 9h
onap-pomba-pomba-networkdiscovery-5f89c498fb-xttln 2/2
Running 0 9h
onap-pomba-pomba-networkdiscoveryctxbuilder-655b56cdb8-sjjc5 2/2
Running 0 9h
onap-pomba-pomba-sdcctxbuilder-64bd6c5d45-j4568 1/1
Running 0 9h
onap-pomba-pomba-search-data-787fb7bf5-wlsl2 2/2
Running 0 9h
onap-pomba-pomba-servicedecomposition-7cf895c688-x5wtl 2/2
Running 0 9h
onap-pomba-pomba-validation-service-dd7d787ff-jfvtw 1/1
Running 0 9h
But we have on oof and ansible-server
kubectl get pods -n onap |grep -v Running
NAME READY STATUS
RESTARTS AGE
onap-oof-oof-has-api-7597966cb9-48wrl 0/1
Init:0/3 56 9h
onap-oof-oof-has-controller-755f689584-rlr8s 0/1
CrashLoopBackOff 112 9h
onap-oof-oof-has-data-c7788fc4d-jmrld 0/1
Init:2/4 55 9h
onap-oof-oof-has-healthcheck-tnssv 0/1
Init:0/1 56 9h
onap-oof-oof-has-reservation-6cb9fd8b48-gk9p5 0/1
Init:2/4 55 9h
onap-oof-oof-has-solver-54f46bbccc-dds5h 0/1
Init:2/4 55 9h
onap-sdnc-sdnc-ansible-server-5886ff7f69-6msxc 0/1
CrashLoopBackOff 111 10h
/Morgan
Le lundi 18 février 2019 à 01:32 +0000, Michael O'Brien a écrit :
Team,
Verified that POMBA is OK on the latest Casablanca – the key is adhering to
the orchestration sequence – DmaaP (specifically the mr pod) – needs to be up
before everything else
https://wiki.onap.org/display/DW/Log+Streaming+Compliance+and+API#LogStreamingComplianceandAPI-DeploymentDependencyTree-Containerlevel
All pods are up no problem – using the branch (essentially 3.0.1-ONAP) (Note:
however on a long running 3.0.0-ONAP system for 60 days I do see crashloops on
several pods across the ONAP deployment – but this is due to our s3p status in
general)
Every 2.0s: kubectl get pods --all-namespaces
Sun Feb 17 18:48:55 2019
NAMESPACE NAME
READY STATUS RESTARTS AGE
kube-system heapster-7b48b696fc-69hqm
1/1 Running 0 44m
kube-system kube-dns-6655f78c68-blgzk
3/3 Running 0 44m
kube-system kubernetes-dashboard-6f54f7c4b-69hd2
1/1 Running 0 44m
kube-system monitoring-grafana-7877679464-fmwws
1/1 Running 0 44m
kube-system monitoring-influxdb-64664c6cf5-kqswn
1/1 Running 0 44m
kube-system tiller-deploy-6f4745cbcf-p55ng
1/1 Running 0 44m
onap onap-dmaap-dbc-pg-0
1/1 Running 0 15m
onap onap-dmaap-dbc-pg-1
1/1 Running 0 15m
onap onap-dmaap-dbc-pgpool-c5f8498-8fbfk
1/1 Running 0 15m
onap onap-dmaap-dbc-pgpool-c5f8498-q4jcm
1/1 Running 0 15m
onap onap-dmaap-dmaap-bus-controller-557dc8c59c-ps99v
1/1 Running 0 15m
onap onap-dmaap-dmaap-dr-db-576f7968b8-dflf4
1/1 Running 0 15m
onap onap-dmaap-dmaap-dr-node-7647f9d6d8-rbwpd
0/1 Running 7 15m
onap onap-dmaap-dmaap-dr-prov-f4d84869f-87mj2
1/1 Running 0 15m
onap onap-dmaap-message-router-76f4799d-j6srw
1/1 Running 0 15m
onap onap-dmaap-message-router-kafka-5d96486cd6-77dkd
1/1 Running 0 15m
onap onap-dmaap-message-router-zookeeper-677d5894d5-g5g9m
1/1 Running 0 15m
onap onap-pomba-pomba-aaictxbuilder-8565f7d75c-dwvfr
2/2 Running 0 10m
onap onap-pomba-pomba-contextaggregator-d9f888c4-zpx2k
1/1 Running 0 10m
onap onap-pomba-pomba-data-router-66c5544446-d9crb
1/1 Running 0 10m
onap onap-pomba-pomba-elasticsearch-5c4f8c6b5b-85tvl
1/1 Running 0 10m
onap onap-pomba-pomba-kibana-849cb74588-hfqps
1/1 Running 0 10m
onap onap-pomba-pomba-networkdiscovery-556bb9d7b8-9ghm4
2/2 Running 0 10m
onap onap-pomba-pomba-networkdiscoveryctxbuilder-6ff7d49464-ngjql
2/2 Running 0 10m
onap onap-pomba-pomba-sdcctxbuilder-7799ffccc9-jd2jx
1/1 Running 0 10m
onap onap-pomba-pomba-search-data-7c649f8cbf-sfb5q
2/2 Running 0 10m
onap onap-pomba-pomba-servicedecomposition-5f8bb9d6f6-qdnqv
2/2 Running 0 10m
onap onap-pomba-pomba-validation-service-79b8d5cb9f-x99r4
1/1 Running 0 10m
/michael
From: [email protected] <[email protected]> On Behalf Of
Morgan Richomme
Sent: Friday, February 15, 2019 10:00 AM
To: [email protected]; ZZZ Onap OPENLAB <[email protected]>;
[email protected]; [email protected];
[email protected]; [email protected];
[email protected]; [email protected]; [email protected];
[email protected]; [email protected];
[email protected]; [email protected]; GARCIA CESPEDES, Francisco
javier OSP <[email protected]>; PAWLAK Pawel O-PL
<[email protected]>; MASTALERZ Wojciech O-PL
<[email protected]>; BRAUD Arnaud TGI/OLN
<[email protected]>; [email protected];
[email protected]; [email protected]; MINODIER David TGI/OLN
<[email protected]>; [email protected]; ROZIECKI Arnaud OLN/QOP
<[email protected]>; [email protected]; [email protected];
[email protected]; SKORUPA Marian O-PL <[email protected]>;
KOWALCZYK Emil O-PL <[email protected]>; GEEREBAERT Matthieu TGI/OLS
<[email protected]>; HASPEKIAN Goar TGI/OLN
<[email protected]>; [email protected]; DEBEAU Eric TGI/OLN
<[email protected]>; [email protected];
[email protected]; [email protected];
[email protected]; [email protected]; DESBUREAUX
Sylvain TGI/OLN <[email protected]>; [email protected];
ROBERT René TGI/OLN <[email protected]>; LY Amadou Houndaisse OLN/QOP
<[email protected]>; WECHSLER Philippe TGI/OLN
<[email protected]>; [email protected]; EDEL Nicolas TGI/OLN
<[email protected]>; SERKIES Bartek O-PL <[email protected]>;
[email protected]; [email protected]; [email protected]; LABIDURIE
Nathalie TGI/OLN <[email protected]>; BARS Remi TGI/OLN
<[email protected]>; HOCHET Remi TGI/OLN <[email protected]>;
[email protected]; ROBERT Ludovic TGI/OLS
<[email protected]>; [email protected]; CANEVALI Alexis OLN/QOP
<[email protected]>; CLAQUIN Marc TGI/OLN <[email protected]>;
[email protected]; Mielowski Rafał 2 - ORPL
<[email protected]>; [email protected]; LABIDURIE Nathalie
TGI/OLN <[email protected]>; GALIPOT Francis TGI/OLS
<[email protected]>; [email protected];
[email protected]; SANCHEZ VILCHEZ Jose Manuel TGI/OLN
<[email protected]>; HAMID Mahmoud Ext OBS/CSO
<[email protected]>; JANASZKA Tomasz O-PL
<[email protected]>; FRINDRICH Ondrej O-SK/ITN
<[email protected]>; SEVILLA Karine TGI/OLN
<[email protected]>; [email protected]; DESTRE Christian
TGI/OLN <[email protected]>; [email protected]; GOARZIN Lionel
TGI/OLN <[email protected]>; [email protected];
[email protected]; [email protected]; BERRICHE Wassim TGI/OLN
<[email protected]>; ROSSANT Olivier DTRS/DREAM
<[email protected]>; BURSZTYNOWSKI Dariusz O-PL
<[email protected]>; SEAUDI Abdelmuhaimen O-EG/HRCS
<[email protected]>; GAREL Romain TGI/OLN
<[email protected]>; CROSSON David TGI/OLS <[email protected]>;
PANEK Grzegorz O-PL <[email protected]>;
[email protected]; [email protected];
[email protected]; [email protected]; LAPLAUD Nicolas
TGI/OLS <[email protected]>; [email protected]; WALY Mohamed O-EG/HRCS
<[email protected]>; [email protected]; OSIŃSKI Tomasz O-PL
<[email protected]>; Pawłowski Michał 1 - Hurt
<[email protected]>; SIVAN Nicolas TGI/OLN
<[email protected]>; [email protected];
[email protected]; [email protected];
[email protected]; [email protected]; SLAVKOVSKÝ Adrián O-SK/ITN
<[email protected]>; [email protected]; RAJEWSKI Lukasz
O-PL <[email protected]>; PERZO Thibault TGI/OLN
<[email protected]>; [email protected]; HUET Vincent TGI/OLN
<[email protected]>; [email protected];
[email protected]; KERRIEN Isabelle TGI/OLN
<[email protected]>; [email protected];
[email protected]; OUZZIF Meryem TGI/OLN <[email protected]>;
[email protected]; HARDY Thierry TGI/OLN <[email protected]>;
DENISIEWICZ Andrzej O-PL <[email protected]>; [email protected];
BLAISONNEAU David TGI/OLN <[email protected]>; [email protected];
[email protected]; OCENAS Martin O-SK/ITN <[email protected]>;
[email protected]; GIMBERT Romain TGI/OLS
<[email protected]>; [email protected]; JEZEQUEL Mickael
TGI/OLN <[email protected]>; Kwiecień Krzysztof 1 - ORPL
<[email protected]>; MIELCZAREK Slawomir O-PL
<[email protected]>; [email protected]; [email protected]
Cc: [email protected]; [email protected];
[email protected]
Subject: Re: [onap-discuss] [ONAP] Orange openlab upgrade casablanca to
casablanca maintenance release
Hi
the Openlab has been reinstalled.
The keys have been pushed to the jumphost .
Basic tests have been performed .
It was possible to onboard and instantiate a basic VM successfuly.
However
please note that some pods are not in Running state (JIRAs are already open)
NAME READY STATUS
RESTARTS AGE
onap-aai-aai-sparky-be-7d778b7c6c-2glrc 1/2
CrashLoopBackOff 55 3h
onap-appc-appc-ansible-server-54f489647d-lrznf 0/1
CrashLoopBackOff 28 2h
onap-pomba-pomba-networkdiscoveryctxbuilder-6bf8dccf8f-96574 1/2
CrashLoopBackOff 47 2h
onap-pomba-pomba-sdcctxbuilder-77d67bcf7b-z525c 0/1
CrashLoopBackOff 47 2h
onap-sdnc-sdnc-ansible-server-7449c969bd-4blc8 0/1
CrashLoopBackOff 37 3h
onap-uui-uui-server-64bf85dc6d-5tl29 0/1
CrashLoopBackOff 33 2h
On the healthcheck are not all successfull
==============================================================================
Testsuites
==============================================================================
Testsuites.Health-Check :: Testing ecomp components are available via calls.
==============================================================================
Basic A&AI Health Check | PASS |
------------------------------------------------------------------------------
Basic AAF Health Check | PASS |
------------------------------------------------------------------------------
Basic AAF SMS Health Check | PASS |
------------------------------------------------------------------------------
Basic APPC Health Check | PASS |
------------------------------------------------------------------------------
Basic CLI Health Check | PASS |
------------------------------------------------------------------------------
Basic CLAMP Health Check | PASS |
------------------------------------------------------------------------------
Basic DCAE Health Check | FAIL |
500 != 200
------------------------------------------------------------------------------
Basic DMAAP Data Router Health Check | PASS |
------------------------------------------------------------------------------
Basic DMAAP Message Router Health Check | PASS |
------------------------------------------------------------------------------
Basic External API NBI Health Check | PASS |
------------------------------------------------------------------------------
Basic Log Elasticsearch Health Check | PASS |
------------------------------------------------------------------------------
Basic Log Kibana Health Check | PASS |
------------------------------------------------------------------------------
Basic Log Logstash Health Check | PASS |
------------------------------------------------------------------------------
Basic Microservice Bus Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-ocata API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-pike API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-titanium_cloud API Health Check | PASS |
------------------------------------------------------------------------------
Basic Multicloud-vio API Health Check | PASS |
------------------------------------------------------------------------------
Basic OOF-Homing Health Check | PASS |
------------------------------------------------------------------------------
Basic OOF-SNIRO Health Check | PASS |
------------------------------------------------------------------------------
Basic OOF-CMSO Health Check | PASS |
------------------------------------------------------------------------------
Basic Policy Health Check | PASS |
------------------------------------------------------------------------------
Basic Pomba AAI-context-builder Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic Pomba SDC-context-builder Health Check | PASS |
------------------------------------------------------------------------------
Basic Pomba Network-discovery-context-builder Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic Portal Health Check | PASS |
------------------------------------------------------------------------------
Basic SDC Health Check
(DMaaP:UP)| PASS |
------------------------------------------------------------------------------
Basic SDNC Health Check | PASS |
------------------------------------------------------------------------------
Basic SO Health Check | PASS |
------------------------------------------------------------------------------
Basic UseCaseUI API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC catalog API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC emsdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC gvnfmdriver API Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic VFC huaweivnfmdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC jujuvnfmdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC multivimproxy API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC nokiavnfmdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC nokiav2driver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC nslcm API Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic VFC resmgr API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC vnflcm API Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic VFC vnfmgr API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC vnfres API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC workflow API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC ztesdncdriver API Health Check | PASS |
------------------------------------------------------------------------------
Basic VFC ztevnfmdriver API Health Check | FAIL |
Test timeout 10 seconds exceeded.
------------------------------------------------------------------------------
Basic VID Health Check | PASS |
------------------------------------------------------------------------------
Basic VNFSDK Health Check | PASS |
------------------------------------------------------------------------------
Basic Holmes Rule Management API Health Check | FAIL |
502 != 200
------------------------------------------------------------------------------
Basic Holmes Engine Management API Health Check | FAIL |
502 != 200
------------------------------------------------------------------------------
Testsuites.Health-Check :: Testing ecomp components are available ... | FAIL |
51 critical tests, 42 passed, 9 failed
51 tests total, 42 passed, 9 failed
==============================================================================
Testsuites | FAIL |
51 critical tests, 42 passed, 9 failed
51 tests total, 42 passed, 9 failed
==============================================================================
/Morgan
Le mardi 12 février 2019 à 15:57 +0100, Morgan Richomme a écrit :
Hi,
after 52 days of very relative stability of the Casablanca release, we finally
lost our OpenLab this afternoon.
There are discrepencies in Kubernetes between the SO pods, replicaset and
deployment.
The Deployment see everything all right, the pod does not exist anymore and the
replicaset cannot be deleted or modified in order to force the respawn of the
so-so pod.
Several pods were in a bad shape anyway
Rancher indicates unhealthy state of addon-starter and
rancher-ingress-controller/
We started troubleshooting but as we planned to re-install the Casablanca
maintenance release soon, we decided to anticipate this reinstallation to spare
some time.
If you want to save anything on your jumphost, please proceed. Save you rc file
(we do not reinstall the OpenStack part)
Everything will be cleaned. and your creds/keys repushed.
Sorry for the incovenience.
Morgan
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
This email and the information contained herein is proprietary and confidential
and subject to the Amdocs Email Terms of Service, which you may review at
https://www.amdocs.com/about/email-terms-of-service
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15624): https://lists.onap.org/g/onap-discuss/message/15624
Mute This Topic: https://lists.onap.org/mt/29754520/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
