I saw that in the console log of the VM
[X]
VM is back
[X]
This VM is a standard node (16Go Ram / 8 vCPU / 160 Go Disk)
If we reached an out of memory, it is a bit worrying...
Moreover I do not have lots of good experience of k8s VM node reboot..
the pods on this node are at the moment
root@sb00-nfs:~# kubectl get pods -n onap -o wide --field-selector
spec.nodeName=sb00-k8s-08
NAME READY STATUS
RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dev-appc-ansible-server-0 1/1 Running 0
14d 10.42.5.6 sb00-k8s-08 <none> <none>
dev-appc-db-0 1/1 Running 1
14d 10.42.5.7 sb00-k8s-08 <none> <none>
dev-awx-postgres-79bc4cc7fd-dfd8d 1/1 Running 0
3h38m 10.42.5.80 sb00-k8s-08 <none> <none>
dev-clamp-5d65d479c7-kz787 1/1 Running 0
4h9m 10.42.5.70 sb00-k8s-08 <none> <none>
dev-clamp-dash-logstash-5d446f74c-nsplb 1/1 Running 0
4h9m 10.42.5.69 sb00-k8s-08 <none> <none>
dev-cli-6ff57dbbc7-qz4tm 1/1 Running 0
10d 10.42.5.40 sb00-k8s-08 <none> <none>
dev-consul-server-1 1/1 Running 0
14d 10.42.5.17 sb00-k8s-08 <none> <none>
dev-dbc-pg-primary-65c8848849-fzh62 1/1 Running 0
4h2m 10.42.5.75 sb00-k8s-08 <none> <none>
dev-dbc-pg-replica-bccb69c97-ccx44 1/1 Running 0
3h38m 10.42.5.81 sb00-k8s-08 <none> <none>
dev-dcae-dashboard-c6c5f9557-7xcbd 2/2 Running 0
10d 10.42.5.43 sb00-k8s-08 <none> <none>
dev-dcae-dashboard-pg-primary-7669755d5b-s7lrn 1/1 Running 0
3h38m 10.42.5.82 sb00-k8s-08 <none> <none>
dev-dcae-db-primary-6b97695fb7-k9f9g 1/1 Running 0
3h38m 10.42.5.83 sb00-k8s-08 <none> <none>
dev-dcae-db-replica-56566d86c5-2r8fb 1/1 Running 0
14d 10.42.5.11 sb00-k8s-08 <none> <none>
dev-dcae-inv-pg-replica-6b547c47f9-md2kh 1/1 Running 0
14d 10.42.5.12 sb00-k8s-08 <none> <none>
dev-dcae-inventory-api-5d46bcb8b4-4lgfj 2/2 Running 0
3h38m 10.42.5.84 sb00-k8s-08 <none> <none>
dev-dcae-policy-handler-6db65dd449-ktfwh 2/2 Running 0
14d 10.42.5.13 sb00-k8s-08 <none> <none>
dev-dmaap-bc-5d5497f8b9-75smb 1/1 Running 6
3d1h 10.42.5.52 sb00-k8s-08 <none> <none>
dev-dmaap-dr-db-0 0/1 CrashLoopBackOff 857
3d1h 10.42.5.55 sb00-k8s-08 <none> <none>
dev-dmaap-dr-node-0 0/2 Init:0/4 436
3d1h 10.42.5.54 sb00-k8s-08 <none> <none>
dev-mariadb-79cfb8b664-t9vkr 1/1 Running 26
4h9m 10.42.5.71 sb00-k8s-08 <none> <none>
dev-mariadb-galera-2 1/1 Running 0
14d 10.42.5.29 sb00-k8s-08 <none> <none>
dev-modeling-mariadb-2 1/1 Running 0
14d 10.42.5.28 sb00-k8s-08 <none> <none>
dev-multicloud-fcaps-6d4487d5c4-lmz84 3/3 Running 0
4h2m 10.42.5.72 sb00-k8s-08 <none> <none>
dev-nengdb-init-config-job-mng4k 0/1 Completed 0
3d 10.42.5.64 sb00-k8s-08 <none> <none>
dev-netbox-nginx-57b9c4cb4c-q5xjs 1/1 Running 0
14d 10.42.5.10 sb00-k8s-08 <none> <none>
dev-oof-cmso-topology-7b686f4bd5-xhjpx 1/1 Running 0
14d 10.42.5.20 sb00-k8s-08 <none> <none>
dev-oof-has-controller-7684db4dd7-5dck8 0/1 Init:0/3 0
4h2m 10.42.5.77 sb00-k8s-08 <none> <none>
dev-oof-has-healthcheck-lbnd6 0/1 Completed 0
14d 10.42.5.19 sb00-k8s-08 <none> <none>
dev-oof-has-solver-b97cc95cb-q7vn9 0/1 Init:3/4 0
4h2m 10.42.5.73 sb00-k8s-08 <none> <none>
dev-pap-8bc999f87-pnvlb 1/1 Running 0
4h2m 10.42.5.74 sb00-k8s-08 <none> <none>
dev-policy-distribution-6b4548b755-sl54l 1/1 Running 0
2d23h 10.42.5.66 sb00-k8s-08 <none> <none>
dev-policy-mariadb-0 1/1 Running 0
2d22h 10.42.5.68 sb00-k8s-08 <none> <none>
dev-portal-cassandra-59f5cb4cf5-pppgf 1/1 Running 0
14d 10.42.5.22 sb00-k8s-08 <none> <none>
dev-portal-zookeeper-58689b85b6-44rph 1/1 Running 0
3h38m 10.42.5.79 sb00-k8s-08 <none> <none>
dev-sdc-cs-config-cassandra-s5q2v 0/1 Completed 0
3d1h 10.42.5.49 sb00-k8s-08 <none> <none>
dev-sdc-dcae-be-tools-2bctm 0/1 Completed 0
3d1h 10.42.5.51 sb00-k8s-08 <none> <none>
dev-sdc-wfd-be-7c69b95669-w55m5 1/1 Running 0
4h2m 10.42.5.78 sb00-k8s-08 <none> <none>
dev-sdnc-ansible-server-b6b8d4ff6-dvm5k 1/1 Running 0
3d 10.42.5.63 sb00-k8s-08 <none> <none>
dev-sdnrdb-coordinating-only-787d785454-hd9vv 2/2 Running 0
3d 10.42.5.62 sb00-k8s-08 <none> <none>
dev-sdnrdb-master-2 1/1 Running 0
3d 10.42.5.65 sb00-k8s-08 <none> <none>
dev-so-mariadb-config-job-4vrfd 0/1 Completed 0
3d1h 10.42.5.50 sb00-k8s-08 <none> <none>
dev-uui-server-764fd549cc-nt5bg 1/1 Running 0
10d 10.42.5.42 sb00-k8s-08 <none> <none>
dev-vfc-mariadb-2 1/1 Running 0
14d 10.42.5.37 sb00-k8s-08 <none> <none>
dev-vfc-multivim-proxy-7555c45bcc-hc4rg 1/1 Running 0
14d 10.42.5.32 sb00-k8s-08 <none> <none>
dev-vfc-redis-7456bf9fd5-btv9x 1/1 Running 0
14d 10.42.5.33 sb00-k8s-08 <none> <none>
dev-vfc-workflow-8769976-wr2rz 1/1 Running 0
14d 10.42.5.34 sb00-k8s-08 <none> <none>
dev-vfc-workflow-engine-76d47fcb8f-km6hn 1/1 Running 0
2d23h 10.42.5.67 sb00-k8s-08 <none> <none>
dev-vfc-zte-sdnc-driver-74d4db6698-77qkw 1/1 Running 0
4h2m 10.42.5.76 sb00-k8s-08 <none> <none>
dev-vid-galera-0 1/1 Running 0
14d 10.42.5.35 sb00-k8s-08 <none> <none>
dev-vnfsdk-postgres-primary-66dcd965bc-zj87h 1/1 Running 0
14d 10.42.5.36 sb00-k8s-08 <none> <none>
I made a top on all of these pods
dev-appc-ansible-server-0 8m 61Mi
dev-appc-db-0 4m 180Mi
dev-awx-postgres-79bc4cc7fd-dfd8d 4m 37Mi
dev-clamp-5d65d479c7-kz787 1m 3Mi
dev-clamp-dash-logstash-5d446f74c-nsplb 6m 655Mi
dev-cli-6ff57dbbc7-qz4tm 1m 32Mi
dev-consul-server-1 19m 35Mi
dev-dbc-pg-primary-65c8848849-fzh62 1m 55Mi
dev-dbc-pg-replica-bccb69c97-ccx44 2m 28Mi
dev-dcae-dashboard-c6c5f9557-7xcbd 2m 484Mi
dev-dcae-dashboard-pg-primary-7669755d5b-s7lrn 2m 57Mi
dev-dcae-db-primary-6b97695fb7-k9f9g 2m 38Mi
dev-dcae-db-replica-56566d86c5-2r8fb 2m 26Mi
dev-dcae-inv-pg-replica-6b547c47f9-md2kh 1m 27Mi
dev-dcae-inventory-api-5d46bcb8b4-4lgfj 2m 219Mi
dev-dcae-policy-handler-6db65dd449-ktfwh 5m 56Mi
dev-dmaap-bc-5d5497f8b9-75smb 3m 248Mi
dev-dmaap-dr-db-0 0m 0Mi
Error from server (NotFound): podmetrics.metrics.k8s.io
"onap/dev-dmaap-dr-node-0" not found
dev-mariadb-79cfb8b664-t9vkr 1m 96Mi
dev-mariadb-galera-2 5m 599Mi
dev-modeling-mariadb-2 4m 140Mi
dev-multicloud-fcaps-6d4487d5c4-lmz84 5m 288Mi
Error from server (NotFound): podmetrics.metrics.k8s.io
"onap/dev-nengdb-init-config-job-mng4k" not found
dev-netbox-nginx-57b9c4cb4c-q5xjs 0m 2Mi
dev-oof-cmso-topology-7b686f4bd5-xhjpx 1m 247Mi
Error from server (NotFound): podmetrics.metrics.k8s.io
"onap/dev-oof-has-controller-7684db4dd7-5dck8" not found
Error from server (NotFound): podmetrics.metrics.k8s.io
"onap/dev-oof-has-healthcheck-lbnd6" not found
Error from server (NotFound): podmetrics.metrics.k8s.io
"onap/dev-oof-has-solver-b97cc95cb-q7vn9" not found
dev-pap-8bc999f87-pnvlb 4m 297Mi
dev-policy-distribution-6b4548b755-sl54l 2m 383Mi
dev-policy-mariadb-0 6m 275Mi
dev-portal-cassandra-59f5cb4cf5-pppgf 9m 2787Mi
dev-portal-zookeeper-58689b85b6-44rph 1m 115Mi
Error from server (NotFound): podmetrics.metrics.k8s.io
"onap/dev-sdc-cs-config-cassandra-s5q2v" not found
Error from server (NotFound): podmetrics.metrics.k8s.io
"onap/dev-sdc-dcae-be-tools-2bctm" not found
dev-sdc-wfd-be-7c69b95669-w55m5 2m 614Mi
dev-sdnc-ansible-server-b6b8d4ff6-dvm5k 7m 43Mi
dev-sdnrdb-coordinating-only-787d785454-hd9vv 2m 328Mi
dev-sdnrdb-master-2 4m 334Mi
Error from server (NotFound): podmetrics.metrics.k8s.io
"onap/dev-so-mariadb-config-job-4vrfd" not found
dev-uui-server-764fd549cc-nt5bg 2m 325Mi
dev-vfc-mariadb-2 5m 147Mi
dev-vfc-multivim-proxy-7555c45bcc-hc4rg 38m 164Mi
dev-vfc-redis-7456bf9fd5-btv9x 3m 462Mi
dev-vfc-workflow-8769976-wr2rz 1m 117Mi
dev-vfc-workflow-engine-76d47fcb8f-km6hn 2m 417Mi
dev-vfc-zte-sdnc-driver-74d4db6698-77qkw 2m 124Mi
dev-vid-galera-0 9m 181Mi
dev-vnfsdk-postgres-primary-66dcd965bc-zj87h 1m 42Mi
If we add the RAM consumption of all the pods on this node => the pods are at
the moment consuming at least 11 Go (as some metrics were not retrieved) on the
16 Go allocated.
I guess the issue could be on applicative side...
We got a security test to test the limits, it was descoped for Frankfurt but
will be set for Guilin.
Morgan
________________________________
De : PLATANIA, MARCO (MARCO) [[email protected]]
Envoyé : mardi 5 mai 2020 16:59
À : RICHOMME Morgan TGI/OLN; [email protected]
Cc : [email protected]
Objet : Re: Issue with sb00-k8s-01 at SB00 ?
I believe it recovered, even if some components aren’t passing health check. I
believe they were having issues before the VM1 outage though.
Marco
From: "[email protected]" <[email protected]>
Date: Tuesday, May 5, 2020 at 10:49 AM
To: "[email protected]" <[email protected]>, "PLATANIA, MARCO (MARCO)"
<[email protected]>
Cc: "[email protected]" <[email protected]>
Subject: RE:Issue with sb00-k8s-01 at SB00 ?
argh..
I will have a look at the VM and we will add this topic for the weekly tomorrow
not good when the full VM is down..
/Morgan
________________________________
De : [email protected] [[email protected]]
Envoyé : mardi 5 mai 2020 15:46
À : RICHOMME Morgan TGI/OLN; PLATANIA, MARCO (MARCO)
Cc : [email protected]
Objet : Issue with sb00-k8s-01 at SB00 ?
Hi, Morgan/Marco,
It looks like vm sb00-k8s-01 is not stable at SB00.
[cid:[email protected]]
Thanks,
Xin Miao
Solution Engineering
Fujitsu Network Communication
(W)972-479-2263 (M)469-268-5226
2811 Telecom Drive
Richardson, TX 75081, USA
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.
This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20960): https://lists.onap.org/g/onap-discuss/message/20960
Mute This Topic: https://lists.onap.org/mt/73997867/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
