I installed a set of ONAP components using OOM rancher/kubernetes, and some
AAI containers have been in CrashLoopBackOff (please see the information
below). I'd appreciate any suggestions on how to get them running.
Thanks much,
George
_____
%: kubectl get pods -o wide -n onap | grep -i aai
dev-aai-5445ffdb99-blw6l 1/1 Running
0 18h 10.42.114.60 ant-vm-29
dev-aai-babel-8475d8bdf5-bsfxr 2/2 Running
0 18h 10.42.185.69 ant-vm-32
dev-aai-cassandra-0 1/1 Running
1 18h 10.42.38.193 ant-vm-28
dev-aai-cassandra-1 1/1 Running
0 18h 10.42.106.243 ant-vm-29
dev-aai-cassandra-2 1/1 Running
1 18h 10.42.240.206 ant-vm-32
dev-aai-champ-56b87c5c87-nrjmk 1/2 CrashLoopBackOff
214 18h 10.42.243.152 ant-vm-29
dev-aai-data-router-5bcf4f7996-r5m4q 0/1 CrashLoopBackOff
212 18h 10.42.241.170 ant-vm-32
dev-aai-elasticsearch-6f7c8cdcd9-5pw5m 1/1 Running
0 18h 10.42.204.241 ant-vm-29
dev-aai-gizmo-65ff8c8dc7-pwblp 2/2 Running
0 18h 10.42.213.170 ant-vm-29
dev-aai-graphadmin-78bdc6d88f-fmm8f 2/2 Running
0 18h 10.42.108.101 ant-vm-28
dev-aai-modelloader-f79cc94fd-hpsq6 2/2 Running
0 18h 10.42.62.243 ant-vm-32
dev-aai-resources-644494cbf-g7vf9 2/2 Running
0 18h 10.42.1.133 ant-vm-29
dev-aai-search-data-6c88ccdc97-pjls2 2/2 Running
0 18h 10.42.155.160 ant-vm-32
dev-aai-sparky-be-7f657c6dcc-qwnhz 1/2 CrashLoopBackOff
214 18h 10.42.74.225 ant-vm-28
dev-aai-spike-68b68b5f6c-jxjxm 2/2 Running
0 18h 10.42.244.96 ant-vm-28
dev-aai-traversal-74b5c96c4f-2kvlq 2/2 Running
0 18h 10.42.183.65 ant-vm-28
_____
%: kubectl describe pod/dev-aai-champ-56b87c5c87-nrjmk -n onap
Name: dev-aai-champ-56b87c5c87-nrjmk
Namespace: onap
Node: ant-vm-29/135.207.129.119
Start Time: Tue, 18 Sep 2018 22:19:20 +0000
Labels: app=aai-champ
pod-template-hash=1264371743
release=dev
Annotations: <none>
Status: Running
IP: 10.42.243.152
Controlled By: ReplicaSet/dev-aai-champ-56b87c5c87
Containers:
aai-champ:
Container ID:
docker://fb970b10b58c197d4e8b02a7ed2377cc39f1b8f2224a293ee14517800b5752b0
Image: nexus3.onap.org:10001/onap/champ:1.2.3
Image ID:
docker-pullable://nexus3.onap.org:10001/onap/champ@sha256:22057abf7aa6002e6f
2fd90537c64e4d95a4ebb405947319b4cfd210f103f66b
Port: 9522/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 19 Sep 2018 16:36:31 +0000
Finished: Wed, 19 Sep 2018 16:36:33 +0000
Ready: False
Restart Count: 215
Readiness: tcp-socket :9522 delay=10s timeout=1s period=10s
#success=1 #failure=3
Environment:
CONFIG_HOME: /opt/app/champ-service/appconfig
GRAPHIMPL: janus-deps
KEY_STORE_PASSWORD: <set to the key 'KEY_STORE_PASSWORD' in secret
'dev-aai-champ-pass'> Optional: false
KEY_MANAGER_PASSWORD: <set to the key 'KEY_MANAGER_PASSWORD' in
secret 'dev-aai-champ-pass'> Optional: false
SERVICE_BEANS: /opt/app/champ-service/dynamic/conf
Mounts:
/etc/localtime from localtime (ro)
/opt/app/champ-service/appconfig/auth from dev-aai-champ-secrets (rw)
/opt/app/champ-service/appconfig/champ-api.properties from
dev-aai-champ-config (rw)
/opt/app/champ-service/bundleconfig/etc/logback.xml from
dev-aai-champ-logback-config (rw)
/opt/app/champ-service/dynamic/conf/champ-beans.xml from
dev-aai-champ-dynamic-config (rw)
/var/log/onap from dev-aai-champ-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz
(ro)
filebeat-onap:
Container ID:
docker://0ed35a07aab5e85a119ee72de03c835530cad645451e3f1b1d80093fd53e697a
Image: docker.elastic.co/beats/filebeat:5.5.0
Image ID:
docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288
f067f7b31ebde14644c4722d9f7960f176d621097a5942
Port: <none>
State: Running
Started: Tue, 18 Sep 2018 22:24:34 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/filebeat/data from aai-filebeat (rw)
/usr/share/filebeat/filebeat.yml from filebeat-conf (rw)
/var/log/onap from dev-aai-champ-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz
(ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
dev-aai-champ-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-champ
Optional: false
dev-aai-champ-secrets:
Type: Secret (a volume populated by a Secret)
SecretName: dev-aai-champ-champ
Optional: false
dev-aai-champ-dynamic-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-champ-dynamic
Optional: false
dev-aai-champ-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
dev-aai-champ-logback-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-champ-log-configmap
Optional: false
filebeat-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: aai-filebeat
Optional: false
aai-filebeat:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-l6dpz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-l6dpz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 3m (x4742 over 18h) kubelet, ant-vm-29 Back-off
restarting failed container
_____
%: kubectl describe pod/dev-aai-data-router-5bcf4f7996-r5m4q -n onap
Name: dev-aai-data-router-5bcf4f7996-r5m4q
Namespace: onap
Node: ant-vm-32/135.207.129.122
Start Time: Tue, 18 Sep 2018 22:19:31 +0000
Labels: app=aai-data-router
pod-template-hash=1679093552
release=dev
Annotations: <none>
Status: Running
IP: 10.42.241.170
Controlled By: ReplicaSet/dev-aai-data-router-5bcf4f7996
Init Containers:
init-sysctl:
Container ID:
docker://1895dc38095bfe29433bfc95a4ca016587fee8cb2a1cb41ddeb38d3d2a51fb30
Image: docker.io/busybox
Image ID:
docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159e
d11a251a09d115fdff9bd
Port: <none>
Command:
/bin/sh
-c
mkdir -p /logroot/data-router/logs
chmod -R 777 /logroot/data-router/logs
chown -R root:root /logroot
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 18 Sep 2018 22:19:56 +0000
Finished: Tue, 18 Sep 2018 22:19:56 +0000
Ready: True
Restart Count: 0
Environment:
NAMESPACE: onap (v1:metadata.namespace)
Mounts:
/logroot/ from dev-aai-data-router-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz
(ro)
Containers:
aai-data-router:
Container ID:
docker://a4e08ca74c86b14d5df29e8674fbdb0aaa03f4cd80ee993142f0f2c6cd6ab90c
Image: nexus3.onap.org:10001/onap/data-router:1.2.2
Image ID:
docker-pullable://nexus3.onap.org:10001/onap/data-router@sha256:2da5ded870c0
e775f5b90cbc257363b5732275ce0f4ec6fef75b709c0044911d
Port: 9502/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 19 Sep 2018 16:38:13 +0000
Finished: Wed, 19 Sep 2018 16:38:20 +0000
Ready: False
Restart Count: 213
Liveness: tcp-socket :9502 delay=10s timeout=1s period=10s
#success=1 #failure=3
Readiness: tcp-socket :9502 delay=10s timeout=1s period=10s
#success=1 #failure=3
Environment:
SERVICE_BEANS: /opt/app/data-router/dynamic/conf
CONFIG_HOME: /opt/app/data-router/config/
KEY_STORE_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10
DYNAMIC_ROUTES: /opt/app/data-router/dynamic/routes
KEY_MANAGER_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10
PATH:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
JAVA_HOME: usr/lib/jvm/java-8-openjdk-amd64
Mounts:
/etc/localtime from localtime (ro)
/logs/ from dev-aai-data-router-logs (rw)
/opt/app/data-router/config/auth from dev-aai-data-router-auth (rw)
/opt/app/data-router/config/data-router.properties from
dev-aai-data-router-properties (rw)
/opt/app/data-router/config/schemaIngest.properties from
dev-aai-data-router-properties (rw)
/opt/app/data-router/dynamic/conf/entity-event-policy.xml from
dev-aai-data-router-dynamic-policy (rw)
/opt/app/data-router/dynamic/routes/entity-event.route from
dev-aai-data-router-dynamic-route (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz
(ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
dev-aai-data-router-auth:
Type: Secret (a volume populated by a Secret)
SecretName: dev-aai-data-router
Optional: false
dev-aai-data-router-properties:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-data-router-prop
Optional: false
dev-aai-data-router-dynamic-route:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-data-router-dynamic
Optional: false
dev-aai-data-router-dynamic-policy:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-data-router-dynamic
Optional: false
dev-aai-data-router-logs:
Type: HostPath (bare host directory volume)
Path: /dockerdata-nfs/dev/aai/data-router/logs
default-token-l6dpz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-l6dpz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 4m (x4895 over 18h) kubelet, ant-vm-32 Back-off
restarting failed container
_____
%: kubectl describe pod/dev-aai-sparky-be-7f657c6dcc-qwnhz -n onap
Name: dev-aai-sparky-be-7f657c6dcc-qwnhz
Namespace: onap
Node: ant-vm-28/135.207.129.118
Start Time: Tue, 18 Sep 2018 22:19:32 +0000
Labels: app=aai-sparky-be
pod-template-hash=3921372877
release=dev
Annotations: <none>
Status: Running
IP: 10.42.74.225
Controlled By: ReplicaSet/dev-aai-sparky-be-7f657c6dcc
Init Containers:
aai-sparky-be-readiness:
Container ID:
docker://e00f4b6a665f6e7fa0f56a176d8ccd03294906b27f1ff05581d095544e71fc41
Image: oomk8s/readiness-check:2.0.0
Image ID:
docker-pullable://oomk8s/readiness-check@sha256:7daa08b81954360a1111d03364fe
bcb3dcfeb723bcc12ce3eb3ed3e53f2323ed
Port: <none>
Command:
/root/ready.py
Args:
--container-name
aai-elasticsearch
--container-name
aai-search-data
--container-name
aai
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 18 Sep 2018 22:19:42 +0000
Finished: Tue, 18 Sep 2018 22:33:08 +0000
Ready: True
Restart Count: 0
Environment:
NAMESPACE: onap (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz
(ro)
Containers:
aai-sparky-be:
Container ID:
docker://5f09aeebe7e7f979e9d55b2a79131b555c8c0eff12d7386ee4613525306bc72f
Image: nexus3.onap.org:10001/onap/sparky-be:1.2.1
Image ID:
docker-pullable://nexus3.onap.org:10001/onap/sparky-be@sha256:93fb133d332e92
efdf06056dfc76ecaf649d4034201827865b882543b3c012b1
Ports: 9517/TCP, 8000/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 19 Sep 2018 16:36:40 +0000
Finished: Wed, 19 Sep 2018 16:36:41 +0000
Ready: False
Restart Count: 215
Liveness: tcp-socket :9517 delay=10s timeout=1s period=10s
#success=1 #failure=3
Readiness: tcp-socket :9517 delay=10s timeout=1s period=10s
#success=1 #failure=3
Environment: <none>
Mounts:
/etc/localtime from localtime (ro)
/opt/app/sparky/config/application-oxm-default.properties from
dev-aai-sparky-be-properties (rw)
/opt/app/sparky/config/application-oxm-override.properties from
dev-aai-sparky-be-properties (rw)
/opt/app/sparky/config/application-oxm-schema-prod.properties from
dev-aai-sparky-be-properties (rw)
/opt/app/sparky/config/application-resources.properties from
dev-aai-sparky-be-properties (rw)
/opt/app/sparky/config/application-ssl.properties from
dev-aai-sparky-be-properties (rw)
/opt/app/sparky/config/application.properties from
dev-aai-sparky-be-properties (rw)
/opt/app/sparky/config/auth/ from dev-aai-sparky-be-auth-config (rw)
/opt/app/sparky/config/portal/ from dev-aai-sparky-be-portal-config
(rw)
/opt/app/sparky/config/portal/BOOT-INF/classes/ from
dev-aai-sparky-be-portal-config-props (rw)
/var/log/onap from dev-aai-sparky-be-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz
(ro)
filebeat-onap:
Container ID:
docker://b8af1b665ba68b31be87d3349897f86e65812afae07f517d6495c1f7f5768149
Image: docker.elastic.co/beats/filebeat:5.5.0
Image ID:
docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288
f067f7b31ebde14644c4722d9f7960f176d621097a5942
Port: <none>
State: Running
Started: Tue, 18 Sep 2018 22:33:37 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/filebeat/data from aai-sparky-filebeat (rw)
/usr/share/filebeat/filebeat.yml from filebeat-conf (rw)
/var/log/onap from dev-aai-sparky-be-logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz
(ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
dev-aai-sparky-be-properties:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-sparky-be-prop
Optional: false
dev-aai-sparky-be-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-sparky-be
Optional: false
dev-aai-sparky-be-portal-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-sparky-be-portal
Optional: false
dev-aai-sparky-be-portal-config-props:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-aai-sparky-be-portal-props
Optional: false
dev-aai-sparky-be-auth-config:
Type: Secret (a volume populated by a Secret)
SecretName: dev-aai-sparky-be
Optional: false
filebeat-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: aai-filebeat
Optional: false
dev-aai-sparky-be-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
aai-sparky-filebeat:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
modeldir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-token-l6dpz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-l6dpz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 23s (x4905 over 18h) kubelet, ant-vm-28 Back-off
restarting failed container
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#12525): https://lists.onap.org/g/onap-discuss/message/12525
Mute This Topic: https://lists.onap.org/mt/25757716/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
