AAI is still looking/fixing those three - they are not needed for distribution
Brian From: [email protected] <[email protected]> On Behalf Of George Clapp Sent: Wednesday, September 19, 2018 12:42 PM To: [email protected] Subject: [onap-discuss] [AAI] AAI containers in CrashLoopBackOff after OOM Rancher/Kubernetes installation I installed a set of ONAP components using OOM rancher/kubernetes, and some AAI containers have been in CrashLoopBackOff (please see the information below). I'd appreciate any suggestions on how to get them running. Thanks much, George ________________________________ %: kubectl get pods -o wide -n onap | grep -i aai dev-aai-5445ffdb99-blw6l 1/1 Running 0 18h 10.42.114.60 ant-vm-29 dev-aai-babel-8475d8bdf5-bsfxr 2/2 Running 0 18h 10.42.185.69 ant-vm-32 dev-aai-cassandra-0 1/1 Running 1 18h 10.42.38.193 ant-vm-28 dev-aai-cassandra-1 1/1 Running 0 18h 10.42.106.243 ant-vm-29 dev-aai-cassandra-2 1/1 Running 1 18h 10.42.240.206 ant-vm-32 dev-aai-champ-56b87c5c87-nrjmk 1/2 CrashLoopBackOff 214 18h 10.42.243.152 ant-vm-29 dev-aai-data-router-5bcf4f7996-r5m4q 0/1 CrashLoopBackOff 212 18h 10.42.241.170 ant-vm-32 dev-aai-elasticsearch-6f7c8cdcd9-5pw5m 1/1 Running 0 18h 10.42.204.241 ant-vm-29 dev-aai-gizmo-65ff8c8dc7-pwblp 2/2 Running 0 18h 10.42.213.170 ant-vm-29 dev-aai-graphadmin-78bdc6d88f-fmm8f 2/2 Running 0 18h 10.42.108.101 ant-vm-28 dev-aai-modelloader-f79cc94fd-hpsq6 2/2 Running 0 18h 10.42.62.243 ant-vm-32 dev-aai-resources-644494cbf-g7vf9 2/2 Running 0 18h 10.42.1.133 ant-vm-29 dev-aai-search-data-6c88ccdc97-pjls2 2/2 Running 0 18h 10.42.155.160 ant-vm-32 dev-aai-sparky-be-7f657c6dcc-qwnhz 1/2 CrashLoopBackOff 214 18h 10.42.74.225 ant-vm-28 dev-aai-spike-68b68b5f6c-jxjxm 2/2 Running 0 18h 10.42.244.96 ant-vm-28 dev-aai-traversal-74b5c96c4f-2kvlq 2/2 Running 0 18h 10.42.183.65 ant-vm-28 ________________________________ %: kubectl describe pod/dev-aai-champ-56b87c5c87-nrjmk -n onap Name: dev-aai-champ-56b87c5c87-nrjmk Namespace: onap Node: ant-vm-29/135.207.129.119 Start Time: Tue, 18 Sep 2018 22:19:20 +0000 Labels: app=aai-champ pod-template-hash=1264371743 release=dev Annotations: <none> Status: Running IP: 10.42.243.152 Controlled By: ReplicaSet/dev-aai-champ-56b87c5c87 Containers: aai-champ: Container ID: docker://fb970b10b58c197d4e8b02a7ed2377cc39f1b8f2224a293ee14517800b5752b0 Image: nexus3.onap.org:10001/onap/champ:1.2.3 Image ID: docker-pullable://nexus3.onap.org:10001/onap/champ@sha256:22057abf7aa6002e6f2fd90537c64e4d95a4ebb405947319b4cfd210f103f66b Port: 9522/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Wed, 19 Sep 2018 16:36:31 +0000 Finished: Wed, 19 Sep 2018 16:36:33 +0000 Ready: False Restart Count: 215 Readiness: tcp-socket :9522 delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: CONFIG_HOME: /opt/app/champ-service/appconfig GRAPHIMPL: janus-deps KEY_STORE_PASSWORD: <set to the key 'KEY_STORE_PASSWORD' in secret 'dev-aai-champ-pass'> Optional: false KEY_MANAGER_PASSWORD: <set to the key 'KEY_MANAGER_PASSWORD' in secret 'dev-aai-champ-pass'> Optional: false SERVICE_BEANS: /opt/app/champ-service/dynamic/conf Mounts: /etc/localtime from localtime (ro) /opt/app/champ-service/appconfig/auth from dev-aai-champ-secrets (rw) /opt/app/champ-service/appconfig/champ-api.properties from dev-aai-champ-config (rw) /opt/app/champ-service/bundleconfig/etc/logback.xml from dev-aai-champ-logback-config (rw) /opt/app/champ-service/dynamic/conf/champ-beans.xml from dev-aai-champ-dynamic-config (rw) /var/log/onap from dev-aai-champ-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz (ro) filebeat-onap: Container ID: docker://0ed35a07aab5e85a119ee72de03c835530cad645451e3f1b1d80093fd53e697a Image: docker.elastic.co/beats/filebeat:5.5.0 Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 Port: <none> State: Running Started: Tue, 18 Sep 2018 22:24:34 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /usr/share/filebeat/data from aai-filebeat (rw) /usr/share/filebeat/filebeat.yml from filebeat-conf (rw) /var/log/onap from dev-aai-champ-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime dev-aai-champ-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-champ Optional: false dev-aai-champ-secrets: Type: Secret (a volume populated by a Secret) SecretName: dev-aai-champ-champ Optional: false dev-aai-champ-dynamic-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-champ-dynamic Optional: false dev-aai-champ-logs: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: dev-aai-champ-logback-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-champ-log-configmap Optional: false filebeat-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: aai-filebeat Optional: false aai-filebeat: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: default-token-l6dpz: Type: Secret (a volume populated by a Secret) SecretName: default-token-l6dpz Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 3m (x4742 over 18h) kubelet, ant-vm-29 Back-off restarting failed container ________________________________ %: kubectl describe pod/dev-aai-data-router-5bcf4f7996-r5m4q -n onap Name: dev-aai-data-router-5bcf4f7996-r5m4q Namespace: onap Node: ant-vm-32/135.207.129.122 Start Time: Tue, 18 Sep 2018 22:19:31 +0000 Labels: app=aai-data-router pod-template-hash=1679093552 release=dev Annotations: <none> Status: Running IP: 10.42.241.170 Controlled By: ReplicaSet/dev-aai-data-router-5bcf4f7996 Init Containers: init-sysctl: Container ID: docker://1895dc38095bfe29433bfc95a4ca016587fee8cb2a1cb41ddeb38d3d2a51fb30 Image: docker.io/busybox Image ID: docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd Port: <none> Command: /bin/sh -c mkdir -p /logroot/data-router/logs chmod -R 777 /logroot/data-router/logs chown -R root:root /logroot State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 18 Sep 2018 22:19:56 +0000 Finished: Tue, 18 Sep 2018 22:19:56 +0000 Ready: True Restart Count: 0 Environment: NAMESPACE: onap (v1:metadata.namespace) Mounts: /logroot/ from dev-aai-data-router-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz (ro) Containers: aai-data-router: Container ID: docker://a4e08ca74c86b14d5df29e8674fbdb0aaa03f4cd80ee993142f0f2c6cd6ab90c Image: nexus3.onap.org:10001/onap/data-router:1.2.2 Image ID: docker-pullable://nexus3.onap.org:10001/onap/data-router@sha256:2da5ded870c0e775f5b90cbc257363b5732275ce0f4ec6fef75b709c0044911d Port: 9502/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Wed, 19 Sep 2018 16:38:13 +0000 Finished: Wed, 19 Sep 2018 16:38:20 +0000 Ready: False Restart Count: 213 Liveness: tcp-socket :9502 delay=10s timeout=1s period=10s #success=1 #failure=3 Readiness: tcp-socket :9502 delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: SERVICE_BEANS: /opt/app/data-router/dynamic/conf CONFIG_HOME: /opt/app/data-router/config/ KEY_STORE_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10 DYNAMIC_ROUTES: /opt/app/data-router/dynamic/routes KEY_MANAGER_PASSWORD: OBF:1y0q1uvc1uum1uvg1pil1pjl1uuq1uvk1uuu1y10 PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin JAVA_HOME: usr/lib/jvm/java-8-openjdk-amd64 Mounts: /etc/localtime from localtime (ro) /logs/ from dev-aai-data-router-logs (rw) /opt/app/data-router/config/auth from dev-aai-data-router-auth (rw) /opt/app/data-router/config/data-router.properties from dev-aai-data-router-properties (rw) /opt/app/data-router/config/schemaIngest.properties from dev-aai-data-router-properties (rw) /opt/app/data-router/dynamic/conf/entity-event-policy.xml from dev-aai-data-router-dynamic-policy (rw) /opt/app/data-router/dynamic/routes/entity-event.route from dev-aai-data-router-dynamic-route (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime dev-aai-data-router-auth: Type: Secret (a volume populated by a Secret) SecretName: dev-aai-data-router Optional: false dev-aai-data-router-properties: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-data-router-prop Optional: false dev-aai-data-router-dynamic-route: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-data-router-dynamic Optional: false dev-aai-data-router-dynamic-policy: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-data-router-dynamic Optional: false dev-aai-data-router-logs: Type: HostPath (bare host directory volume) Path: /dockerdata-nfs/dev/aai/data-router/logs default-token-l6dpz: Type: Secret (a volume populated by a Secret) SecretName: default-token-l6dpz Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 4m (x4895 over 18h) kubelet, ant-vm-32 Back-off restarting failed container ________________________________ %: kubectl describe pod/dev-aai-sparky-be-7f657c6dcc-qwnhz -n onap Name: dev-aai-sparky-be-7f657c6dcc-qwnhz Namespace: onap Node: ant-vm-28/135.207.129.118 Start Time: Tue, 18 Sep 2018 22:19:32 +0000 Labels: app=aai-sparky-be pod-template-hash=3921372877 release=dev Annotations: <none> Status: Running IP: 10.42.74.225 Controlled By: ReplicaSet/dev-aai-sparky-be-7f657c6dcc Init Containers: aai-sparky-be-readiness: Container ID: docker://e00f4b6a665f6e7fa0f56a176d8ccd03294906b27f1ff05581d095544e71fc41 Image: oomk8s/readiness-check:2.0.0 Image ID: docker-pullable://oomk8s/readiness-check@sha256:7daa08b81954360a1111d03364febcb3dcfeb723bcc12ce3eb3ed3e53f2323ed Port: <none> Command: /root/ready.py Args: --container-name aai-elasticsearch --container-name aai-search-data --container-name aai State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 18 Sep 2018 22:19:42 +0000 Finished: Tue, 18 Sep 2018 22:33:08 +0000 Ready: True Restart Count: 0 Environment: NAMESPACE: onap (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz (ro) Containers: aai-sparky-be: Container ID: docker://5f09aeebe7e7f979e9d55b2a79131b555c8c0eff12d7386ee4613525306bc72f Image: nexus3.onap.org:10001/onap/sparky-be:1.2.1 Image ID: docker-pullable://nexus3.onap.org:10001/onap/sparky-be@sha256:93fb133d332e92efdf06056dfc76ecaf649d4034201827865b882543b3c012b1 Ports: 9517/TCP, 8000/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Wed, 19 Sep 2018 16:36:40 +0000 Finished: Wed, 19 Sep 2018 16:36:41 +0000 Ready: False Restart Count: 215 Liveness: tcp-socket :9517 delay=10s timeout=1s period=10s #success=1 #failure=3 Readiness: tcp-socket :9517 delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: <none> Mounts: /etc/localtime from localtime (ro) /opt/app/sparky/config/application-oxm-default.properties from dev-aai-sparky-be-properties (rw) /opt/app/sparky/config/application-oxm-override.properties from dev-aai-sparky-be-properties (rw) /opt/app/sparky/config/application-oxm-schema-prod.properties from dev-aai-sparky-be-properties (rw) /opt/app/sparky/config/application-resources.properties from dev-aai-sparky-be-properties (rw) /opt/app/sparky/config/application-ssl.properties from dev-aai-sparky-be-properties (rw) /opt/app/sparky/config/application.properties from dev-aai-sparky-be-properties (rw) /opt/app/sparky/config/auth/ from dev-aai-sparky-be-auth-config (rw) /opt/app/sparky/config/portal/ from dev-aai-sparky-be-portal-config (rw) /opt/app/sparky/config/portal/BOOT-INF/classes/ from dev-aai-sparky-be-portal-config-props (rw) /var/log/onap from dev-aai-sparky-be-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz (ro) filebeat-onap: Container ID: docker://b8af1b665ba68b31be87d3349897f86e65812afae07f517d6495c1f7f5768149 Image: docker.elastic.co/beats/filebeat:5.5.0 Image ID: docker-pullable://docker.elastic.co/beats/filebeat@sha256:fe7602b641ed8ee288f067f7b31ebde14644c4722d9f7960f176d621097a5942 Port: <none> State: Running Started: Tue, 18 Sep 2018 22:33:37 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /usr/share/filebeat/data from aai-sparky-filebeat (rw) /usr/share/filebeat/filebeat.yml from filebeat-conf (rw) /var/log/onap from dev-aai-sparky-be-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-l6dpz (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime dev-aai-sparky-be-properties: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-sparky-be-prop Optional: false dev-aai-sparky-be-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-sparky-be Optional: false dev-aai-sparky-be-portal-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-sparky-be-portal Optional: false dev-aai-sparky-be-portal-config-props: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-aai-sparky-be-portal-props Optional: false dev-aai-sparky-be-auth-config: Type: Secret (a volume populated by a Secret) SecretName: dev-aai-sparky-be Optional: false filebeat-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: aai-filebeat Optional: false dev-aai-sparky-be-logs: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: aai-sparky-filebeat: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: modeldir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: default-token-l6dpz: Type: Secret (a volume populated by a Secret) SecretName: default-token-l6dpz Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 23s (x4905 over 18h) kubelet, ant-vm-28 Back-off restarting failed container -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12526): https://lists.onap.org/g/onap-discuss/message/12526 Mute This Topic: https://lists.onap.org/mt/25757716/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
