Hi Brian, Yang - thanks for the feedback and the ticket. Yes, I also see "Basic SDC Health Check (DMaaP:None)| PASS |" in the robot health check.
In case it has any bearing on dependencies, in my environment I have not installed the following ONAP components: * dcaegen2, multicloud, oof, pnda, contrib, vfc The rest of the installed components in the system appear to be healthy based on the robot health check. Also, I did also notice a problem with the new data router image: onap/data-router:1.3.2 that is used in both AAI and pomba components. My triage workaround was to drop the image back to 1.3.1, which seems to have resolved the problem there, but might have other unintended effects I'm not aware of yet. See ticket: https://jira.onap.org/browse/AAI-2091 This install was in AWS using EKS and Rancher 2.2. I have some brief lessons learned I can share that may save others time with that type of environment. I'll comment on that briefly in a separate email thread since infrastructure discussion here would be getting off topic a little. Thanks, John From: Yang Xu (Yang, Fixed Network) <[email protected]> Sent: Monday, January 21, 2019 9:04 AM To: FREEMAN, BRIAN D <[email protected]>; [email protected]; Lotoski, John <[email protected]> Subject: RE: [onap-discuss] Sdnc Ueb Listener crashloopbackoff Brian, Good point, it should be Dmaap issue, will look into it. Just not sure why ueb-listener is in CrashLoopBackOff state. Regards, -Yang From: FREEMAN, BRIAN D [mailto:[email protected]] Sent: Monday, January 21, 2019 8:55 AM To: Yang Xu (Yang, Fixed Network) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: [onap-discuss] Sdnc Ueb Listener crashloopbackoff If SDC HealthCheck shows DMaaP:None then SDC isnt talking to DMaaP so this is either a DMaaP or SDC issue not an SDNC issue. We added restart to ueb-listener if it couldnt communicate with DMaaP MR successfully. Brian From: Yang Xu (Yang, Fixed Network) <[email protected]<mailto:[email protected]>> Sent: Monday, January 21, 2019 8:53 AM To: [email protected]<mailto:[email protected]>; FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: RE: [onap-discuss] Sdnc Ueb Listener crashloopbackoff It happens in windriver Integration lab OOM-Daily as well, SDC healthcheck shows DMaaP:None. Just logged a JIRA for it https://jira.onap.org/browse/SDNC-593<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__jira.onap.org_browse_SDNC-2D593%26d%3DDwQFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DcEhpsPfsYOqY8_ULtaNYsAqq0_CRfWMUYwAjnB_daRo%26s%3DHUit2LElxdBKkFv5QWzTep5IZ5VKD9zft_s2xcPeB7k%26e%3D&data=01%7C01%7Cjohn.lotoski%40arris.com%7C75fe96f8528d4d700ce508d67fa94a51%7Cf27929ade5544d55837ac561519c3091%7C1&sdata=GaTCmYBO7E0zBt3KUt6ZHx3HvbFy%2FskG6vn0LkpvPlI%3D&reserved=0> Thanks John for reporting it and Brian for looking into it. Regards, -Yang From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Brian Sent: Monday, January 21, 2019 8:38 AM To: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: Re: [onap-discuss] Sdnc Ueb Listener crashloopbackoff Are SDC and DMaaP MR passing health check ? SDC in partciular is BE reporting UP ? From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of Lotoski, John Sent: Monday, January 21, 2019 12:31 AM To: [email protected]<mailto:[email protected]> Subject: [onap-discuss] Sdnc Ueb Listener crashloopbackoff Hi all, On an install using the latest git branch clone of Casablanca, the sdnc component has pod errors on the ueb-listener. I don't see anything in the pod logs and the pod describe doesn't really explain much. Has anyone seen this? onap-sdnc-controller-blueprints-db-0 1/1 Running 0 15m 172.16.37.193 ip-172-16-32-85.ec2.internal <none> onap-sdnc-nengdb-0 1/1 Running 0 15m 172.16.46.227 ip-172-16-32-85.ec2.internal <none> onap-sdnc-network-name-gen-9f5f44b69-dcfzj 1/1 Running 0 15m 172.16.40.105 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-0 2/2 Running 0 15m 172.16.27.216 ip-172-16-21-150.ec2.internal <none> onap-sdnc-sdnc-ansible-server-857cc5f6fc-24mjx 1/1 Running 0 15m 172.16.46.0 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-db-0 2/2 Running 0 15m 172.16.33.200 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-dgbuilder-56c796c68d-ht6tn 1/1 Running 0 15m 172.16.46.196 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-dmaap-listener-67d869f578-8fxj2 1/1 Running 0 15m 172.16.25.187 ip-172-16-21-150.ec2.internal <none> onap-sdnc-sdnc-portal-5fc5497489-nlp22 1/1 Running 0 15m 172.16.33.255 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 0/1 CrashLoopBackOff 7 15m 172.16.38.233 ip-172-16-32-85.ec2.internal <none> $ kubectl -n onap logs -f onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 $ $ kubectl -n onap describe pod onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 Name: onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 Namespace: onap Priority: 0 PriorityClassName: <none> Node: ip-172-16-32-85.ec2.internal/172.16.32.85 Start Time: Mon, 21 Jan 2019 02:54:45 +0000 Labels: app=sdnc-ueb-listener pod-template-hash=2418945452 release=onap-sdnc Annotations: <none> Status: Running IP: 172.16.38.233 Controlled By: ReplicaSet/onap-sdnc-sdnc-ueb-listener-685df89896 Init Containers: sdnc-ueb-listener-readiness: Container ID: docker://689a4c34766d941426714beb7baf75357d27d59f9e30920dbd1bac5e9c37602c Image: oomk8s/readiness-check:2.0.0 Image ID: docker-pullable://oomk8s/readiness-check@sha256:7daa08b81954360a1111d03364febcb3dcfeb723bcc12ce3eb3ed3e53f2323ed Port: <none> Host Port: <none> Command: /root/ready.py Args: --container-name sdnc-db --container-name sdnc --container-name sdc-be --container-name message-router State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 21 Jan 2019 02:54:59 +0000 Finished: Mon, 21 Jan 2019 02:56:24 +0000 Ready: True Restart Count: 0 Environment: NAMESPACE: onap (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-dmnsz (ro) Containers: sdnc-ueb-listener: [16/1767] Container ID: docker://b937a8f3d058d80d63943b2062f18928cba52f7f7f634a665d1d83dd53111518 Image: nexus3.onap.org:10001/onap/sdnc-ueb-listener-image:1.4.4 Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-ueb-listener-image@sha256:ce1c0c0ed26583089e64381cecf3a404e3b90753ac2a8cacdcdbb32dd1851de9 Port: <none> Host Port: <none> Command: /opt/onap/sdnc/ueb-listener/bin/start-ueb-listener.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 21 Jan 2019 03:07:54 +0000 Finished: Mon, 21 Jan 2019 03:07:55 +0000 Ready: False Restart Count: 7 Environment: PROPERTY_DIR: /opt/onap/sdnc/data/properties SDNC_CONFIG_DIR: /opt/onap/sdnc/data/properties Mounts: /etc/localtime from localtime (ro) /opt/onap/sdnc/data/properties/dblib.properties from properties (rw) /opt/onap/sdnc/data/properties/ueb-listener.properties from properties (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-dmnsz (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: properties: Type: ConfigMap (a volume populated by a ConfigMap) Name: onap-sdnc-sdnc-ueb-listener Optional: false default-token-dmnsz: Type: Secret (a volume populated by a Secret) SecretName: default-token-dmnsz Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 17m default-scheduler Successfully assigned onap/onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 to ip-172-16-32-85.ec2.internal Normal Pulled 17m kubelet, ip-172-16-32-85.ec2.internal Container image "oomk8s/readiness-check:2.0.0" already present on machine Normal Created 17m kubelet, ip-172-16-32-85.ec2.internal Created container Normal Started 17m kubelet, ip-172-16-32-85.ec2.internal Started container Normal Created 13m (x5 over 15m) kubelet, ip-172-16-32-85.ec2.internal Created container Normal Started 13m (x5 over 15m) kubelet, ip-172-16-32-85.ec2.internal Started container Normal Pulled 12m (x6 over 15m) kubelet, ip-172-16-32-85.ec2.internal Container image "nexus3.onap.org:10001/onap/sdnc-ueb-listener-image:1.4.4" already present on machine Warning BackOff 2m (x46 over 14m) kubelet, ip-172-16-32-85.ec2.internal Back-off restarting failed container Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15057): https://lists.onap.org/g/onap-discuss/message/15057 Mute This Topic: https://lists.onap.org/mt/29355549/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
