Brian and John, Service distribution was working on SB-00 last week before rancher VM was rebooted. I compared docker image versions on SB-00 with currently installed OOM-Daily, they are all the same. SB-00 was deployed on 01/16/19, since then there were some OOM merges, but none of the changes seem related to SDC or DMaaP.
SDC-BE docker on SB-00 is still running on oom-k8s-10 node, in case you want to check logs. Regards, -Yang From: Lotoski, John [mailto:john.loto...@arris.com] Sent: Monday, January 21, 2019 12:29 PM To: FREEMAN, BRIAN D <bf1...@att.com>; Yang Xu (Yang, Fixed Network) <yang....@huawei.com>; onap-discuss@lists.onap.org Subject: RE: [onap-discuss] Sdnc Ueb Listener crashloopbackoff Hi Brian, Yes, I'm getting that type of error: 2019-01-21T17:19:14.519Z [pool-60-thread-1] WARN c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "CreateDistributionTopic" D_ENV = "AUTO" TOPIC_NAME = "SDC-D ISTR-NOTIF-TOPIC-AUTO" STATUS = "ALREADY_EXISTS" RequestId=null ErrorCategory=ERROR ServiceName=registerToTopic ErrorCode=500 TargetEntity=registerToTopic PUT http:/ /message-router.onap:3904/topics/SDC-DISTR-NOTIF-TOPIC-AUTO/producers/iPIxkpAMI8qTcQj8 will send credentials over a clear channel. As well as a few others in that log as well: 2019-01-21T17:22:41.978Z [qtp215145189-60383] ERROR o.o.s.c.config.EcompErrorLogUtil alarmSeverity=MAJOR AuditBeginTimestamp=2019-01-21 17:22:41.684Z AuditMessage=ACTION = "HttpAuthentication" URL = "v1/registerForDistribution" USER = "aai" AUTH_STATUS = "AUTH_SUCCESS" REALM = "ASDC" RequestId=48f70349-cca0-4ced-89a6-0d79876e2fa3 ErrorCategory=ERROR ServerIPAddress=172.16.16.227 ServiceName=/v1/registerForDistribution ErrorCode=500 PartnerName=Apache-HttpClient/4.5.6 (Java/1.8.0_171) auditOn=true ServerFQDN=onap-sdc-sdc-be-5cc4c4754-5xqpl TargetEntity=registerInDistributionEngine Error occured in Distribution Engine. Failed operation: registration validation failed 2019-01-21T17:22:41.978Z [qtp215145189-60383] INFO o.o.sdc.be.filters.BeServletFilter AuditMessage=ACTION = "HttpAuthentication" URL = "v1/registerForDistribution" USER = "aai" AUTH_STATUS = "AUTH_SUCCESS" REALM = "ASDC" AlertSeverity=0 ElapsedTime=294 EndTimestamp=2019-01-21 17:22:41.978Z auditOn=false ServerFQDN=onap-sdc-sdc-be-5cc4c4754-5xqplStatusCode=ERROR timer=294 ServiceInstanceId=null ClassName=org.openecomp.sdc.be.filters.BeServletFilter ResponseDescription=Internal Server Error ResponseCode=500 InstanceUUID=aai-ml RequestId=48f70349-cca0-4ced-89a6-0d79876e2fa3 PartnerName=Apache-HttpClient/4.5.6 (Java/1.8.0_171) TargetEntity=registerInDistributionEngine CustomField1=POST: https://sdc-be.onap:8443/sdc/v1/registerForDistribution CustomField2=500 AuditBeginTimestamp=2019-01-21 17:22:41.684Z RemoteHost=172.16.35.251 ErrorCategory=INFO ServerIPAddress=172.16.16.227 ServiceName=/v1/registerForDistribution ErrorCode=0 POST /sdc/v1/registerForDistribution HTTP/1.1 SC="500" 2019-01-21T17:22:14.361Z [pool-60-thread-1] INFO c.att.nsa.apiClient.http.HttpClient AuditMessage=ACTION = "AddKeyToTopicACL" D_ENV = "AUTO" TOPIC_NAME = "SDC-DISTR-NOTIF-TOPIC-AUTO" ROLE = "PRODUCER" API_KEY = "iPIxkpAMI8qTcQj8" STATUS = "500" RequestId=null ErrorCategory=ERROR ServiceName=registerToTopic ErrorCode=500 TargetEntity=registerToTopic \t--> HTTP/1.1 200 OK John From: FREEMAN, BRIAN D <bf1...@att.com<mailto:bf1...@att.com>> Sent: Monday, January 21, 2019 12:08 PM To: Lotoski, John <john.loto...@arris.com<mailto:john.loto...@arris.com>>; Yang Xu (Yang, Fixed Network) <yang....@huawei.com<mailto:yang....@huawei.com>>; onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org> Subject: RE: [onap-discuss] Sdnc Ueb Listener crashloopbackoff Checkt he SDC-BE logs (kubectl -n onap exec -it [sdc-be-pod] bash Cd /var/log/onap/sdc/sdc-be Error.log See if you are getting Servier Error 500 from the message-router:3904 publish events Brain From: Lotoski, John <john.loto...@arris.com<mailto:john.loto...@arris.com>> Sent: Monday, January 21, 2019 11:15 AM To: Yang Xu (Yang, Fixed Network) <yang....@huawei.com<mailto:yang....@huawei.com>>; FREEMAN, BRIAN D <bf1...@att.com<mailto:bf1...@att.com>>; onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org> Subject: RE: [onap-discuss] Sdnc Ueb Listener crashloopbackoff Hi Brian, Yang - thanks for the feedback and the ticket. Yes, I also see "Basic SDC Health Check (DMaaP:None)| PASS |" in the robot health check. In case it has any bearing on dependencies, in my environment I have not installed the following ONAP components: - dcaegen2, multicloud, oof, pnda, contrib, vfc The rest of the installed components in the system appear to be healthy based on the robot health check. Also, I did also notice a problem with the new data router image: onap/data-router:1.3.2 that is used in both AAI and pomba components. My triage workaround was to drop the image back to 1.3.1, which seems to have resolved the problem there, but might have other unintended effects I'm not aware of yet. See ticket: https://jira.onap.org/browse/AAI-2091<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__jira.onap.org_browse_AAI-2D2091%26d%3DDwMFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DOu1q4OhW8nZ0dvs55QcErUvR02XyXX1-_tHGDD1dpG8%26s%3DJy4Z4zSll5JuwjFcfg6WAr5CAS3kCDtEfDee78dtIp4%26e%3D&data=01%7C01%7CJohn.Lotoski%40arris.com%7C74ceaa53182b41037aa008d67fc35fbf%7Cf27929ade5544d55837ac561519c3091%7C1&sdata=mvwTibRpidegJ4gHD%2Fc9%2BOEUWa8xnjcWIRbsXNjKt0E%3D&reserved=0> This install was in AWS using EKS and Rancher 2.2. I have some brief lessons learned I can share that may save others time with that type of environment. I'll comment on that briefly in a separate email thread since infrastructure discussion here would be getting off topic a little. Thanks, John From: Yang Xu (Yang, Fixed Network) <yang....@huawei.com<mailto:yang....@huawei.com>> Sent: Monday, January 21, 2019 9:04 AM To: FREEMAN, BRIAN D <bf1...@att.com<mailto:bf1...@att.com>>; onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>; Lotoski, John <john.loto...@arris.com<mailto:john.loto...@arris.com>> Subject: RE: [onap-discuss] Sdnc Ueb Listener crashloopbackoff Brian, Good point, it should be Dmaap issue, will look into it. Just not sure why ueb-listener is in CrashLoopBackOff state. Regards, -Yang From: FREEMAN, BRIAN D [mailto:bf1...@att.com] Sent: Monday, January 21, 2019 8:55 AM To: Yang Xu (Yang, Fixed Network) <yang....@huawei.com<mailto:yang....@huawei.com>>; onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>; john.loto...@arris.com<mailto:john.loto...@arris.com> Subject: RE: [onap-discuss] Sdnc Ueb Listener crashloopbackoff If SDC HealthCheck shows DMaaP:None then SDC isnt talking to DMaaP so this is either a DMaaP or SDC issue not an SDNC issue. We added restart to ueb-listener if it couldnt communicate with DMaaP MR successfully. Brian From: Yang Xu (Yang, Fixed Network) <yang....@huawei.com<mailto:yang....@huawei.com>> Sent: Monday, January 21, 2019 8:53 AM To: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>; FREEMAN, BRIAN D <bf1...@att.com<mailto:bf1...@att.com>>; john.loto...@arris.com<mailto:john.loto...@arris.com> Subject: RE: [onap-discuss] Sdnc Ueb Listener crashloopbackoff It happens in windriver Integration lab OOM-Daily as well, SDC healthcheck shows DMaaP:None. Just logged a JIRA for it https://jira.onap.org/browse/SDNC-593<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.proofpoint.com%2Fv2%2Furl%3Fu%3Dhttps-3A__na01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Furldefense.proofpoint.com-252Fv2-252Furl-253Fu-253Dhttps-2D3A-5F-5Fjira.onap.org-5Fbrowse-5FSDNC-2D2D593-2526d-253DDwQFAg-2526c-253DLFYZ-2Do9-5FHUMeMTSQicvjIg-2526r-253De3d1ehx3DI5AoMgDmi2Fzw-2526m-253DcEhpsPfsYOqY8-5FULtaNYsAqq0-5FCRfWMUYwAjnB-5FdaRo-2526s-253DHUit2LElxdBKkFv5QWzTep5IZ5VKD9zft-5Fs2xcPeB7k-2526e-253D-26data-3D01-257C01-257Cjohn.lotoski-2540arris.com-257C75fe96f8528d4d700ce508d67fa94a51-257Cf27929ade5544d55837ac561519c3091-257C1-26sdata-3DGaTCmYBO7E0zBt3KUt6ZHx3HvbFy-252FskG6vn0LkpvPlI-253D-26reserved-3D0%26d%3DDwMFAg%26c%3DLFYZ-o9_HUMeMTSQicvjIg%26r%3De3d1ehx3DI5AoMgDmi2Fzw%26m%3DOu1q4OhW8nZ0dvs55QcErUvR02XyXX1-_tHGDD1dpG8%26s%3D1VjL9b8vCSkqKocTzH_uTDO1H929YTvarF1SOZpAahA%26e%3D&data=01%7C01%7CJohn.Lotoski%40arris.com%7C74ceaa53182b41037aa008d67fc35fbf%7Cf27929ade5544d55837ac561519c3091%7C1&sdata=B4jzov5vHxNrm4YLkvp3M9s3Hxk7nf6Xhy4y7I2cPGk%3D&reserved=0> Thanks John for reporting it and Brian for looking into it. Regards, -Yang From: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org> [mailto:onap-discuss@lists.onap.org] On Behalf Of Brian Sent: Monday, January 21, 2019 8:38 AM To: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>; john.loto...@arris.com<mailto:john.loto...@arris.com> Subject: Re: [onap-discuss] Sdnc Ueb Listener crashloopbackoff Are SDC and DMaaP MR passing health check ? SDC in partciular is BE reporting UP ? From: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org> <onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>> On Behalf Of Lotoski, John Sent: Monday, January 21, 2019 12:31 AM To: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org> Subject: [onap-discuss] Sdnc Ueb Listener crashloopbackoff Hi all, On an install using the latest git branch clone of Casablanca, the sdnc component has pod errors on the ueb-listener. I don't see anything in the pod logs and the pod describe doesn't really explain much. Has anyone seen this? onap-sdnc-controller-blueprints-db-0 1/1 Running 0 15m 172.16.37.193 ip-172-16-32-85.ec2.internal <none> onap-sdnc-nengdb-0 1/1 Running 0 15m 172.16.46.227 ip-172-16-32-85.ec2.internal <none> onap-sdnc-network-name-gen-9f5f44b69-dcfzj 1/1 Running 0 15m 172.16.40.105 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-0 2/2 Running 0 15m 172.16.27.216 ip-172-16-21-150.ec2.internal <none> onap-sdnc-sdnc-ansible-server-857cc5f6fc-24mjx 1/1 Running 0 15m 172.16.46.0 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-db-0 2/2 Running 0 15m 172.16.33.200 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-dgbuilder-56c796c68d-ht6tn 1/1 Running 0 15m 172.16.46.196 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-dmaap-listener-67d869f578-8fxj2 1/1 Running 0 15m 172.16.25.187 ip-172-16-21-150.ec2.internal <none> onap-sdnc-sdnc-portal-5fc5497489-nlp22 1/1 Running 0 15m 172.16.33.255 ip-172-16-32-85.ec2.internal <none> onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 0/1 CrashLoopBackOff 7 15m 172.16.38.233 ip-172-16-32-85.ec2.internal <none> $ kubectl -n onap logs -f onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 $ $ kubectl -n onap describe pod onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 Name: onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 Namespace: onap Priority: 0 PriorityClassName: <none> Node: ip-172-16-32-85.ec2.internal/172.16.32.85 Start Time: Mon, 21 Jan 2019 02:54:45 +0000 Labels: app=sdnc-ueb-listener pod-template-hash=2418945452 release=onap-sdnc Annotations: <none> Status: Running IP: 172.16.38.233 Controlled By: ReplicaSet/onap-sdnc-sdnc-ueb-listener-685df89896 Init Containers: sdnc-ueb-listener-readiness: Container ID: docker://689a4c34766d941426714beb7baf75357d27d59f9e30920dbd1bac5e9c37602c Image: oomk8s/readiness-check:2.0.0 Image ID: docker-pullable://oomk8s/readiness-check@sha256:7daa08b81954360a1111d03364febcb3dcfeb723bcc12ce3eb3ed3e53f2323ed Port: <none> Host Port: <none> Command: /root/ready.py Args: --container-name sdnc-db --container-name sdnc --container-name sdc-be --container-name message-router State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 21 Jan 2019 02:54:59 +0000 Finished: Mon, 21 Jan 2019 02:56:24 +0000 Ready: True Restart Count: 0 Environment: NAMESPACE: onap (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-dmnsz (ro) Containers: sdnc-ueb-listener: [16/1767] Container ID: docker://b937a8f3d058d80d63943b2062f18928cba52f7f7f634a665d1d83dd53111518 Image: nexus3.onap.org:10001/onap/sdnc-ueb-listener-image:1.4.4 Image ID: docker-pullable://nexus3.onap.org:10001/onap/sdnc-ueb-listener-image@sha256:ce1c0c0ed26583089e64381cecf3a404e3b90753ac2a8cacdcdbb32dd1851de9 Port: <none> Host Port: <none> Command: /opt/onap/sdnc/ueb-listener/bin/start-ueb-listener.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 21 Jan 2019 03:07:54 +0000 Finished: Mon, 21 Jan 2019 03:07:55 +0000 Ready: False Restart Count: 7 Environment: PROPERTY_DIR: /opt/onap/sdnc/data/properties SDNC_CONFIG_DIR: /opt/onap/sdnc/data/properties Mounts: /etc/localtime from localtime (ro) /opt/onap/sdnc/data/properties/dblib.properties from properties (rw) /opt/onap/sdnc/data/properties/ueb-listener.properties from properties (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-dmnsz (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: properties: Type: ConfigMap (a volume populated by a ConfigMap) Name: onap-sdnc-sdnc-ueb-listener Optional: false default-token-dmnsz: Type: Secret (a volume populated by a Secret) SecretName: default-token-dmnsz Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 17m default-scheduler Successfully assigned onap/onap-sdnc-sdnc-ueb-listener-685df89896-fcnd2 to ip-172-16-32-85.ec2.internal Normal Pulled 17m kubelet, ip-172-16-32-85.ec2.internal Container image "oomk8s/readiness-check:2.0.0" already present on machine Normal Created 17m kubelet, ip-172-16-32-85.ec2.internal Created container Normal Started 17m kubelet, ip-172-16-32-85.ec2.internal Started container Normal Created 13m (x5 over 15m) kubelet, ip-172-16-32-85.ec2.internal Created container Normal Started 13m (x5 over 15m) kubelet, ip-172-16-32-85.ec2.internal Started container Normal Pulled 12m (x6 over 15m) kubelet, ip-172-16-32-85.ec2.internal Container image "nexus3.onap.org:10001/onap/sdnc-ueb-listener-image:1.4.4" already present on machine Warning BackOff 2m (x46 over 14m) kubelet, ip-172-16-32-85.ec2.internal Back-off restarting failed container Thanks, John -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15063): https://lists.onap.org/g/onap-discuss/message/15063 Mute This Topic: https://lists.onap.org/mt/29355549/21656 Group Owner: onap-discuss+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
