Please change your images for all cds ms from 0.7.2 to 0.7.5.
[cid:[email protected]] Yuriy Malakov ONAP/ECOMP SA/SE 732-420-3030, Q-Chat<qto://talk/ym9479> [email protected]<mailto:[email protected]> From: [email protected] <[email protected]> On Behalf Of [email protected] Sent: Wednesday, August 05, 2020 7:06 PM To: MALAKOV, YURIY <[email protected]>; [email protected]; SINGAL, KAPIL <[email protected]> Subject: Re: [onap-discuss] [onap-cds] dev-cds-sdc-listener failure Yes, Yuriy, They are up and running. //........................................................................................................................................ ubuntu@onap-rancher-vm:~$ kubectl get po -n onap -o=wide | grep -i dev-dmaap dev-dmaap-bc-6c6db6d57-snw4w 1/1 Running 0 40d 10.42.1.187 onap-k8s-vm-2 <none> <none> dev-dmaap-dr-db-0 1/1 Running 0 40d 10.42.1.191 onap-k8s-vm-2 <none> <none> dev-dmaap-dr-db-1 1/1 Running 0 40d 10.42.2.156 onap-k8s-vm-1 <none> <none> dev-dmaap-dr-node-0 2/2 Running 0 40d 10.42.1.189 onap-k8s-vm-2 <none> <none> dev-dmaap-dr-prov-76c68d969b-l4fmb 2/2 Running 0 40d 10.42.1.188 onap-k8s-vm-2 <none> <none> ubuntu@onap-rancher-vm:~$ kubectl get po -n onap -o=wide | grep -i dev-message dev-message-router-0 1/1 Running 0 40d 10.42.2.152 onap-k8s-vm-1 <none> <none> dev-message-router-kafka-0 1/1 Running 2 40d 10.42.2.153 onap-k8s-vm-1 <none> <none> dev-message-router-kafka-1 1/1 Running 2 40d 10.42.1.190 onap-k8s-vm-2 <none> <none> dev-message-router-kafka-2 1/1 Running 2 40d 10.42.2.154 onap-k8s-vm-1 <none> <none> dev-message-router-zookeeper-0 1/1 Running 0 40d 10.42.1.193 onap-k8s-vm-2 <none> <none> dev-message-router-zookeeper-1 1/1 Running 0 40d 10.42.2.155 onap-k8s-vm-1 <none> <none> dev-message-router-zookeeper-2 1/1 Running 0 40d 10.42.1.192 onap-k8s-vm-2 <none> <none> ubuntu@onap-rancher-vm:~$ kubectl get po -n onap -o=wide | grep -i dev-sdc dev-sdc-be-5db6cc766d-wj72f 2/2 Running 0 40d 10.42.2.94 onap-k8s-vm-1 <none> <none> dev-sdc-be-config-backend-k8brm 0/1 Completed 0 40d 10.42.2.144 onap-k8s-vm-1 <none> <none> dev-sdc-cs-config-cassandra-bhj8p 0/1 Completed 0 40d 10.42.1.180 onap-k8s-vm-2 <none> <none> dev-sdc-dcae-be-6c779d9f75-2phnj 2/2 Running 0 40d 10.42.2.93 onap-k8s-vm-1 <none> <none> dev-sdc-dcae-be-tools-bxvw9 0/1 Completed 0 40d 10.42.2.145 onap-k8s-vm-1 <none> <none> dev-sdc-dcae-dt-dfc58fb4d-twbf8 2/2 Running 0 40d 10.42.1.135 onap-k8s-vm-2 <none> <none> dev-sdc-dcae-fe-5c748cdcfc-t8mgr 2/2 Running 0 40d 10.42.2.96 onap-k8s-vm-1 <none> <none> dev-sdc-dcae-tosca-lab-79789c87cc-5pzfv 2/2 Running 0 40d 10.42.1.137 onap-k8s-vm-2 <none> <none> dev-sdc-fe-5f9cbc6bd-zgkgv 2/2 Running 0 40d 10.42.1.138 onap-k8s-vm-2 <none> <none> dev-sdc-onboarding-be-749cf8745c-2qnrr 2/2 Running 0 40d 10.42.2.98 onap-k8s-vm-1 <none> <none> dev-sdc-onboarding-be-cassandra-init-cgxj8 0/1 Completed 0 40d 10.42.1.179 onap-k8s-vm-2 <none> <none> dev-sdc-wfd-be-db46c8797-4r9f8 1/1 Running 0 40d 10.42.1.139 onap-k8s-vm-2 <none> <none> dev-sdc-wfd-be-workflow-init-6v8hr 0/1 Completed 0 40d 10.42.1.177 onap-k8s-vm-2 <none> <none> dev-sdc-wfd-fe-6fb8d5c7b-skf9t 2/2 Running 0 40d 10.42.2.97 onap-k8s-vm-1 <none> <none> Thanks, Xin Miao Solution Engineering Fujitsu Network Communication (W)972-479-2263 (M)469-268-5226 2811 Telecom Drive Richardson, TX 75081, USA From: MALAKOV, YURIY [mailto:[email protected]] Sent: Wednesday, August 5, 2020 4:32 PM To: [email protected]<mailto:[email protected]>; Miao, Xin <[email protected]<mailto:[email protected]>>; SINGAL, KAPIL <[email protected]<mailto:[email protected]>> Subject: RE: [onap-cds] dev-cds-sdc-listener failure CDS Listener requires SDC & DMAAP to be up running to work. Can you verify that SDC & DMAAP is up & running. [cid:[email protected]] Yuriy Malakov ONAP/ECOMP SA/SE 732-420-3030, Q-Chat<qto://talk/ym9479> [email protected]<mailto:[email protected]> From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of [email protected]<mailto:[email protected]> Sent: Wednesday, August 05, 2020 5:25 PM To: SINGAL, KAPIL <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> Subject: [onap-discuss] [onap-cds] dev-cds-sdc-listener failure Hi, Kapil/Team, I just tried to bring up cds pods in my ONAP cluster (Frankfurt). But I always get dev-cds-listener failure. Any insight? Please help. dev-cds-blueprints-processor-788f99666f-xq99x 1/1 Running 0 23m 10.42.1.150 onap-k8s-vm-2 <none> <none> dev-cds-command-executor-75bf75c564-bfcb4 1/1 Running 0 23m 10.42.2.248 onap-k8s-vm-1 <none> <none> dev-cds-db-0 1/1 Running 0 23m 10.42.2.250 onap-k8s-vm-1 <none> <none> dev-cds-py-executor-6d5dd756f-fr68c 1/1 Running 0 23m 10.42.2.247 onap-k8s-vm-1 <none> <none> dev-cds-sdc-listener-678cbdcd4d-x6rkx 0/1 CrashLoopBackOff 8 23m 10.42.2.246 onap-k8s-vm-1 <none> <none> dev-cds-ui-6d55754b65-zkrfb 1/1 Running 0 23m 10.42.2.249 onap-k8s-vm-1 <none> <none> Here is the log: ------------------------------------------------------------------------------------------------------------------------------- ubuntu@onap-rancher-vm:~$ kubectl logs -n onap dev-cds-sdc-listener-678cbdcd4d-x6rkx Starting SDC Listener APP Config HOME : /opt/app/onap/config 20:58:33,624 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml] 20:58:33,624 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy] 20:58:33,624 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar!/logback.xml] 20:58:33,625 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath. 20:58:33,625 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar!/logback.xml] 20:58:33,625 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/opt/app/onap/config/logback.xml] 20:58:33,643 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@77cd7a0<mailto:ch.qos.logback.core.joran.spi.ConfigurationWatchList@77cd7a0> - URL [jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar!/logback.xml] is not of type file 20:58:33,705 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 20:58:33,706 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender] 20:58:33,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT] 20:58:33,723 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 20:58:33,790 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO 20:58:33,790 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT] 20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework] to INFO 20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework.web] to INFO 20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.onap.ccsdk.cds] to INFO 20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 20:58:33,794 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@204f30ec<mailto:ch.qos.logback.classic.joran.JoranConfigurator@204f30ec> - Registering current configuration as safe fallback point . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.2.4.RELEASE) 2020-08-05 20:58:34,624|||main||||INFO||||||| Starting SdcListenerApplication v0.7.2 on dev-cds-sdc-listener-678cbdcd4d-x6rkx with PID 6 (/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar started by onap in /) 2020-08-05 20:58:34,628|||main||||INFO||||||| No active profile set, falling back to default profiles: default 2020-08-05 20:58:38,466|||main||||INFO||||||| Exposing 13 endpoint(s) beneath base path '/actuator' 2020-08-05 20:58:38,813|||main||||INFO||||||| Using generated security password: 035a7a05-4dc7-4de8-a9bb-29f774c4446c 2020-08-05 20:58:39,162|||main||||INFO||||||| Netty started on port(s): 8080 2020-08-05 20:58:39,165|||main||||INFO||||||| Started SdcListenerApplication in 5.163 seconds (JVM running for 5.901) 2020-08-05 20:58:39,167|||main||||INFO||||||| Initialize the SDC distribution client 2020-08-05 20:58:40,062|||main||||ERROR||||||| status from ASDC is org.onap.sdc.http.HttpAsdcResponse@502c2278<mailto:org.onap.sdc.http.HttpAsdcResponse@502c2278> 2020-08-05 20:58:40,063|||main||||ERROR||||||| DistributionClientResultImpl [responseStatus=ASDC_SERVER_PROBLEM, responseMessage=ASDC server problem] 2020-08-05 20:58:40,074|||main||||WARN||||||| DELETE http://message-router.onap:3904/apiKeys/EocCVzI1y7IlLBzc<https://urldefense.proofpoint.com/v2/url?u=http-3A__message-2Drouter.onap-3A3904_apiKeys_EocCVzI1y7IlLBzc&d=DwQFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=4IIOMdJjMRUSnNTEG5jeUQ&m=zT7vVPPkImKWakzP_6T1Ka41_GJxSx5-nUEDGNemjEI&s=v--bJbR9WUKNvlajMqlJyl7BIqaYjuC9JtKgYT-JzCg&e=> will send credentials over a clear channel. 2020-08-05 20:58:40,129|||main||||ERROR||||||| Application run failed java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener method HandlerMethod details: Bean [org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient] Method [public void org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.initSdcClient() throws org.onap.ccsdk.cds.sdclistener.exceptions.SdcListenerException] Resolved arguments: at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:322) at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:190) at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:153) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:403) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:360) at org.springframework.boot.context.event.EventPublishingRunListener.running(EventPublishingRunListener.java:103) at org.springframework.boot.SpringApplicationRunListeners.running(SpringApplicationRunListeners.java:77) at org.springframework.boot.SpringApplication.run(SpringApplication.java:330) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) at org.onap.ccsdk.cds.sdclistener.SdcListenerApplication.main(SdcListenerApplication.java:30) Caused by: org.onap.ccsdk.cds.sdclistener.exceptions.SdcListenerException: SDC distribution client init failed with reason:ASDC server problem at org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.startSdcClientBasedOnTheResult(SdcListenerClient.java:75) at org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.initSdcClient(SdcListenerClient.java:69) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:305) ... 13 common frames omitted 2020-08-05 20:58:40,136|||main||||INFO||||||| Stopping GRPC connection to CDS backend And description: ubuntu@onap-rancher-vm:~$ kubectl describe po -n onap dev-cds-sdc-listener-678cbdcd4d-x6rkx Name: dev-cds-sdc-listener-678cbdcd4d-x6rkx Namespace: onap Priority: 0 Node: onap-k8s-vm-1/167.254.204.51 Start Time: Wed, 05 Aug 2020 20:49:43 +0000 Labels: app=cds-sdc-listener pod-template-hash=678cbdcd4d release=dev Annotations: cni.projectcalico.org/podIP: 10.42.2.246/32 Status: Running IP: 10.42.2.246 Controlled By: ReplicaSet/dev-cds-sdc-listener-678cbdcd4d Init Containers: cds-sdc-listener-readiness: Container ID: docker://773097670ad4689e0765084b05fc965bc2d2d2a18f2b07c83c9c7d6504ec18fe Image: oomk8s/readiness-check:2.0.2 Image ID: docker-pullable://oomk8s/readiness-check@sha256:875814cc853d7d9b3227165740584c347884bc8c0d831b4148cb5f7952e4cbff Port: <none> Host Port: <none> Command: /root/ready.py Args: --container-name sdc-be --container-name message-router --container-name cds-blueprints-processor State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 05 Aug 2020 20:50:03 +0000 Finished: Wed, 05 Aug 2020 20:53:55 +0000 Ready: True Restart Count: 0 Environment: NAMESPACE: onap (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-2qr2x (ro) Containers: cds-sdc-listener: Container ID: docker://3b5432c3e7b6eaeeb57b0d9c0b372963f0ac205f6932c82c25363bbb499c9a45 Image: nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2 Image ID: docker-pullable://nexus3.onap.org:10001/onap/ccsdk-sdclistener@sha256:50b33cb972a20a2a56a5bf997c852a1f4824be77ed71669181af359f69bf1f5e Port: 8080/TCP Host Port: 0/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Wed, 05 Aug 2020 20:58:33 +0000 Finished: Wed, 05 Aug 2020 20:58:43 +0000 Ready: False Restart Count: 5 Liveness: http-get http://:8080/api/v1/sdclistener/healthcheck delay=10s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/api/v1/sdclistener/healthcheck delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: APP_CONFIG_HOME: /opt/app/onap/config Mounts: /opt/app/onap/config/application.yaml from dev-cds-sdc-listener-config (rw,path="application.yaml") /opt/app/onap/config/logback.xml from dev-cds-sdc-listener-config (rw,path="logback.xml") /var/run/secrets/kubernetes.io/serviceaccount from default-token-2qr2x (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: dev-cds-sdc-listener-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-cds-sdc-listener-configmap Optional: false default-token-2qr2x: Type: Secret (a volume populated by a Secret) SecretName: default-token-2qr2x Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 14m kubelet, onap-k8s-vm-1 Container image "oomk8s/readiness-check:2.0.2" already present on machine Normal Created 14m kubelet, onap-k8s-vm-1 Created container cds-sdc-listener-readiness Normal Started 14m kubelet, onap-k8s-vm-1 Started container cds-sdc-listener-readiness Normal Scheduled 10m default-scheduler Successfully assigned onap/dev-cds-sdc-listener-678cbdcd4d-x6rkx to onap-k8s-vm-1 Normal Pulling 10m kubelet, onap-k8s-vm-1 Pulling image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2" Normal Pulled 9m53s kubelet, onap-k8s-vm-1 Successfully pulled image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2" Warning Unhealthy 9m32s kubelet, onap-k8s-vm-1 Readiness probe failed: Get http://10.42.2.246:8080/api/v1/sdclistener/healthcheck<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.42.2.246-3A8080_api_v1_sdclistener_healthcheck&d=DwQFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=4IIOMdJjMRUSnNTEG5jeUQ&m=zT7vVPPkImKWakzP_6T1Ka41_GJxSx5-nUEDGNemjEI&s=8UdlnXXlkERwdGJhHtivJQjlJBT1g86tCrXBJU8JdPk&e=>: dial tcp 10.42.2.246:8080: connect: connection refused Warning Unhealthy 9m30s kubelet, onap-k8s-vm-1 Liveness probe failed: Get http://10.42.2.246:8080/api/v1/sdclistener/healthcheck<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.42.2.246-3A8080_api_v1_sdclistener_healthcheck&d=DwQFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=4IIOMdJjMRUSnNTEG5jeUQ&m=zT7vVPPkImKWakzP_6T1Ka41_GJxSx5-nUEDGNemjEI&s=8UdlnXXlkERwdGJhHtivJQjlJBT1g86tCrXBJU8JdPk&e=>: dial tcp 10.42.2.246:8080: connect: connection refused Normal Pulled 8m13s (x3 over 9m28s) kubelet, onap-k8s-vm-1 Container image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2" already present on machine Normal Created 8m12s (x4 over 9m45s) kubelet, onap-k8s-vm-1 Created container cds-sdc-listener Normal Started 8m11s (x4 over 9m43s) kubelet, onap-k8s-vm-1 Started container cds-sdc-listener Warning BackOff 4m13s (x23 over 9m15s) kubelet, onap-k8s-vm-1 Back-off restarting failed container Thanks, Xin Miao Solution Engineering Fujitsu Network Communication (W)972-479-2263 (M)469-268-5226 2811 Telecom Drive Richardson, TX 75081, USA -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21843): https://lists.onap.org/g/onap-discuss/message/21843 Mute This Topic: https://lists.onap.org/mt/76016611/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
