Hi, Kapil/Team,
I just tried to bring up cds pods in my ONAP cluster (Frankfurt). But I always
get dev-cds-listener failure. Any insight? Please help.
dev-cds-blueprints-processor-788f99666f-xq99x 1/1 Running 0
23m 10.42.1.150 onap-k8s-vm-2 <none> <none>
dev-cds-command-executor-75bf75c564-bfcb4 1/1 Running 0
23m 10.42.2.248 onap-k8s-vm-1 <none> <none>
dev-cds-db-0
1/1 Running 0 23m 10.42.2.250 onap-k8s-vm-1
<none> <none>
dev-cds-py-executor-6d5dd756f-fr68c 1/1 Running
0 23m 10.42.2.247 onap-k8s-vm-1 <none> <none>
dev-cds-sdc-listener-678cbdcd4d-x6rkx 0/1
CrashLoopBackOff 8 23m 10.42.2.246 onap-k8s-vm-1 <none>
<none>
dev-cds-ui-6d55754b65-zkrfb 1/1
Running 0 23m 10.42.2.249 onap-k8s-vm-1 <none>
<none>
Here is the log:
-------------------------------------------------------------------------------------------------------------------------------
ubuntu@onap-rancher-vm:~$ kubectl logs -n onap
dev-cds-sdc-listener-678cbdcd4d-x6rkx
Starting SDC Listener
APP Config HOME : /opt/app/onap/config
20:58:33,624 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
NOT find resource [logback-test.xml]
20:58:33,624 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could
NOT find resource [logback.groovy]
20:58:33,624 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found
resource [logback.xml] at
[jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar!/logback.xml]
20:58:33,625 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource
[logback.xml] occurs multiple times on the classpath.
20:58:33,625 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource
[logback.xml] occurs at
[jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar!/logback.xml]
20:58:33,625 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource
[logback.xml] occurs at [file:/opt/app/onap/config/logback.xml]
20:58:33,643 |-INFO in
ch.qos.logback.core.joran.spi.ConfigurationWatchList@77cd7a0 - URL
[jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar!/logback.xml]
is not of type file
20:58:33,705 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction
- debug attribute not set
20:58:33,706 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About
to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
20:58:33,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming
appender as [STDOUT]
20:58:33,723 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA
- Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder]
for [encoder] property
20:58:33,790 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction -
Setting level of ROOT logger to INFO
20:58:33,790 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction -
Attaching appender named [STDOUT] to Logger[ROOT]
20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction -
Setting level of logger [org.springframework] to INFO
20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction -
Setting level of logger [org.springframework.web] to INFO
20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction -
Setting level of logger [org.onap.ccsdk.cds] to INFO
20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction
- End of configuration.
20:58:33,794 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@204f30ec
- Registering current configuration as safe fallback point
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.2.4.RELEASE)
2020-08-05 20:58:34,624|||main||||INFO||||||| Starting SdcListenerApplication
v0.7.2 on dev-cds-sdc-listener-678cbdcd4d-x6rkx with PID 6
(/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar started
by onap in /)
2020-08-05 20:58:34,628|||main||||INFO||||||| No active profile set, falling
back to default profiles: default
2020-08-05 20:58:38,466|||main||||INFO||||||| Exposing 13 endpoint(s) beneath
base path '/actuator'
2020-08-05 20:58:38,813|||main||||INFO|||||||
Using generated security password: 035a7a05-4dc7-4de8-a9bb-29f774c4446c
2020-08-05 20:58:39,162|||main||||INFO||||||| Netty started on port(s): 8080
2020-08-05 20:58:39,165|||main||||INFO||||||| Started SdcListenerApplication in
5.163 seconds (JVM running for 5.901)
2020-08-05 20:58:39,167|||main||||INFO||||||| Initialize the SDC distribution
client
2020-08-05 20:58:40,062|||main||||ERROR||||||| status from ASDC is
org.onap.sdc.http.HttpAsdcResponse@502c2278
2020-08-05 20:58:40,063|||main||||ERROR||||||| DistributionClientResultImpl
[responseStatus=ASDC_SERVER_PROBLEM, responseMessage=ASDC server problem]
2020-08-05 20:58:40,074|||main||||WARN||||||| DELETE
http://message-router.onap:3904/apiKeys/EocCVzI1y7IlLBzc will send credentials
over a clear channel.
2020-08-05 20:58:40,129|||main||||ERROR||||||| Application run failed
java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener
method
HandlerMethod details:
Bean [org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient]
Method [public void
org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.initSdcClient() throws
org.onap.ccsdk.cds.sdclistener.exceptions.SdcListenerException]
Resolved arguments:
at
org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:322)
at
org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:190)
at
org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:153)
at
org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at
org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at
org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at
org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:403)
at
org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:360)
at
org.springframework.boot.context.event.EventPublishingRunListener.running(EventPublishingRunListener.java:103)
at
org.springframework.boot.SpringApplicationRunListeners.running(SpringApplicationRunListeners.java:77)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:330)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
at
org.springframework.boot.SpringApplication.run(SpringApplication.java:1215)
at
org.onap.ccsdk.cds.sdclistener.SdcListenerApplication.main(SdcListenerApplication.java:30)
Caused by: org.onap.ccsdk.cds.sdclistener.exceptions.SdcListenerException: SDC
distribution client init failed with reason:ASDC server problem
at
org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.startSdcClientBasedOnTheResult(SdcListenerClient.java:75)
at
org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.initSdcClient(SdcListenerClient.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:305)
... 13 common frames omitted
2020-08-05 20:58:40,136|||main||||INFO||||||| Stopping GRPC connection to CDS
backend
And description:
ubuntu@onap-rancher-vm:~$ kubectl describe po -n onap
dev-cds-sdc-listener-678cbdcd4d-x6rkx
Name: dev-cds-sdc-listener-678cbdcd4d-x6rkx
Namespace: onap
Priority: 0
Node: onap-k8s-vm-1/167.254.204.51
Start Time: Wed, 05 Aug 2020 20:49:43 +0000
Labels: app=cds-sdc-listener
pod-template-hash=678cbdcd4d
release=dev
Annotations: cni.projectcalico.org/podIP: 10.42.2.246/32
Status: Running
IP: 10.42.2.246
Controlled By: ReplicaSet/dev-cds-sdc-listener-678cbdcd4d
Init Containers:
cds-sdc-listener-readiness:
Container ID:
docker://773097670ad4689e0765084b05fc965bc2d2d2a18f2b07c83c9c7d6504ec18fe
Image: oomk8s/readiness-check:2.0.2
Image ID:
docker-pullable://oomk8s/readiness-check@sha256:875814cc853d7d9b3227165740584c347884bc8c0d831b4148cb5f7952e4cbff
Port: <none>
Host Port: <none>
Command:
/root/ready.py
Args:
--container-name
sdc-be
--container-name
message-router
--container-name
cds-blueprints-processor
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 05 Aug 2020 20:50:03 +0000
Finished: Wed, 05 Aug 2020 20:53:55 +0000
Ready: True
Restart Count: 0
Environment:
NAMESPACE: onap (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-2qr2x
(ro)
Containers:
cds-sdc-listener:
Container ID:
docker://3b5432c3e7b6eaeeb57b0d9c0b372963f0ac205f6932c82c25363bbb499c9a45
Image: nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2
Image ID:
docker-pullable://nexus3.onap.org:10001/onap/ccsdk-sdclistener@sha256:50b33cb972a20a2a56a5bf997c852a1f4824be77ed71669181af359f69bf1f5e
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 05 Aug 2020 20:58:33 +0000
Finished: Wed, 05 Aug 2020 20:58:43 +0000
Ready: False
Restart Count: 5
Liveness: http-get http://:8080/api/v1/sdclistener/healthcheck
delay=10s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:8080/api/v1/sdclistener/healthcheck
delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
APP_CONFIG_HOME: /opt/app/onap/config
Mounts:
/opt/app/onap/config/application.yaml from dev-cds-sdc-listener-config
(rw,path="application.yaml")
/opt/app/onap/config/logback.xml from dev-cds-sdc-listener-config
(rw,path="logback.xml")
/var/run/secrets/kubernetes.io/serviceaccount from default-token-2qr2x
(ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
dev-cds-sdc-listener-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-cds-sdc-listener-configmap
Optional: false
default-token-2qr2x:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-2qr2x
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 14m kubelet, onap-k8s-vm-1 Container
image "oomk8s/readiness-check:2.0.2" already present on machine
Normal Created 14m kubelet, onap-k8s-vm-1 Created
container cds-sdc-listener-readiness
Normal Started 14m kubelet, onap-k8s-vm-1 Started
container cds-sdc-listener-readiness
Normal Scheduled 10m default-scheduler
Successfully assigned onap/dev-cds-sdc-listener-678cbdcd4d-x6rkx to
onap-k8s-vm-1
Normal Pulling 10m kubelet, onap-k8s-vm-1 Pulling
image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2"
Normal Pulled 9m53s kubelet, onap-k8s-vm-1
Successfully pulled image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2"
Warning Unhealthy 9m32s kubelet, onap-k8s-vm-1 Readiness
probe failed: Get http://10.42.2.246:8080/api/v1/sdclistener/healthcheck: dial
tcp 10.42.2.246:8080: connect: connection refused
Warning Unhealthy 9m30s kubelet, onap-k8s-vm-1 Liveness
probe failed: Get http://10.42.2.246:8080/api/v1/sdclistener/healthcheck: dial
tcp 10.42.2.246:8080: connect: connection refused
Normal Pulled 8m13s (x3 over 9m28s) kubelet, onap-k8s-vm-1 Container
image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2" already present on
machine
Normal Created 8m12s (x4 over 9m45s) kubelet, onap-k8s-vm-1 Created
container cds-sdc-listener
Normal Started 8m11s (x4 over 9m43s) kubelet, onap-k8s-vm-1 Started
container cds-sdc-listener
Warning BackOff 4m13s (x23 over 9m15s) kubelet, onap-k8s-vm-1 Back-off
restarting failed container
Thanks,
Xin Miao
Solution Engineering
Fujitsu Network Communication
(W)972-479-2263 (M)469-268-5226
2811 Telecom Drive
Richardson, TX 75081, USA
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21840): https://lists.onap.org/g/onap-discuss/message/21840
Mute This Topic: https://lists.onap.org/mt/76016611/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
