Hi Team, Could anyone please help me to resolve this issue.
My message router pod is stuck in init state due to which my dmaap and sdc pods are not comming up. Please find the events as follows for the pod: Name: development-message-router-0 Namespace: onap Priority: 0 Node: onap/40.40.40.55 Start Time: Mon, 29 Aug 2022 07:49:45 -0400 Labels: app.kubernetes.io/instance=development app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=message-router controller-revision-hash=development-message-router-5c9d585855 helm.sh/chart=message-router-10.0.0 statefulset.kubernetes.io/pod-name=development-message-router-0 Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: StatefulSet/development-message-router Init Containers: dmaap-mr-cert-initializer-readiness: Container ID: Image: nexus3.onap.org:10001/onap/oom/readiness:3.0.1 Image ID: Port: <none> Host Port: <none> Command: /app/ready.py Args: --container-name aaf-locate --container-name aaf-cm --container-name aaf-service State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Limits: cpu: 100m memory: 100Mi Requests: cpu: 3m memory: 20Mi Environment: NAMESPACE: onap (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dmtn2 (ro) message-router-aaf-config: Container ID: Image: nexus3.onap.org:10001/onap/aaf/aaf_agent:2.1.20 Image ID: Port: <none> Host Port: <none> Command: sh -c /opt/app/aaf_config/bin/agent.sh . /opt/app/aaf_config/bin/retrieval_check.sh /opt/app/aaf_config/bin/aaf-add-config.sh State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: APP_FQI: [email protected] aaf_locate_url: https://aaf-locate.onap:8095 aaf_locator_container_ns: onap aaf_locator_container: oom aaf_locator_fqdn: dmaap-mr aaf_locator_app_ns: org.osaaf.aaf DEPLOY_FQI: <set to the key 'login' in secret 'development-dmaap-mr-cert-initializer-deployer-creds'> Optional: false DEPLOY_PASSWORD: <set to the key 'password' in secret 'development-dmaap-mr-cert-initializer-deployer-creds'> Optional: false cadi_longitude: -122.26147 cadi_latitude: 37.78187 aaf_locator_public_fqdn: mr.dmaap.onap.org Mounts: /opt/app/aaf_config/bin/aaf-add-config.sh from aaf-add-config (rw,path="aaf-add-config.sh") /opt/app/aaf_config/bin/retrieval_check.sh from aaf-add-config (rw,path="retrieval_check.sh") /opt/app/aaf_config/cert/truststoreONAP.p12.b64 from aaf-agent-certs (rw,path="truststoreONAP.p12.b64") /opt/app/aaf_config/cert/truststoreONAPall.jks.b64 from aaf-agent-certs (rw,path="truststoreONAPall.jks.b64") /opt/app/osaaf from development-message-router-aaf-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dmtn2 (ro) message-router-update-config: Container ID: Image: docker.io/dibi/envsubst:1 Image ID: Port: <none> Host Port: <none> Command: sh Args: -c export $(cat /appl/dmaapMR1/bundleconfig/etc/sysprops/local/mycreds.prop | xargs -0); cd /config-input && for PFILE in `ls -1 .`; do envsubst <${PFILE} >/config/${PFILE}; done State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: <none> Mounts: /appl/dmaapMR1/bundleconfig/etc/sysprops from development-message-router-aaf-config (rw) /config from jetty (rw) /config-input from etc (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dmtn2 (ro) Containers: srimzi-zk-entrance: Container ID: Image: docker.io/scholzj/zoo-entrance:latest Image ID: Port: 2181/TCP Host Port: 0/TCP Command: /opt/stunnel/stunnel_run.sh State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Liveness: exec [/opt/stunnel/stunnel_healthcheck.sh 2181] delay=15s timeout=5s period=10s #success=1 #failure=3 Readiness: exec [/opt/stunnel/stunnel_healthcheck.sh 2181] delay=15s timeout=5s period=10s #success=1 #failure=3 Environment: LOG_LEVEL: debug STRIMZI_ZOOKEEPER_CONNECT: development-strimzi-zookeeper-client:2181 Mounts: /etc/cluster-ca-certs/ from cluster-ca-certs (rw) /etc/cluster-operator-certs/ from cluster-operator-certs (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dmtn2 (ro) message-router: Container ID: Image: nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.3.2 Image ID: Port: 3904/TCP Host Port: 0/TCP Command: sh Args: -c cp /jetty-config/ajsc-jetty.xml /appl/dmaapMR1/etc/ cp /jetty-config/cadi.properties /appl/dmaapMR1/bundleconfig/etc/sysprops/local/cadi.properties /bin/sh /appl/startup.sh State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Liveness: tcp-socket :api delay=10s timeout=1s period=10s #success=1 #failure=3 Readiness: tcp-socket :api delay=10s timeout=1s period=10s #success=1 #failure=3 Startup: tcp-socket :api delay=10s timeout=1s period=10s #success=1 #failure=70 Environment: JAASLOGIN: <set to the key 'sasl.jaas.config' in secret 'strimzi-kafka-admin'> Optional: false SASLMECH: scram-sha-512 enableCadi: true Mounts: /appl/dmaapMR1/bundleconfig/etc/appprops/MsgRtrApi.properties from appprops (rw,path="MsgRtrApi.properties") /appl/dmaapMR1/bundleconfig/etc/logback.xml from logback (rw,path="logback.xml") /appl/dmaapMR1/bundleconfig/etc/sysprops from development-message-router-aaf-config (rw) /appl/dmaapMR1/bundleconfig/etc/sysprops/sys-props.properties from sys-props (rw,path="sys-props.properties") /appl/dmaapMR1/etc/runner-web.xml from etc (rw,path="runner-web.xml") /etc/localtime from localtime (ro) /jetty-config from jetty (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dmtn2 (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: development-message-router-aaf-config: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: <unset> aaf-agent-certs: Type: ConfigMap (a volume populated by a ConfigMap) Name: development-cert-wrapper-certs Optional: false aaf-add-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: development-dmaap-mr-cert-initializer-add-config Optional: false localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: appprops: Type: ConfigMap (a volume populated by a ConfigMap) Name: development-message-router-msgrtrapi-prop-configmap Optional: false etc: Type: ConfigMap (a volume populated by a ConfigMap) Name: development-message-router-etc Optional: false logback: Type: ConfigMap (a volume populated by a ConfigMap) Name: development-message-router-logback-xml-configmap Optional: false sys-props: Type: ConfigMap (a volume populated by a ConfigMap) Name: development-message-router-sys-props Optional: false jetty: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> cluster-operator-certs: Type: Secret (a volume populated by a Secret) SecretName: development-strimzi-cluster-operator-certs Optional: false cluster-ca-certs: Type: Secret (a volume populated by a Secret) SecretName: development-strimzi-cluster-ca-cert Optional: false kube-api-access-dmtn2: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 5m25s (x572 over 21h) kubelet MountVolume.SetUp failed for volume "cluster-ca-certs" : secret "development-strimzi-cluster-ca-cert" not found Warning FailedMount 22s (x724 over 21h) kubelet (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[cluster-ca-certs cluster-operator-certs], unattached volumes=[sys-props kube-api-access-dmtn2 aaf-add-config cluster-ca-certs appprops development-message-router-aaf-config cluster-operator-certs localtime jetty etc aaf-agent-certs logback]: timed out waiting for the condition [root@ONAP message-router]# k get all -A | grep "cluster-ca-certs" We have tried the following links: We found the following link : https://lists.onap.org/g/onap-discuss/topic/onap_istanbul_sdnc/89018472?p=,,,20,0,0,0::recentpostdate/sticky,,,20,0,0,89018472,previd=0,nextid=1644331819719447215&previd=0&nextid=1644331819719447215 This link states that we need to install portal as well, as it's needed for the cert manager, which generates certificates on startup. We tried the mentioned link but no luck. Further, we have cert-manager and strimizi pods running, but they are running in a different name-space as suggested in the link: https://docs.onap.org/projects/onap-oom/en/jakarta/oom_user_guide.html#deploy Can this be the cause of the issue? Please help us out as we are completely stuck. Thanks and Regards, Kushagra Gupta -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#24172): https://lists.onap.org/g/onap-discuss/message/24172 Mute This Topic: https://lists.onap.org/mt/93247058/21656 Mute #dmaap:https://lists.onap.org/g/onap-discuss/mutehashtag/dmaap Mute #dmmap-bc:https://lists.onap.org/g/onap-discuss/mutehashtag/dmmap-bc Mute #jakarta:https://lists.onap.org/g/onap-discuss/mutehashtag/jakarta Mute #install:https://lists.onap.org/g/onap-discuss/mutehashtag/install Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
