Hi Taka,

We have 3 node k8s cluster and each one is having 32 vCPU + 64GB RAM.

APPC was in the same state for a day and i tried bouncing dev-appc-appc-0
& dev-appc-appc-ansible-server. Now, i see that all APPC PODs are up and
running.

Not sure is this something to do with network issues in accessing ODL
repository or something. Is this expected?

*# Here are the image version details*

dev-appc-appc-0 Image:
nexus3.onap.org:10001/onap/appc-image:1.4.4
dev-appc-appc-ansible-server Image:
nexus3.onap.org:10001/onap/ccsdk-ansible-server-image:0.4.1-STAGING-latest

*Workaround*

# Today I tried bouncing dev-appc-appc-0 & dev-appc-appc-ansible-server
# Everything came up

*$ kubectl get pods -n onap | grep appc*
dev-appc-appc-0                                               2/2
 Running            0          41m
dev-appc-appc-ansible-server-6877b497df-rfgnj                 1/1
 Running            0          1m
dev-appc-appc-cdt-77bccf4847-fmtpw                            1/1
 Running            0          1d
dev-appc-appc-db-0                                            1/1
 Running            1          1d
dev-appc-appc-db-1                                            1/1
 Running            0          1d
dev-appc-appc-db-2                                            1/1
 Running            1          1d
dev-appc-appc-dgbuilder-f7565468-fnrz6                        1/1
 Running            0          1d

On Mon, Mar 4, 2019 at 7:13 PM CHO, TAKAMUNE <[email protected]> wrote:

> Hi,
>
>
>
> In Casablanca, Those ODL features were installed when you are installing
> APPC container. We moved ODL feature installation when APPC container gets
> built in R4, so it won’t take that long in R4.
>
>
>
> But 1802 seconds installing is just too long. Did you check your k8s env
> or VM capacity?
>
>
>
> Taka
>
>
>
> *From:* [email protected] <[email protected]> *On
> Behalf Of *Vivekanandan Muthukrishnan
> *Sent:* Sunday, March 3, 2019 6:23 AM
> *To:* [email protected]
> *Subject:* [onap-discuss] Casablanca APPC Readiness probe failed: APPC is
> not healthy for more than 2 hours
>
>
>
> Hi All,
>
>
>
> It seems like APPC POD dev-appc-appc-0 (Container aapc) is taking more
> time to install KARAF bundles. And dev-appc-appc-ansible-server is keep
> getting restarted.
>
>
>
> It this expected ? Are there any workaround to offload KARAF packages from
> local maven repository?
>
>
>
> # All APPC PODs
>
> $ kubectl get pods -n onap | grep appc
>
> dev-appc-appc-0                                               1/2
>  Running            0          2h
>
> dev-appc-appc-ansible-server-6877b497df-j544r                 0/1
>  Init:0/1           3          37m
>
> dev-appc-appc-cdt-77bccf4847-fmtpw                            1/1
>  Running            0          2h
>
> dev-appc-appc-db-0                                            1/1
>  Running            1          2h
>
> dev-appc-appc-db-1                                            1/1
>  Running            0          2h
>
> dev-appc-appc-db-2                                            1/1
>  Running            1          2h
>
> dev-appc-appc-dgbuilder-f7565468-fnrz6                        1/1
>  Running            0          2h
>
>
>
>
>
> # It seems like ODL KARAF features are still getting installed and it is
> taking more time
>
> *$ kubectl logs -n onap dev-appc-appc-0 -c appc*
>
> Adding feature url
> mvn:org.onap.appc/onap-appc-design-services/1.4.4/xml/features
>
> Archive:
> /opt/onap/appc/features/appc-interfaces-service/appc-interfaces-service-1.4.4.zip
>
>    creating: /opt/opendaylight/system/com/google/code/gson/gson/2.8.0/
>
>    creating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/
>
>    creating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/1.4.4/
>
>    creating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/
>
>    creating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/1.4.4/
>
>    creating:
> /opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/
>
>    creating:
> /opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/1.4.4/
>
>   inflating:
> /opt/opendaylight/system/com/google/code/gson/gson/2.8.0/_remote.repositories
>
>   inflating:
> /opt/opendaylight/system/com/google/code/gson/gson/2.8.0/gson-2.8.0.jar
>
>   inflating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/1.4.4/appc-interfaces-service-model-1.4.4.jar
>
>   inflating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/1.4.4/_remote.repositories
>
>   inflating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/maven-metadata-local.xml
>
>   inflating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/1.4.4/appc-interfaces-service-bundle-1.4.4.jar
>
>   inflating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/1.4.4/_remote.repositories
>
>   inflating:
> /opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/maven-metadata-local.xml
>
>   inflating:
> /opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/1.4.4/_remote.repositories
>
>   inflating:
> /opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/1.4.4/onap-appc-interfaces-service-1.4.4-features.xml
>
>   inflating:
> /opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/maven-metadata-local.xml
>
> Adding feature url
> mvn:org.onap.appc/onap-appc-interfaces-service/1.4.4/xml/features
>
> Installing onap-appc-core
>
> Install of onap-appc-core took 1802 seconds
>
> Sleep Finished
>
> Installing onap-appc-metric
>
>
>
> *$ kubectl describe pod -n onap dev-appc-appc-0*
>
> Name:           dev-appc-appc-0
>
> Namespace:      onap
>
> Node:           casablanca03/192.168.122.233
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.122.233&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=5QdUf3h3TDBxaeo_dw16EKiqx8IfBbGdt_NveymVWFc&e=>
>
> Start Time:     Sun, 03 Mar 2019 08:39:03 +0000
>
> Labels:         app=appc
>
>                 controller-revision-hash=dev-appc-appc-69d746947b
>
>                 release=dev-appc
>
>                 statefulset.kubernetes.io/pod-name=dev-appc-appc-0
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__statefulset.kubernetes.io_pod-2Dname-3Ddev-2Dappc-2Dappc-2D0&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=RxKrnUof5JnuS1iyR6UR_vS682wQdsNjgRy0LMrRu_s&e=>
>
> Annotations:    <none>
>
> Status:         Running
>
> IP:             10.42.135.188
>
> Controlled By:  StatefulSet/dev-appc-appc
>
> Init Containers:
>
>   appc-readiness:
>
>     Container ID:
> docker://219c491421ce1c5f83539e84508a642c0726930fe4078a72fbf0de65d17aa236
>
>     Image:         oomk8s/readiness-check:2.0.0
>
>     Image ID:
> docker://sha256:867cb038e1d2445a6e5aedc3b5f970dacc8249ab119d6c2e088e10df886ff51f
>
>     Port:          <none>
>
>     Host Port:     <none>
>
>     Command:
>
>       /root/ready.py
>
>     Args:
>
>       --container-name
>
>       appc-db
>
>     State:          Terminated
>
>       Reason:       Completed
>
>       Exit Code:    0
>
>       Started:      Sun, 03 Mar 2019 08:40:03 +0000
>
>       Finished:     Sun, 03 Mar 2019 08:45:25 +0000
>
>     Ready:          True
>
>     Restart Count:  0
>
>     Environment:
>
>       NAMESPACE:  onap (v1:metadata.namespace)
>
>     Mounts:
>
>       /var/run/secrets/kubernetes.io/serviceaccount
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=K_7RtZDFgrdJQ-k66gJ91ENalU4kW5c5vPLpdlFU4tY&e=>
> from default-token-lw9wt (ro)
>
> Containers:
>
>   appc:
>
>     Container ID:
> docker://f1c058a8fb540c20beaffcad0349191877527824a0b6e11d55150694c65d6427
>
>     Image:         nexus3.onap.org:10001/onap/appc-image:1.4.4
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_appc-2Dimage-3A1.4.4&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=L6GfGwiv2Emr4_HdXjZBEWlS1WNnZZ1VF2wD6zQe5Wc&e=>
>
>     Image ID:      docker-pullable://
> nexus3.onap.org:10001/onap/appc-image@sha256:a60c532af06550e0fe9847c5f9d7e30d1902980439497dc98b2b32068839d2b6
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_appc-2Dimage-40sha256-3Aa60c532af06550e0fe9847c5f9d7e30d1902980439497dc98b2b32068839d2b6&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=BvjPIa7liaAeP3XAggqQY4NvJZWOxNG26BYanJjbcwk&e=>
>
>     Ports:         8181/TCP, 1830/TCP
>
>     Host Ports:    0/TCP, 0/TCP
>
>     Command:
>
>       /opt/appc/bin/startODL.sh
>
>     State:          Running
>
>       Started:      Sun, 03 Mar 2019 08:48:29 +0000
>
>     Ready:          False
>
>     Restart Count:  0
>
>     Readiness:      exec [/opt/appc/bin/health_check.sh] delay=10s
> timeout=1s period=10s #success=1 #failure=3
>
>     Environment:
>
>       MYSQL_ROOT_PASSWORD:  <set to the key 'db-root-password' in secret
> 'dev-appc-appc'>  Optional: false
>
>       SDNC_CONFIG_DIR:      /opt/onap/appc/data/properties
>
>       APPC_CONFIG_DIR:      /opt/onap/appc/data/properties
>
>       DMAAP_TOPIC_ENV:      SUCCESS
>
>       ENABLE_AAF:           true
>
>       ENABLE_ODL_CLUSTER:   false
>
>       APPC_REPLICAS:        1
>
>     Mounts:
>
>       /etc/localtime from localtime (ro)
>
>       /opt/onap/appc/bin/health_check.sh from onap-appc-bin (rw)
>
>       /opt/onap/appc/bin/installAppcDb.sh from onap-appc-bin (rw)
>
>       /opt/onap/appc/bin/startODL.sh from onap-appc-bin (rw)
>
>       /opt/onap/appc/data/properties/aaa-app-config.xml from
> onap-appc-data-properties (rw)
>
>       /opt/onap/appc/data/properties/aaiclient.properties from
> onap-appc-data-properties (rw)
>
>       /opt/onap/appc/data/properties/appc.properties from
> onap-appc-data-properties (rw)
>
>       /opt/onap/appc/data/properties/cadi.properties from
> onap-appc-data-properties (rw)
>
>       /opt/onap/appc/data/properties/dblib.properties from
> onap-appc-data-properties (rw)
>
>       /opt/onap/appc/data/properties/svclogic.properties from
> onap-appc-data-properties (rw)
>
>       /opt/onap/appc/svclogic/bin/showActiveGraphs.sh from
> onap-appc-svclogic-bin (rw)
>
>       /opt/onap/appc/svclogic/config/svclogic.properties from
> onap-appc-svclogic-config (rw)
>
>       /opt/onap/ccsdk/bin/installSdncDb.sh from onap-sdnc-bin (rw)
>
>       /opt/onap/ccsdk/bin/startODL.sh from onap-sdnc-bin (rw)
>
>       /opt/onap/ccsdk/data/properties/aaiclient.properties from
> onap-sdnc-data-properties (rw)
>
>       /opt/onap/ccsdk/data/properties/dblib.properties from
> onap-sdnc-data-properties (rw)
>
>       /opt/onap/ccsdk/data/properties/svclogic.properties from
> onap-sdnc-data-properties (rw)
>
>       /opt/onap/ccsdk/svclogic/bin/showActiveGraphs.sh from
> onap-sdnc-svclogic-bin (rw)
>
>       /opt/onap/ccsdk/svclogic/config/svclogic.properties from
> onap-sdnc-svclogic-config (rw)
>
>       /opt/opendaylight/current/daexim from dev-appc-appc-data (rw)
>
>       /opt/opendaylight/current/etc/org.ops4j.pax.logging.cfg from
> log-config (rw)
>
>       /var/log/onap from logs (rw)
>
>       /var/run/secrets/kubernetes.io/serviceaccount
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=K_7RtZDFgrdJQ-k66gJ91ENalU4kW5c5vPLpdlFU4tY&e=>
> from default-token-lw9wt (ro)
>
>   filebeat-onap:
>
>     Container ID:
>  docker://daf7ddc4a3e4945a1a4cab940906022248696e72c90bb15fa01144cacd3a1833
>
>     Image:          docker.elastic.co/beats/filebeat:5.5.0
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__docker.elastic.co_beats_filebeat-3A5.5.0&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=8NTH89zzSFu9SMF9-3wjBtn0DoxJDqZsB9iv-3UTZdc&e=>
>
>     Image ID:
>  
> docker://sha256:b61327632415b6d374b9f34cea71cb14f9c352e5259140ce6e3c8eaf8becaa1b
>
>     Port:           <none>
>
>     Host Port:      <none>
>
>     State:          Running
>
>       Started:      Sun, 03 Mar 2019 08:48:30 +0000
>
>     Ready:          True
>
>     Restart Count:  0
>
>     Environment:    <none>
>
>     Mounts:
>
>       /usr/share/filebeat/data from data-filebeat (rw)
>
>       /usr/share/filebeat/filebeat.yml from filebeat-conf (rw)
>
>       /var/log/onap from logs (rw)
>
>       /var/run/secrets/kubernetes.io/serviceaccount
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=K_7RtZDFgrdJQ-k66gJ91ENalU4kW5c5vPLpdlFU4tY&e=>
> from default-token-lw9wt (ro)
>
> Conditions:
>
>   Type              Status
>
>   Initialized       True
>
>   Ready             False
>
>   ContainersReady   False
>
>   PodScheduled      True
>
> Volumes:
>
>   dev-appc-appc-data:
>
>     Type:       PersistentVolumeClaim (a reference to a
> PersistentVolumeClaim in the same namespace)
>
>     ClaimName:  dev-appc-appc-data-dev-appc-appc-0
>
>     ReadOnly:   false
>
>   localtime:
>
>     Type:          HostPath (bare host directory volume)
>
>     Path:          /etc/localtime
>
>     HostPathType:
>
>   filebeat-conf:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-filebeat
>
>     Optional:  false
>
>   log-config:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-logging-cfg
>
>     Optional:  false
>
>   logs:
>
>     Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
>
>     Medium:
>
>   data-filebeat:
>
>     Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
>
>     Medium:
>
>   onap-appc-data-properties:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-onap-appc-data-properties
>
>     Optional:  false
>
>   onap-appc-svclogic-config:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-onap-appc-svclogic-config
>
>     Optional:  false
>
>   onap-appc-svclogic-bin:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-onap-appc-svclogic-bin
>
>     Optional:  false
>
>   onap-appc-bin:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-onap-appc-bin
>
>     Optional:  false
>
>   onap-sdnc-data-properties:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-onap-sdnc-data-properties
>
>     Optional:  false
>
>   onap-sdnc-svclogic-config:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-onap-sdnc-svclogic-config
>
>     Optional:  false
>
>   onap-sdnc-svclogic-bin:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-onap-sdnc-svclogic-bin
>
>     Optional:  false
>
>   onap-sdnc-bin:
>
>     Type:      ConfigMap (a volume populated by a ConfigMap)
>
>     Name:      dev-appc-appc-onap-sdnc-bin
>
>     Optional:  false
>
>   default-token-lw9wt:
>
>     Type:        Secret (a volume populated by a Secret)
>
>     SecretName:  default-token-lw9wt
>
>     Optional:    false
>
> QoS Class:       BestEffort
>
> Node-Selectors:  <none>
>
> Tolerations:     node.kubernetes.io/not-ready:NoExecute
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_not-2Dready-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=r99_KtHvCDVNp-zIMMlkm8mrd6LpRRJbJEB9yR9yXLY&e=>
> for 300s
>
>                  node.kubernetes.io/unreachable:NoExecute
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_unreachable-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=i5VHNTZ3SDPgIii87sudZA&m=PeuJ6XKmM0ul8_BO9rGOx8ZhqqQzlnaReJYk6TIJ6gE&s=ByCvoW97wNZ5FWPnIYUkhGPdq3rSabaE9J4sGAQr1nw&e=>
> for 300s
>
> Events:
>
>   Type     Reason     Age               From                   Message
>
>   ----     ------     ----              ----                   -------
>
>   Warning  Unhealthy  26m (x9 over 1h)  kubelet, casablanca03  Readiness
> probe failed: APPC is not healthy.
>
> ++ ps -e
>
> ++ grep startODL
>
> ++ wc -l
>
> + startODL_status=1
>
> ++ grep Waiting
>
> ++ wc -l
>
> ++ /opt/opendaylight/current/bin/client bundle:list
>
> + waiting_bundles=0
>
> ++ /opt/opendaylight/current/bin/client system:start-level
>
> + run_level='Level 100'
>
> + '[' 'Level 100' == 'Level 100' ']'
>
> + '[' 1 -lt 1 ']'
>
> + echo APPC is not healthy.
>
> + exit 1
>
>   Warning  Unhealthy  20m (x212 over 1h)  kubelet, casablanca03  (combined
> from similar events): Readiness probe failed: APPC is not healthy.
>
> ++ ps -e
>
> ++ grep startODL
>
> ++ wc -l
>
> + startODL_status=1
>
> ++ grep Waiting
>
> ++ /opt/opendaylight/current/bin/client bundle:list
>
> ++ wc -l
>
> + waiting_bundles=0
>
> ++ /opt/opendaylight/current/bin/client system:start-level
>
> + run_level='Level 100'
>
> + '[' 'Level 100' == 'Level 100' ']'
>
> + '[' 1 -lt 1 ']'
>
> + echo APPC is not healthy.
>
> + exit 1
>
>   Warning  Unhealthy  16m  kubelet, casablanca03  Readiness probe failed:
> APPC is not healthy.
>
> ++ wc -l
>
> ++ ps -e
>
> ++ grep startODL
>
> + startODL_status=1
>
> ++ /opt/opendaylight/current/bin/client bundle:list
>
> ++ grep Waiting
>
> ++ wc -l
>
> + waiting_bundles=0
>
> ++ /opt/opendaylight/current/bin/client system:start-level
>
> + run_level='Level 100'
>
> + '[' 'Level 100' == 'Level 100' ']'
>
> + '[' 1 -lt 1 ']'
>
> + echo APPC is not healthy.
>
> + exit 1
>
>   Warning  Unhealthy  11m (x203 over 1h)  kubelet, casablanca03  Readiness
> probe failed: APPC is not healthy.
>
> ++ ps -e
>
> ++ grep startODL
>
> ++ wc -l
>
> + startODL_status=1
>
> ++ /opt/opendaylight/current/bin/client bundle:list
>
> ++ grep Waiting
>
> ++ wc -l
>
> + waiting_bundles=0
>
> ++ /opt/opendaylight/current/bin/client system:start-level
>
> + run_level='Level 100'
>
> + '[' 'Level 100' == 'Level 100' ']'
>
> + '[' 1 -lt 1 ']'
>
> + echo APPC is not healthy.
>
> + exit 1
>
>   Warning  Unhealthy  5m (x12 over 1h)  kubelet, casablanca03  Readiness
> probe failed: APPC is not healthy.
>
> ++ ps -e
>
> ++ wc -l
>
> ++ grep startODL
>
> + startODL_status=1
>
> ++ /opt/opendaylight/current/bin/client bundle:list
>
> ++ wc -l
>
> ++ grep Waiting
>
> + waiting_bundles=0
>
> ++ /opt/opendaylight/current/bin/client system:start-level
>
> + run_level='Level 100'
>
> + '[' 'Level 100' == 'Level 100' ']'
>
> + '[' 1 -lt 1 ']'
>
> + echo APPC is not healthy.
>
> + exit 1
>
>   Warning  Unhealthy  1m (x31 over 1h)  kubelet, casablanca03  Readiness
> probe failed: APPC is not healthy.
>
> ++ ps -e
>
> ++ wc -l
>
> ++ grep startODL
>
> + startODL_status=1
>
> ++ /opt/opendaylight/current/bin/client bundle:list
>
> ++ grep Waiting
>
> ++ wc -l
>
> + waiting_bundles=0
>
> ++ /opt/opendaylight/current/bin/client system:start-level
>
> + run_level='Level 100'
>
> + '[' 'Level 100' == 'Level 100' ']'
>
> + '[' 1 -lt 1 ']'
>
> + echo APPC is not healthy.
>
> + exit 1
>
>
>
>
>
> === We can access the karaf
>
>
>
> *$ kubectl exec -it -n onap dev-appc-appc-0 -c appc -- /bin/bash*
>
> root@dev-appc-appc-0:/# ls
>
> bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
>
> boot  etc  lib   media  opt  root  sbin  sys  usr
>
> root@dev-appc-appc-0:/# ps -ef | grep java
>
> root       217   156 15 08:48 ?        00:21:44
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java
> -Djava.security.properties=/opt/opendaylight/etc/odl.java.security -Xms128M
> -Xmx2048m -XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemoryError
> -Dcom.sun.management.jmxremote -Djava.security.egd=file:/dev/./urandom
> -Djava.endorsed.dirs=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/endorsed:/usr/lib/jvm/java-8-openjdk-amd64/lib/endorsed:/opt/opendaylight/lib/endorsed
> -Djava.ext.dirs=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/lib/jvm/java-8-openjdk-amd64/lib/ext:/opt/opendaylight/lib/ext
> -Dkaraf.instances=/opt/opendaylight/instances
> -Dkaraf.home=/opt/opendaylight -Dkaraf.base=/opt/opendaylight
> -Dkaraf.data=/opt/opendaylight/data -Dkaraf.etc=/opt/opendaylight/etc
> -Dkaraf.restart.jvm.supported=true
> -Djava.io.tmpdir=/opt/opendaylight/data/tmp
> -Djava.util.logging.config.file=/opt/opendaylight/etc/java.util.logging.properties
> -Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -classpath
> /opt/opendaylight/lib/boot/org.apache.karaf.diagnostic.boot-4.1.5.jar:/opt/opendaylight/lib/boot/org.apache.karaf.jaas.boot-4.1.5.jar:/opt/opendaylight/lib/boot/org.apache.karaf.main-4.1.5.jar:/opt/opendaylight/lib/boot/org.osgi.core-6.0.0.jar
> org.apache.karaf.main.Main
>
> root     10908  1660  0 10:53 ?        00:00:04
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java
> -Dkaraf.instances=/opt/opendaylight/instances
> -Dkaraf.home=/opt/opendaylight -Dkaraf.base=/opt/opendaylight
> -Dkaraf.etc=/opt/opendaylight/etc
> -Djava.io.tmpdir=/opt/opendaylight/data/tmp
> -Djava.util.logging.config.file=/opt/opendaylight/etc/java.util.logging.properties
> -classpath
> /opt/opendaylight/system/org/apache/karaf/org.apache.karaf.client/4.1.5/org.apache.karaf.client-4.1.5.jar:/opt/opendaylight/system/org/apache/sshd/sshd-core/1.6.0/sshd-core-1.6.0.jar:/opt/opendaylight/system/org/fusesource/jansi/jansi/1.17/jansi-1.17.jar:/opt/opendaylight/system/org/jline/jline/3.6.0/jline-3.6.0.jar:/opt/opendaylight/system/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar
> org.apache.karaf.client.Main feature:install -r onap-appc-metric
>
> root     25586 25509  0 11:10 ?        00:00:00
> /usr/lib/jvm/java-8-openjdk-amd64/bin/java
> -Dkaraf.instances=/opt/opendaylight/current/instances
> -Dkaraf.home=/opt/opendaylight/current
> -Dkaraf.base=/opt/opendaylight/current
> -Dkaraf.etc=/opt/opendaylight/current/etc
> -Djava.io.tmpdir=/opt/opendaylight/current/data/tmp
> -Djava.util.logging.config.file=/opt/opendaylight/current/etc/java.util.logging.properties
> -classpath
> /opt/opendaylight/current/system/org/apache/karaf/org.apache.karaf.client/4.1.5/org.apache.karaf.client-4.1.5.jar:/opt/opendaylight/current/system/org/apache/sshd/sshd-core/1.6.0/sshd-core-1.6.0.jar:/opt/opendaylight/current/system/org/fusesource/jansi/jansi/1.17/jansi-1.17.jar:/opt/opendaylight/current/system/org/jline/jline/3.6.0/jline-3.6.0.jar:/opt/opendaylight/current/system/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar
> org.apache.karaf.client.Main system:start-level
>
> root     25637 25428  0 11:10 ?        00:00:00 grep --color=auto java
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> 
>

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#15857): https://lists.onap.org/g/onap-discuss/message/15857
Mute This Topic: https://lists.onap.org/mt/30194640/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub  
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Reply via email to