Hi All,
It seems like APPC POD dev-appc-appc-0 (Container aapc) is taking more time
to install KARAF bundles. And dev-appc-appc-ansible-server is keep getting
restarted.
It this expected ? Are there any workaround to offload KARAF packages from
local maven repository?
# All APPC PODs
$ kubectl get pods -n onap | grep appc
dev-appc-appc-0 1/2
Running 0 2h
dev-appc-appc-ansible-server-6877b497df-j544r 0/1
Init:0/1 3 37m
dev-appc-appc-cdt-77bccf4847-fmtpw 1/1
Running 0 2h
dev-appc-appc-db-0 1/1
Running 1 2h
dev-appc-appc-db-1 1/1
Running 0 2h
dev-appc-appc-db-2 1/1
Running 1 2h
dev-appc-appc-dgbuilder-f7565468-fnrz6 1/1
Running 0 2h
# It seems like ODL KARAF features are still getting installed and it is
taking more time
*$ kubectl logs -n onap dev-appc-appc-0 -c appc*
Adding feature url
mvn:org.onap.appc/onap-appc-design-services/1.4.4/xml/features
Archive:
/opt/onap/appc/features/appc-interfaces-service/appc-interfaces-service-1.4.4.zip
creating: /opt/opendaylight/system/com/google/code/gson/gson/2.8.0/
creating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/
creating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/1.4.4/
creating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/
creating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/1.4.4/
creating:
/opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/
creating:
/opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/1.4.4/
inflating:
/opt/opendaylight/system/com/google/code/gson/gson/2.8.0/_remote.repositories
inflating:
/opt/opendaylight/system/com/google/code/gson/gson/2.8.0/gson-2.8.0.jar
inflating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/1.4.4/appc-interfaces-service-model-1.4.4.jar
inflating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/1.4.4/_remote.repositories
inflating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-model/maven-metadata-local.xml
inflating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/1.4.4/appc-interfaces-service-bundle-1.4.4.jar
inflating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/1.4.4/_remote.repositories
inflating:
/opt/opendaylight/system/org/onap/appc/appc-interfaces-service-bundle/maven-metadata-local.xml
inflating:
/opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/1.4.4/_remote.repositories
inflating:
/opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/1.4.4/onap-appc-interfaces-service-1.4.4-features.xml
inflating:
/opt/opendaylight/system/org/onap/appc/onap-appc-interfaces-service/maven-metadata-local.xml
Adding feature url
mvn:org.onap.appc/onap-appc-interfaces-service/1.4.4/xml/features
Installing onap-appc-core
Install of onap-appc-core took 1802 seconds
Sleep Finished
Installing onap-appc-metric
*$ kubectl describe pod -n onap dev-appc-appc-0*
Name: dev-appc-appc-0
Namespace: onap
Node: casablanca03/192.168.122.233
Start Time: Sun, 03 Mar 2019 08:39:03 +0000
Labels: app=appc
controller-revision-hash=dev-appc-appc-69d746947b
release=dev-appc
statefulset.kubernetes.io/pod-name=dev-appc-appc-0
Annotations: <none>
Status: Running
IP: 10.42.135.188
Controlled By: StatefulSet/dev-appc-appc
Init Containers:
appc-readiness:
Container ID:
docker://219c491421ce1c5f83539e84508a642c0726930fe4078a72fbf0de65d17aa236
Image: oomk8s/readiness-check:2.0.0
Image ID:
docker://sha256:867cb038e1d2445a6e5aedc3b5f970dacc8249ab119d6c2e088e10df886ff51f
Port: <none>
Host Port: <none>
Command:
/root/ready.py
Args:
--container-name
appc-db
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 03 Mar 2019 08:40:03 +0000
Finished: Sun, 03 Mar 2019 08:45:25 +0000
Ready: True
Restart Count: 0
Environment:
NAMESPACE: onap (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from
default-token-lw9wt (ro)
Containers:
appc:
Container ID:
docker://f1c058a8fb540c20beaffcad0349191877527824a0b6e11d55150694c65d6427
Image: nexus3.onap.org:10001/onap/appc-image:1.4.4
Image ID: docker-pullable://
nexus3.onap.org:10001/onap/appc-image@sha256:a60c532af06550e0fe9847c5f9d7e30d1902980439497dc98b2b32068839d2b6
Ports: 8181/TCP, 1830/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/opt/appc/bin/startODL.sh
State: Running
Started: Sun, 03 Mar 2019 08:48:29 +0000
Ready: False
Restart Count: 0
Readiness: exec [/opt/appc/bin/health_check.sh] delay=10s
timeout=1s period=10s #success=1 #failure=3
Environment:
MYSQL_ROOT_PASSWORD: <set to the key 'db-root-password' in secret
'dev-appc-appc'> Optional: false
SDNC_CONFIG_DIR: /opt/onap/appc/data/properties
APPC_CONFIG_DIR: /opt/onap/appc/data/properties
DMAAP_TOPIC_ENV: SUCCESS
ENABLE_AAF: true
ENABLE_ODL_CLUSTER: false
APPC_REPLICAS: 1
Mounts:
/etc/localtime from localtime (ro)
/opt/onap/appc/bin/health_check.sh from onap-appc-bin (rw)
/opt/onap/appc/bin/installAppcDb.sh from onap-appc-bin (rw)
/opt/onap/appc/bin/startODL.sh from onap-appc-bin (rw)
/opt/onap/appc/data/properties/aaa-app-config.xml from
onap-appc-data-properties (rw)
/opt/onap/appc/data/properties/aaiclient.properties from
onap-appc-data-properties (rw)
/opt/onap/appc/data/properties/appc.properties from
onap-appc-data-properties (rw)
/opt/onap/appc/data/properties/cadi.properties from
onap-appc-data-properties (rw)
/opt/onap/appc/data/properties/dblib.properties from
onap-appc-data-properties (rw)
/opt/onap/appc/data/properties/svclogic.properties from
onap-appc-data-properties (rw)
/opt/onap/appc/svclogic/bin/showActiveGraphs.sh from
onap-appc-svclogic-bin (rw)
/opt/onap/appc/svclogic/config/svclogic.properties from
onap-appc-svclogic-config (rw)
/opt/onap/ccsdk/bin/installSdncDb.sh from onap-sdnc-bin (rw)
/opt/onap/ccsdk/bin/startODL.sh from onap-sdnc-bin (rw)
/opt/onap/ccsdk/data/properties/aaiclient.properties from
onap-sdnc-data-properties (rw)
/opt/onap/ccsdk/data/properties/dblib.properties from
onap-sdnc-data-properties (rw)
/opt/onap/ccsdk/data/properties/svclogic.properties from
onap-sdnc-data-properties (rw)
/opt/onap/ccsdk/svclogic/bin/showActiveGraphs.sh from
onap-sdnc-svclogic-bin (rw)
/opt/onap/ccsdk/svclogic/config/svclogic.properties from
onap-sdnc-svclogic-config (rw)
/opt/opendaylight/current/daexim from dev-appc-appc-data (rw)
/opt/opendaylight/current/etc/org.ops4j.pax.logging.cfg from
log-config (rw)
/var/log/onap from logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from
default-token-lw9wt (ro)
filebeat-onap:
Container ID:
docker://daf7ddc4a3e4945a1a4cab940906022248696e72c90bb15fa01144cacd3a1833
Image: docker.elastic.co/beats/filebeat:5.5.0
Image ID:
docker://sha256:b61327632415b6d374b9f34cea71cb14f9c352e5259140ce6e3c8eaf8becaa1b
Port: <none>
Host Port: <none>
State: Running
Started: Sun, 03 Mar 2019 08:48:30 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/usr/share/filebeat/data from data-filebeat (rw)
/usr/share/filebeat/filebeat.yml from filebeat-conf (rw)
/var/log/onap from logs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from
default-token-lw9wt (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
dev-appc-appc-data:
Type: PersistentVolumeClaim (a reference to a
PersistentVolumeClaim in the same namespace)
ClaimName: dev-appc-appc-data-dev-appc-appc-0
ReadOnly: false
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
HostPathType:
filebeat-conf:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-filebeat
Optional: false
log-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-logging-cfg
Optional: false
logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
data-filebeat:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
onap-appc-data-properties:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-onap-appc-data-properties
Optional: false
onap-appc-svclogic-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-onap-appc-svclogic-config
Optional: false
onap-appc-svclogic-bin:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-onap-appc-svclogic-bin
Optional: false
onap-appc-bin:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-onap-appc-bin
Optional: false
onap-sdnc-data-properties:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-onap-sdnc-data-properties
Optional: false
onap-sdnc-svclogic-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-onap-sdnc-svclogic-config
Optional: false
onap-sdnc-svclogic-bin:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-onap-sdnc-svclogic-bin
Optional: false
onap-sdnc-bin:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: dev-appc-appc-onap-sdnc-bin
Optional: false
default-token-lw9wt:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-lw9wt
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 26m (x9 over 1h) kubelet, casablanca03 Readiness
probe failed: APPC is not healthy.
++ ps -e
++ grep startODL
++ wc -l
+ startODL_status=1
++ grep Waiting
++ wc -l
++ /opt/opendaylight/current/bin/client bundle:list
+ waiting_bundles=0
++ /opt/opendaylight/current/bin/client system:start-level
+ run_level='Level 100'
+ '[' 'Level 100' == 'Level 100' ']'
+ '[' 1 -lt 1 ']'
+ echo APPC is not healthy.
+ exit 1
Warning Unhealthy 20m (x212 over 1h) kubelet, casablanca03 (combined
from similar events): Readiness probe failed: APPC is not healthy.
++ ps -e
++ grep startODL
++ wc -l
+ startODL_status=1
++ grep Waiting
++ /opt/opendaylight/current/bin/client bundle:list
++ wc -l
+ waiting_bundles=0
++ /opt/opendaylight/current/bin/client system:start-level
+ run_level='Level 100'
+ '[' 'Level 100' == 'Level 100' ']'
+ '[' 1 -lt 1 ']'
+ echo APPC is not healthy.
+ exit 1
Warning Unhealthy 16m kubelet, casablanca03 Readiness probe failed:
APPC is not healthy.
++ wc -l
++ ps -e
++ grep startODL
+ startODL_status=1
++ /opt/opendaylight/current/bin/client bundle:list
++ grep Waiting
++ wc -l
+ waiting_bundles=0
++ /opt/opendaylight/current/bin/client system:start-level
+ run_level='Level 100'
+ '[' 'Level 100' == 'Level 100' ']'
+ '[' 1 -lt 1 ']'
+ echo APPC is not healthy.
+ exit 1
Warning Unhealthy 11m (x203 over 1h) kubelet, casablanca03 Readiness
probe failed: APPC is not healthy.
++ ps -e
++ grep startODL
++ wc -l
+ startODL_status=1
++ /opt/opendaylight/current/bin/client bundle:list
++ grep Waiting
++ wc -l
+ waiting_bundles=0
++ /opt/opendaylight/current/bin/client system:start-level
+ run_level='Level 100'
+ '[' 'Level 100' == 'Level 100' ']'
+ '[' 1 -lt 1 ']'
+ echo APPC is not healthy.
+ exit 1
Warning Unhealthy 5m (x12 over 1h) kubelet, casablanca03 Readiness
probe failed: APPC is not healthy.
++ ps -e
++ wc -l
++ grep startODL
+ startODL_status=1
++ /opt/opendaylight/current/bin/client bundle:list
++ wc -l
++ grep Waiting
+ waiting_bundles=0
++ /opt/opendaylight/current/bin/client system:start-level
+ run_level='Level 100'
+ '[' 'Level 100' == 'Level 100' ']'
+ '[' 1 -lt 1 ']'
+ echo APPC is not healthy.
+ exit 1
Warning Unhealthy 1m (x31 over 1h) kubelet, casablanca03 Readiness
probe failed: APPC is not healthy.
++ ps -e
++ wc -l
++ grep startODL
+ startODL_status=1
++ /opt/opendaylight/current/bin/client bundle:list
++ grep Waiting
++ wc -l
+ waiting_bundles=0
++ /opt/opendaylight/current/bin/client system:start-level
+ run_level='Level 100'
+ '[' 'Level 100' == 'Level 100' ']'
+ '[' 1 -lt 1 ']'
+ echo APPC is not healthy.
+ exit 1
=== We can access the karaf
*$ kubectl exec -it -n onap dev-appc-appc-0 -c appc -- /bin/bash*
root@dev-appc-appc-0:/# ls
bin dev home lib64 mnt proc run srv tmp var
boot etc lib media opt root sbin sys usr
root@dev-appc-appc-0:/# ps -ef | grep java
root 217 156 15 08:48 ? 00:21:44
/usr/lib/jvm/java-8-openjdk-amd64/bin/java
-Djava.security.properties=/opt/opendaylight/etc/odl.java.security -Xms128M
-Xmx2048m -XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemoryError
-Dcom.sun.management.jmxremote -Djava.security.egd=file:/dev/./urandom
-Djava.endorsed.dirs=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/endorsed:/usr/lib/jvm/java-8-openjdk-amd64/lib/endorsed:/opt/opendaylight/lib/endorsed
-Djava.ext.dirs=/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/lib/jvm/java-8-openjdk-amd64/lib/ext:/opt/opendaylight/lib/ext
-Dkaraf.instances=/opt/opendaylight/instances
-Dkaraf.home=/opt/opendaylight -Dkaraf.base=/opt/opendaylight
-Dkaraf.data=/opt/opendaylight/data -Dkaraf.etc=/opt/opendaylight/etc
-Dkaraf.restart.jvm.supported=true
-Djava.io.tmpdir=/opt/opendaylight/data/tmp
-Djava.util.logging.config.file=/opt/opendaylight/etc/java.util.logging.properties
-Dkaraf.startLocalConsole=false -Dkaraf.startRemoteShell=true -classpath
/opt/opendaylight/lib/boot/org.apache.karaf.diagnostic.boot-4.1.5.jar:/opt/opendaylight/lib/boot/org.apache.karaf.jaas.boot-4.1.5.jar:/opt/opendaylight/lib/boot/org.apache.karaf.main-4.1.5.jar:/opt/opendaylight/lib/boot/org.osgi.core-6.0.0.jar
org.apache.karaf.main.Main
root 10908 1660 0 10:53 ? 00:00:04
/usr/lib/jvm/java-8-openjdk-amd64/bin/java
-Dkaraf.instances=/opt/opendaylight/instances
-Dkaraf.home=/opt/opendaylight -Dkaraf.base=/opt/opendaylight
-Dkaraf.etc=/opt/opendaylight/etc
-Djava.io.tmpdir=/opt/opendaylight/data/tmp
-Djava.util.logging.config.file=/opt/opendaylight/etc/java.util.logging.properties
-classpath
/opt/opendaylight/system/org/apache/karaf/org.apache.karaf.client/4.1.5/org.apache.karaf.client-4.1.5.jar:/opt/opendaylight/system/org/apache/sshd/sshd-core/1.6.0/sshd-core-1.6.0.jar:/opt/opendaylight/system/org/fusesource/jansi/jansi/1.17/jansi-1.17.jar:/opt/opendaylight/system/org/jline/jline/3.6.0/jline-3.6.0.jar:/opt/opendaylight/system/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar
org.apache.karaf.client.Main feature:install -r onap-appc-metric
root 25586 25509 0 11:10 ? 00:00:00
/usr/lib/jvm/java-8-openjdk-amd64/bin/java
-Dkaraf.instances=/opt/opendaylight/current/instances
-Dkaraf.home=/opt/opendaylight/current
-Dkaraf.base=/opt/opendaylight/current
-Dkaraf.etc=/opt/opendaylight/current/etc
-Djava.io.tmpdir=/opt/opendaylight/current/data/tmp
-Djava.util.logging.config.file=/opt/opendaylight/current/etc/java.util.logging.properties
-classpath
/opt/opendaylight/current/system/org/apache/karaf/org.apache.karaf.client/4.1.5/org.apache.karaf.client-4.1.5.jar:/opt/opendaylight/current/system/org/apache/sshd/sshd-core/1.6.0/sshd-core-1.6.0.jar:/opt/opendaylight/current/system/org/fusesource/jansi/jansi/1.17/jansi-1.17.jar:/opt/opendaylight/current/system/org/jline/jline/3.6.0/jline-3.6.0.jar:/opt/opendaylight/current/system/org/slf4j/slf4j-api/1.7.12/slf4j-api-1.7.12.jar
org.apache.karaf.client.Main system:start-level
root 25637 25428 0 11:10 ? 00:00:00 grep --color=auto java
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15835): https://lists.onap.org/g/onap-discuss/message/15835
Mute This Topic: https://lists.onap.org/mt/30194640/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
