Hi Dat,
Like we discuss on slack:
>
> The Init:Error is caused by the readiness check container that waiting for
> some pods/resource to be available.The sdc and aai waiting for Cassandra
> pod. Like I saw in your log the Cassandra pod is in a pending state. So
> this is a potential problem. Please check the Cassandra logs and debug
> why Cassandra is in the pending state.
You also add some logs from the Cassandra pod deployments:
>
> Sorry for I can not response more quickly. I deploy 3 worker nodes with
> 100G HDD each node. I checked cassandra pod log:
> [root@onap-infra-node ~]# kubectl describe pod -n onap onap-cassandra-0
> Name: onap-cassandra-0
> Namespace: onap
> Priority: 0
> Node: <none>
> Labels: app.kubernetes.io/instance=onap
> app.kubernetes.io/managed-by=Tiller
> app.kubernetes.io/name=cassandra
> controller-revision-hash=onap-cassandra-c869f6769
> helm.sh/chart=cassandra-7.0.0
> statefulset.kubernetes.io/pod-name=onap-cassandra-0
> Annotations: <none>
> Status: Pending
> IP:
> Controlled By: StatefulSet/onap-cassandra
> Containers:
> cassandra:
> Image: nexus3.onap.org:10001/library/cassandra:3.11.4
> Ports: 7000/TCP, 7001/TCP, 7199/TCP, 9042/TCP, 9160/TCP, 61621/TCP
> Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
> Limits:
> cpu: 800m
> memory: 4Gi
> Requests:
> cpu: 200m
> memory: 2560Mi
> Liveness: exec [/bin/bash -c nodetool status | grep $POD_IP | awk
> '$1!="UN" { exit 1; }'] delay=60s timeout=3s period=10s #success=1
> #failure=3
> Readiness: exec [/bin/bash -c nodetool status | grep $POD_IP | awk
> '$1!="UN" { exit 1; }'] delay=120s timeout=3s period=10s #success=1
> #failure=3
> Environment:
> CASSANDRA_SEEDS:
> onap-cassandra-0.cassandra.onap.svc.cluster.local,onap-cassandra-1.cassandra.onap.svc.cluster.local,onap-cassandra-2.cassandra.onap.svc.cluster.local
>
> MAX_HEAP_SIZE: 2048M
> HEAP_NEWSIZE: 100M
> JVM_OPTS: -Dcassandra.consistent.rangemovement=false
> CASSANDRA_CLUSTER_NAME: cassandra
> CASSANDRA_DC: Pod
> CASSANDRA_RACK: Rack
> CASSANDRA_AUTO_BOOTSTRAP: true
> CASSANDRA_START_RPC: true
> CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
> CASSANDRA_AUTHENTICATOR: PasswordAuthenticator
> POD_IP: (v1:status.podIP)
> Mounts:
> /docker-entrypoint.sh from cassandra-entrypoint
> (rw,path="docker-entrypoint.sh")
> /etc/localtime from localtime (ro)
> /var/lib/cassandra from onap-cassandra-data (rw)
> /var/run/secrets/kubernetes.io/serviceaccount from default-token-vmnz8
> (ro)
> Conditions:
> Type Status
> PodScheduled False
> Volumes:
> onap-cassandra-data:
> Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim
> in the same namespace)
> ClaimName: onap-cassandra-data-onap-cassandra-0
> ReadOnly: false
> localtime:
> Type: HostPath (bare host directory volume)
> Path: /etc/localtime
> HostPathType:
> cassandra-entrypoint:
> Type: ConfigMap (a volume populated by a ConfigMap)
> Name: onap-cassandra-entrypoint
> Optional: false
> default-token-vmnz8:
> Type: Secret (a volume populated by a Secret)
> SecretName: default-token-vmnz8
> Optional: false
> QoS Class: Burstable
> Node-Selectors: <none>
> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
> node.kubernetes.io/unreachable:NoExecute for 300s
> Events:
> Type Reason Age From
> Message
> ---- ------ ---- ----
> -------
> Warning FailedScheduling 118s (x462 over 11h) default-scheduler pod
> has unbound immediate PersistentVolumeClaims (repeated 3 times)
> [root@onap-infra-node ~]# kubectl describe pod -n onap onap-cassandra-0
> Name: onap-cassandra-0
> Namespace: onap
> Priority: 0
> Node: <none>
> Labels: app.kubernetes.io/instance=onap
> app.kubernetes.io/managed-by=Tiller
> app.kubernetes.io/name=cassandra
> controller-revision-hash=onap-cassandra-c869f6769
> helm.sh/chart=cassandra-7.0.0
> statefulset.kubernetes.io/pod-name=onap-cassandra-0
> Annotations: <none>
> Status: Pending
> IP:
> Controlled By: StatefulSet/onap-cassandra
> Containers:
> cassandra:
> Image: nexus3.onap.org:10001/library/cassandra:3.11.4
> Ports: 7000/TCP, 7001/TCP, 7199/TCP, 9042/TCP, 9160/TCP, 61621/TCP
> Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
> Limits:
> cpu: 800m
> memory: 4Gi
> Requests:
> cpu: 200m
> memory: 2560Mi
> Liveness: exec [/bin/bash -c nodetool status | grep $POD_IP | awk
> '$1!="UN" { exit 1; }'] delay=60s timeout=3s period=10s #success=1
> #failure=3
> Readiness: exec [/bin/bash -c nodetool status | grep $POD_IP | awk
> '$1!="UN" { exit 1; }'] delay=120s timeout=3s period=10s #success=1
> #failure=3
> Environment:
> CASSANDRA_SEEDS:
> onap-cassandra-0.cassandra.onap.svc.cluster.local,onap-cassandra-1.cassandra.onap.svc.cluster.local,onap-cassandra-2.cassandra.onap.svc.cluster.local
>
> MAX_HEAP_SIZE: 2048M
> HEAP_NEWSIZE: 100M
> JVM_OPTS: -Dcassandra.consistent.rangemovement=false
> CASSANDRA_CLUSTER_NAME: cassandra
> CASSANDRA_DC: Pod
> CASSANDRA_RACK: Rack
> CASSANDRA_AUTO_BOOTSTRAP: true
> CASSANDRA_START_RPC: true
> CASSANDRA_ENDPOINT_SNITCH: GossipingPropertyFileSnitch
> CASSANDRA_AUTHENTICATOR: PasswordAuthenticator
> POD_IP: (v1:status.podIP)
> Mounts:
> /docker-entrypoint.sh from cassandra-entrypoint
> (rw,path="docker-entrypoint.sh")
> /etc/localtime from localtime (ro)
> /var/lib/cassandra from onap-cassandra-data (rw)
> /var/run/secrets/kubernetes.io/serviceaccount from default-token-vmnz8
> (ro)
> Conditions:
> Type Status
> PodScheduled False
> Volumes:
> onap-cassandra-data:
> Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim
> in the same namespace)
> ClaimName: onap-cassandra-data-onap-cassandra-0
> ReadOnly: false
> localtime:
> Type: HostPath (bare host directory volume)
> Path: /etc/localtime
> HostPathType:
> cassandra-entrypoint:
> Type: ConfigMap (a volume populated by a ConfigMap)
> Name: onap-cassandra-entrypoint
> Optional: false
> default-token-vmnz8:
> Type: Secret (a volume populated by a Secret)
> SecretName: default-token-vmnz8
> Optional: false
> QoS Class: Burstable
> Node-Selectors: <none>
> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
> node.kubernetes.io/unreachable:NoExecute for 300s
> Events:
> Type Reason Age From
> Message
> ---- ------ ---- ----
> -------
> Warning FailedScheduling 118s (x462 over 11h) default-scheduler pod
> has unbound immediate PersistentVolumeClaims (repeated 3 times)
> Cassandra pod has not deployed on any worker nodes
I add this information also to this mail thread because this can help us to
solve this Cassandra issue and the Cassandra issue can be connected with
Init:Error.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22926): https://lists.onap.org/g/onap-discuss/message/22926
Mute This Topic: https://lists.onap.org/mt/80998131/21656
Mute #deployment:https://lists.onap.org/g/onap-discuss/mutehashtag/deployment
Mute #deploy:https://lists.onap.org/g/onap-discuss/mutehashtag/deploy
Mute #guilin:https://lists.onap.org/g/onap-discuss/mutehashtag/guilin
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
