Hi Team,
I am performing a POC on ONAP using OOM.
But my portal-cassandra pod is going into crashloopbackoff again and again.
Please find the events from description of the pod.
Name: development-portal-cassandra-75b756869f-8hzdf
Namespace: onap
Priority: 0
Node: onap/10.0.1.167
Start Time: Fri, 16 Sep 2022 19:35:57 +0000
Labels: app=portal-cassandra
pod-template-hash=75b756869f
release=development
Annotations: cni.projectcalico.org/containerID:
57e7cfde1cc9d09266eb9a0d589c61feb5e64e677e3772f459a4995f37e521b0
cni.projectcalico.org/podIP: 192.168.229.36/32
cni.projectcalico.org/podIPs: 192.168.229.36/32
Status: Running
IP: 192.168.229.36
IPs:
IP: 192.168.229.36
Controlled By: ReplicaSet/development-portal-cassandra-75b756869f
Containers:
portal-cassandra:
Container ID:
docker://020e7d7c8076629143ccb6e0d8022bc550fba55a2bc2bf7ef49b0ec148560f55
Image: docker.io/bitnami/cassandra:3.11.9-debian-10-r30
Image ID:
docker-pullable://bitnami/cassandra@sha256:b0b79f004175851b7530a8d9330a5229699f4acebbf83b0d77894acd818e8caf
Ports: 9160/TCP, 7000/TCP, 7001/TCP, 7199/TCP, 9042/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
/opt/bitnami/scripts/cassandra/entrypoint.sh
/opt/bitnami/scripts/cassandra/run.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Fri, 16 Sep 2022 20:08:58 +0000
Finished: Fri, 16 Sep 2022 20:10:34 +0000
Ready: False
Restart Count: 11
Limits:
cpu: 1
memory: 2Gi
Requests:
cpu: 100m
memory: 1Gi
Liveness: exec [/bin/bash -ec nodetool status
] delay=10s timeout=10s period=20s #success=1 #failure=3
Readiness: exec [/bin/bash -ec nodetool status | grep -E "^UN\\s+${POD_IP}"
] delay=10s timeout=10s period=20s #success=1 #failure=3
Environment:
CASSANDRA_USER: <set to the key 'login' in secret
'development-portal-cassandra-db-creds'> Optional: false
CASSANDRA_PASSWORD: <set to the key 'password' in secret
'development-portal-cassandra-db-creds'> Optional: false
POD_IP: (v1:status.podIP)
CASSANDRA_PASSWORD_SEEDER: yes
BITNAMI_DEBUG: true
CASSANDRA_CLUSTER_NAME: cassandra
CASSANDRA_NUM_TOKENS: 256
CASSANDRA_DATACENTER: dc1
CASSANDRA_ENDPOINT_SNITCH: SimpleSnitch
CASSANDRA_RACK: rack1
CASSANDRA_ENABLE_RPC: true
MAX_HEAP_SIZE: 3G
HEAP_NEWSIZE: 100M
Mounts:
/docker-entrypoint-initdb.d/aaa_portal.cql from
cassandra-docker-entrypoint-initdb (rw,path="portal.cql")
/etc/localtime from localtime (ro)
/var/lib/cassandra/data from development-portal-cassandra-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-phb2g (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cassandra-docker-entrypoint-initdb:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: development-portal-cassandra-docker-entry-initd
Optional: false
localtime:
Type: HostPath (bare host directory volume)
Path: /etc/localtime
HostPathType:
development-portal-cassandra-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in
the same namespace)
ClaimName: development-portal-cassandra
ReadOnly: false
kube-api-access-phb2g:
Type: Projected (a volume that contains injected data from
multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists
for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 38m default-scheduler Successfully assigned
onap/development-portal-cassandra-75b756869f-8hzdf to onap
Normal Pulled 38m kubelet Successfully pulled image
"docker.io/bitnami/cassandra:3.11.9-debian-10-r30" in 39.241690567s
Warning FailedPreStopHook 37m kubelet Exec lifecycle hook ([bash
-ec nodetool decommission]) for Container "portal-cassandra" in Pod
"development-portal-cassandra-75b756869f-8hzdf_onap(bc5babf3-acd2-43b1-ab13-f390fc7ee91d)"
failed - error: command 'bash -ec nodetool decommission' exited with 1:
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection
refused (Connection refused)'.
, message: "nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
'Connection refused (Connection refused)'.\n"
Normal Pulling 36m (x2 over 38m) kubelet Pulling image
"docker.io/bitnami/cassandra:3.11.9-debian-10-r30"
Warning Unhealthy 36m kubelet Readiness probe errored: rpc
error: code = Unknown desc = container not running
(7a604524b0bcfd4764642847cedd6642658ba1f1d85967251a23af5d16e4f29d)
Normal Pulled 36m kubelet Successfully pulled image
"docker.io/bitnami/cassandra:3.11.9-debian-10-r30" in 5.917205899s
Normal Created 36m (x2 over 38m) kubelet Created container
portal-cassandra
Normal Started 36m (x2 over 38m) kubelet Started container
portal-cassandra
Warning Unhealthy 35m (x6 over 37m) kubelet Liveness probe failed:
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection
refused (Connection refused)'.
Normal Killing 33m (x3 over 37m) kubelet Container portal-cassandra
failed liveness probe, will be restarted
Warning Unhealthy 13m (x49 over 37m) kubelet Readiness probe failed:
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection
refused (Connection refused)'.
Warning BackOff 3m40s (x74 over 26m) kubelet Back-off restarting failed
container
Although I am unable to see any error in the logs:
cassandra 20:08:58.73
cassandra 20:08:58.74 Welcome to the Bitnami cassandra container
cassandra 20:08:58.74 Subscribe to project updates by watching
https://github.com/bitnami/bitnami-docker-cassandra
cassandra 20:08:58.75 Submit issues and feature requests at
https://github.com/bitnami/bitnami-docker-cassandra/issues
cassandra 20:08:58.75
cassandra 20:08:58.76 INFO ==> ** Starting Cassandra setup **
cassandra 20:08:58.80 INFO ==> Validating settings in CASSANDRA_* env vars..
cassandra 20:08:58.95 INFO ==> Initializing Cassandra database...
cassandra 20:08:59.02 DEBUG ==> No injected README.txt file found. Creating
default README.txt file
cassandra 20:08:59.06 DEBUG ==> No injected cassandra-env.ps1 file found.
Creating default cassandra-env.ps1 file
cassandra 20:08:59.10 DEBUG ==> No injected cassandra-env.sh file found.
Creating default cassandra-env.sh file
cassandra 20:08:59.14 DEBUG ==> No injected cassandra-jaas.config file found.
Creating default cassandra-jaas.config file
cassandra 20:08:59.18 DEBUG ==> No injected cassandra-rackdc.properties file
found. Creating default cassandra-rackdc.properties file
cassandra 20:08:59.23 DEBUG ==> No injected cassandra-topology.properties file
found. Creating default cassandra-topology.properties file
cassandra 20:08:59.27 DEBUG ==> No injected cassandra.yaml file found. Creating
default cassandra.yaml file
cassandra 20:08:59.31 DEBUG ==> No injected commitlog_archiving.properties file
found. Creating default commitlog_archiving.properties file
cassandra 20:08:59.36 DEBUG ==> No injected cqlshrc.sample file found. Creating
default cqlshrc.sample file
cassandra 20:08:59.40 DEBUG ==> No injected hotspot_compiler file found.
Creating default hotspot_compiler file
cassandra 20:08:59.45 DEBUG ==> No injected jvm.options file found. Creating
default jvm.options file
cassandra 20:08:59.48 DEBUG ==> No injected logback-tools.xml file found.
Creating default logback-tools.xml file
cassandra 20:08:59.52 DEBUG ==> No injected logback.xml file found. Creating
default logback.xml file
cassandra 20:08:59.56 DEBUG ==> No injected metrics-reporter-config-sample.yaml
file found. Creating default metrics-reporter-config-sample.yaml file
cassandra 20:08:59.60 DEBUG ==> No injected triggers/README.txt file found.
Creating default triggers/README.txt file
cassandra 20:08:59.99 DEBUG ==> Ensuring expected directories/files exist...
cassandra 20:09:00.02 INFO ==> Deploying Cassandra from scratch
cassandra 20:09:00.02 INFO ==> Starting Cassandra
cassandra 20:09:00.04 INFO ==> Checking that it started up correctly
cassandra 20:09:00.05 DEBUG ==> Checking that log
/opt/bitnami/cassandra/logs/cassandra_first_boot.log contains entry "Starting
listening for CQL clients"
/opt/bitnami/scripts/libos.sh: line 210: 202 Killed
"${cmd[@]}" "${args[@]}" > "$logger" 2>&1
Could anyone please help me with this.
Thanks and Regards,
Kushagra Gupta
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#24201): https://lists.onap.org/g/onap-discuss/message/24201
Mute This Topic: https://lists.onap.org/mt/93731617/21656
Mute #jakarta:https://lists.onap.org/g/onap-discuss/mutehashtag/jakarta
Mute #portal:https://lists.onap.org/g/onap-discuss/mutehashtag/portal
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
