Hi Team,
As a part of frankfurt deployment, we identifed an issue related to the
mariadb-galera.
In the helm deployment trace also we observed that two times the mariadb-galera
& cassandra charts are deploying, so 1st time its installing & 2nd time it's
upgrading. We are attaching our environment traces in the helm_trace.txt
file,which explain the issue step by step. Please have a look of that file once.
Trace:
Release "frankfurt-mariadb-galera" does not exist. Installing it now.
NAME: frankfurt-mariadb-galera
LAST DEPLOYED: Fri Jun 19 14:24:38 2020
NAMESPACE: onap
STATUS: DEPLOYED
Release "frankfurt-mariadb-galera" has been upgraded.
LAST DEPLOYED: Fri Jun 19 14:57:39 2020
NAMESPACE: onap
STATUS: DEPLOYED
Note : Similarly aaf & cassandra charts are also deploying twice. You can see
the same in helm list revisions.
root@frankfurtrnd-nfs:/home/ubuntu# helm list
NAME REVISION UPDATED STATUS CHART
APP VERSION NAMESPACE
frankfurt 1 Fri Jun 19 14:22:48 2020 DEPLOYED onap-6.0.0
El Alto onap
frankfurt-aaf 2 Fri Jun 19 14:49:23 2020 DEPLOYED aaf-6.0.0
onap
frankfurt-aai 1 Fri Jun 19 14:50:11 2020 DEPLOYED aai-6.0.0
onap
frankfurt-appc 1 Fri Jun 19 14:50:48 2020 DEPLOYED appc-6.0.0
onap
frankfurt-cassandra 2 Fri Jun 19 14:51:23 2020 DEPLOYED
cassandra-6.0.0 onap
frankfurt-cds 1 Fri Jun 19 14:51:53 2020 DEPLOYED cds-6.0.0
onap
frankfurt-clamp 1 Fri Jun 19 14:52:26 2020 DEPLOYED clamp-6.0.0
onap
frankfurt-cli 1 Fri Jun 19 14:52:58 2020 DEPLOYED cli-6.0.0
onap
frankfurt-consul 1 Fri Jun 19 14:53:29 2020 DEPLOYED consul-6.0.0
onap
frankfurt-contrib 1 Fri Jun 19 14:54:00 2020 DEPLOYED contrib-6.0.0
onap
frankfurt-dcaegen2 1 Fri Jun 19 14:54:36 2020 DEPLOYED
dcaegen2-6.0.0 onap
frankfurt-esr 1 Fri Jun 19 14:56:37 2020 DEPLOYED esr-6.0.0
onap
frankfurt-log 1 Fri Jun 19 14:57:08 2020 DEPLOYED log-6.0.0
onap
frankfurt-mariadb-galera 2 Fri Jun 19 14:57:39 2020 DEPLOYED
mariadb-galera-6.0.0 onap
frankfurt-modeling 1 Fri Jun 19 15:08:07 2020 DEPLOYED
modeling-6.0.0 onap
In the process of helm deployment, mariadb-galera had got installed intially,
but after sometime when a new job i.e(mariadb-galera-pre-upgrade job) got
initialized the existing mariadb-galera pods had got terminated, again after
sometime this pre-upgrade job cleared from the list and a new job (i.e
mariadb-galera-post-upgrade job) had got created even that also cleared from
list. We are attaching our environment traces in the Job_Pod_trace.txt
file,which explain the issue step by step. Please have a look of that file once.
For the same we raised a jira also : https://jira.onap.org/browse/INT-1633
Regards,
________________________________________________________
Velugubantla Praveen
Engineer - CTO-Common
L&T TECHNOLOGY SERVICES LIMITED
L3 Building, Manyata Embassy Business Park,
Nagawara Hobli, Bengaluru-560045
________________________________________________________
Mobile: +91 9154111420
www.LTTS.com<http://www.ltts.com/>
[cid:9e304832-1259-409e-b12b-9a357fa49782]
L&T Technology Services Ltd
www.LTTS.com
L&T Technology Services Limited (LTTS) is committed to safeguard your data
privacy. For more information to view our commitment towards data privacy under
GDPR, please visit the privacy policy on our website www.Ltts.com. This Email
may contain confidential or privileged information for the intended recipient
(s). If you are not the intended recipient, please do not use or disseminate
the information, notify the sender and delete it from your system.
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#21455): https://lists.onap.org/g/onap-discuss/message/21455
Mute This Topic: https://lists.onap.org/mt/74984907/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
########################### frankfurt-cassandra Installing
################################
Release "frankfurt-cassandra" does not exist. Installing it now.
NAME: frankfurt-cassandra
LAST DEPLOYED: Fri Jun 19 14:24:08 2020
NAMESPACE: onap
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
frankfurt-cassandra-entrypoint 1 0s
==> v1/PersistentVolume
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS
CLAIM STORAGECLASS
REASON AGE
frankfurt-cassandra--0 2Gi RWO Retain Bound
onap/frankfurt-cassandra-data-frankfurt-cassandra-0 frankfurt-cassandra-data
0s
frankfurt-cassandra--1 2Gi RWO Retain Available
frankfurt-cassandra-data 0s
frankfurt-cassandra--2 2Gi RWO Retain Available
frankfurt-cassandra-data 0s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
frankfurt-cassandra-0 0/1 Pending 0 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
AGE
cassandra ClusterIP None <none>
7000/TCP,7001/TCP,7199/TCP,9042/TCP,9160/TCP,61621/TCP 0s
==> v1/StatefulSet
NAME READY AGE
frankfurt-cassandra 0/3 0s
---------------------------------------------------------------------------------------------------------
############################## frankfurt-mariadb-galera Installing
##############################
sleep 30s
Release "frankfurt-mariadb-galera" does not exist. Installing it now.
NAME: frankfurt-mariadb-galera
LAST DEPLOYED: Fri Jun 19 14:24:38 2020
NAMESPACE: onap
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
frankfurt-mariadb-galera 1 0s
==> v1/PersistentVolume
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS
CLAIM STORAGECLASS
REASON AGE
frankfurt-mariadb-galera-data-0 2Gi RWO Retain
Available frankfurt-mariadb-galera-data 0s
frankfurt-mariadb-galera-data-1 2Gi RWO Retain Bound
onap/frankfurt-mariadb-galera-data-frankfurt-mariadb-galera-0
frankfurt-mariadb-galera-data 0s
frankfurt-mariadb-galera-data-2 2Gi RWO Retain
Available frankfurt-mariadb-galera-data 0s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
frankfurt-mariadb-galera-0 0/1 Pending 0 0s
==> v1/Secret
NAME TYPE DATA AGE
frankfurt-mariadb-galera-db-root-password Opaque 1 0s
frankfurt-mariadb-galera-db-user-credentials Opaque 2 0s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mariadb-galera ClusterIP None <none> 3306/TCP 0s
==> v1beta1/StatefulSet
NAME READY AGE
frankfurt-mariadb-galera 0/3 0s
============================================================================================================================
################################# frankfurt-cassandra upgraded
#################################
Release "frankfurt-cassandra" has been upgraded.
LAST DEPLOYED: Fri Jun 19 14:51:23 2020
NAMESPACE: onap
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
frankfurt-cassandra-entrypoint 1 27m
==> v1/PersistentVolume
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
STORAGECLASS REASON
AGE
frankfurt-cassandra--0 2Gi RWO Retain Bound
onap/frankfurt-cassandra-data-frankfurt-cassandra-0 frankfurt-cassandra-data
27m
frankfurt-cassandra--1 2Gi RWO Retain Bound
onap/frankfurt-cassandra-data-frankfurt-cassandra-2 frankfurt-cassandra-data
27m
frankfurt-cassandra--2 2Gi RWO Retain Bound
onap/frankfurt-cassandra-data-frankfurt-cassandra-1 frankfurt-cassandra-data
27m
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
frankfurt-cassandra-0 1/1 Running 0 27m
frankfurt-cassandra-1 1/1 Running 0 20m
frankfurt-cassandra-2 1/1 Running 0 16m
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
AGE
cassandra ClusterIP None <none>
7000/TCP,7001/TCP,7199/TCP,9042/TCP,9160/TCP,61621/TCP 27m
==> v1/StatefulSet
NAME READY AGE
frankfurt-cassandra 3/3 27m
---------------------------------------------------------------------------------------------------------------------------------------------------------
################################ frankfurt-mariadb-galera Upgraded
#####################################
Release "frankfurt-mariadb-galera" has been upgraded.
LAST DEPLOYED: Fri Jun 19 14:57:39 2020
NAMESPACE: onap
STATUS: DEPLOYED
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
frankfurt-mariadb-galera 1 42m
==> v1/PersistentVolume
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS
CLAIM STORAGECLASS
REASON AGE
frankfurt-mariadb-galera-data-0 2Gi RWO Retain
Released onap/frankfurt-mariadb-galera-data-frankfurt-mariadb-galera-2
frankfurt-mariadb-galera-data 42m
frankfurt-mariadb-galera-data-1 2Gi RWO Retain
Released onap/frankfurt-mariadb-galera-data-frankfurt-mariadb-galera-0
frankfurt-mariadb-galera-data 42m
frankfurt-mariadb-galera-data-2 2Gi RWO Retain
Released onap/frankfurt-mariadb-galera-data-frankfurt-mariadb-galera-1
frankfurt-mariadb-galera-data 42m
==> v1/Pod(related)
==> v1/Secret
NAME TYPE DATA AGE
frankfurt-mariadb-galera-db-root-password Opaque 1 42m
frankfurt-mariadb-galera-db-user-credentials Opaque 2 42m
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mariadb-galera ClusterIP None <none> 3306/TCP 42m
==> v1beta1/StatefulSet
NAME READY AGE
frankfurt-mariadb-galera 0/0 42m
NOTES:
# Copyright © 2018 Amdocs, Bell Canada
#
# This file is licensed under the CREATIVE COMMONS ATTRIBUTION 4.0
INTERNATIONAL LICENSE
#
# Full license text at https://creativecommons.org/licenses/by/4.0/legalcode
MariaDB-Galera service can be accessed via port 3306 on the following DNS name
from within your cluster:
frankfurt-mariadb-galera.onap.svc.cluster.local
To connect to your database:
1. Run a pod that you can use as a client:
kubectl run frankfurt-mariadb-galera-client --rm --tty -i --image mariadb
--command -- bash
2. Connect using the mysql cli, then provide your password:
$ mysql -h frankfurt-mariadb-galera
----------------------------------------------------------------------
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu# helm list
NAME REVISION UPDATED
STATUS CHART APP VERSION NAMESPACE
frankfurt 1 Fri Jun 19 14:22:48 2020
DEPLOYED onap-6.0.0 El Alto onap
frankfurt-aaf 2 Fri Jun 19 14:49:23 2020
DEPLOYED aaf-6.0.0 onap
frankfurt-aai 1 Fri Jun 19 14:50:11 2020
DEPLOYED aai-6.0.0 onap
frankfurt-appc 1 Fri Jun 19 14:50:48 2020
DEPLOYED appc-6.0.0 onap
frankfurt-cassandra 2 Fri Jun 19 14:51:23 2020
DEPLOYED cassandra-6.0.0 onap
frankfurt-cds 1 Fri Jun 19 14:51:53 2020
DEPLOYED cds-6.0.0 onap
frankfurt-clamp 1 Fri Jun 19 14:52:26 2020
DEPLOYED clamp-6.0.0 onap
frankfurt-cli 1 Fri Jun 19 14:52:58 2020
DEPLOYED cli-6.0.0 onap
frankfurt-consul 1 Fri Jun 19 14:53:29 2020
DEPLOYED consul-6.0.0 onap
frankfurt-contrib 1 Fri Jun 19 14:54:00 2020
DEPLOYED contrib-6.0.0 onap
frankfurt-dcaegen2 1 Fri Jun 19 14:54:36 2020
DEPLOYED dcaegen2-6.0.0 onap
frankfurt-esr 1 Fri Jun 19 14:56:37 2020
DEPLOYED esr-6.0.0 onap
frankfurt-log 1 Fri Jun 19 14:57:08 2020
DEPLOYED log-6.0.0 onap
frankfurt-mariadb-galera 2 Fri Jun 19 14:57:39 2020
DEPLOYED mariadb-galera-6.0.0 onap
frankfurt-modeling 1 Fri Jun 19 15:08:07 2020
DEPLOYED modeling-6.0.0 onap
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu# kubectl get jobs
NAME COMPLETIONS DURATION AGE
frankfurt-aaf-sms-preload 0/1 34m 34m
frankfurt-aaf-sshsm-distcenter 1/1 5m2s 34m
frankfurt-aaf-sshsm-testca 1/1 8m27s 34m
frankfurt-aai-graphadmin-create-db-schema 1/1 5m16s 8m5s
frankfurt-aai-traversal-update-query-data 0/1 8m5s 8m5s
frankfurt-awx 0/1 4m17s 4m17s
frankfurt-ejbca-config-config-job 0/1 4m17s 4m17s
frankfurt-mariadb-galera-pre-upgrade 0/1 39s 39s
frankfurt-netbox-app-provisioning 1/1 3m32s 4m17s
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu# kubectl get pods | grep galera
frankfurt-mariadb-galera-0 1/1 Running
0 33m
frankfurt-mariadb-galera-1 1/1 Running
0 29m
frankfurt-mariadb-galera-2 1/1 Running
0 26m
frankfurt-mariadb-galera-pre-upgrade-5sf27 0/1
ContainerCreating 0 46s
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu# kubectl get pods | grep galera
frankfurt-mariadb-galera-0 1/1 Running
0 37m
frankfurt-mariadb-galera-1 1/1 Running
0 33m
frankfurt-mariadb-galera-2 1/1 Running
0 30m
frankfurt-mariadb-galera-pre-upgrade-5sf27 0/1
ContainerCreating 0 4m52s
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu# kubectl get jobs
NAME COMPLETIONS DURATION AGE
frankfurt-aaf-sms-preload 0/1 40m 40m
frankfurt-aaf-sshsm-distcenter 1/1 5m2s 40m
frankfurt-aaf-sshsm-testca 1/1 8m27s 40m
frankfurt-aai-graphadmin-create-db-schema 1/1 5m16s 14m
frankfurt-aai-traversal-update-query-data 0/1 14m 14m
frankfurt-awx 0/1 10m 10m
frankfurt-ejbca-config-config-job 1/1 8m5s 10m
frankfurt-mariadb-galera-pre-upgrade 0/1 6m36s 6m36s
frankfurt-netbox-app-provisioning 1/1 3m32s 10m
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu# kubectl get pods | grep mariadb
NAME READY STATUS
RESTARTS AGE
frankfurt-cassandra-0 1/1 Running
0 40m
frankfurt-cassandra-1 1/1 Running
0 33m
frankfurt-cassandra-2 1/1 Running
0 29m
frankfurt-mariadb-75985d9b88-fg8rz 1/1 Running
1 11m
frankfurt-mariadb-galera-upgrade-deployment-66bd4c654f-tnvqn 1/1 Running
0 101s
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu# kubectl get jobs
NAME COMPLETIONS DURATION AGE
frankfurt-aaf-sms-preload 0/1 43m 43m
frankfurt-aaf-sshsm-distcenter 1/1 5m2s 43m
frankfurt-aaf-sshsm-testca 1/1 8m27s 43m
frankfurt-aai-graphadmin-create-db-schema 1/1 5m16s 16m
frankfurt-aai-traversal-update-query-data 0/1 16m 16m
frankfurt-awx 1/1 11m 12m
frankfurt-ejbca-config-config-job 1/1 8m5s 12m
frankfurt-mariadb-galera-post-upgrade 0/1 2m29s 2m29s
frankfurt-netbox-app-provisioning 1/1 3m32s 12m
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu# kubectl get pods | grep mariadb
NAME READY STATUS
RESTARTS AGE
frankfurt-mariadb-75985d9b88-fg8rz 1/1 Running
1 14m
frankfurt-mariadb-galera-post-upgrade-xwx27 0/1
PodInitializing 0 2m43s
frankfurt-mariadb-galera-upgrade-deployment-66bd4c654f-tnvqn 1/1 Running
0 4m27s
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu#
root@frankfurtrnd-nfs:/home/ubuntu# kubectl get pods | grep mariadb-galera
root@frankfurtrnd-nfs:/home/ubuntu#
