Beware that some pods become out-of-band of kubernetes for their pv/pvc - a
redeploy may fail outright - the helm deploy is listed as failed - check the
log in this case
https://wiki.onap.org/display/DW/ONAP+Development#ONAPDevelopment-RemoveaDeployment
sudo cat ~/.helm/plugins/deploy/cache/onap/logs/onap-*.log
then delete specific pv's not in general like
kubectl delete pv --all
kubectl delete pvc --all
kubectl delete secrets --all
kubectl delete clusterrolebinding --all
otherwise be aware that depending on the DB - a wipe of /dockerdata-nfs/onap-*
may be required allow for the config jobs to rerun - will be fixed by the
upcoming helm hooks change
bottom line is to be save wipe the subdir between deploys and clean up any pvs
that show up in the deploy/undeploy logs if there any - as in
https://git.onap.org/logging-analytics/tree/deploy/cd.sh#n142
3.0.0-ONAP - up for a week OK
onap onap-dmaap-dbc-pg-0
1/1 Running 0 7d
onap onap-dmaap-dbc-pg-1
1/1 Running 0 7d
onap onap-dmaap-dbc-pgpool-c5f8498-4mr2d
1/1 Running 0 7d
onap onap-dmaap-dbc-pgpool-c5f8498-vhgcs
1/1 Running 0 7d
onap onap-dmaap-dmaap-bus-controller-557dc8c59c-p8fjd
1/1 Running 0 7d
onap onap-dmaap-dmaap-dr-db-576f7968b8-qsr6f
1/1 Running 0 7d
onap onap-dmaap-dmaap-dr-node-7647f9d6d8-bsfmx
1/1 Running 0 7d
onap onap-dmaap-dmaap-dr-prov-f4d84869f-45jl6
1/1 Running 0 7d
onap onap-dmaap-message-router-76f4799d-g6gh5
1/1 Running 0 7d
onap onap-dmaap-message-router-kafka-56c7c546c-lm7hj
1/1 Running 8 7d
onap onap-dmaap-message-router-zookeeper-7d69b496bc-n82qg
1/1 Running 0 7d
/michael
From: [email protected] <[email protected]> On Behalf Of
Dominic Lunanuova
Sent: Wednesday, January 16, 2019 8:53 AM
To: [email protected]; [email protected]; Michael
O'Brien <[email protected]>
Subject: Re: [onap-discuss] [oom] DMAAP pg-0 pod failing in Casablanca
Bharath,
You didn’t say you were redeploying, but this resembles issue
https://jira.onap.org/browse/OOM-1463
If this is not your issue, please provide more background on what the sequence
of events was.
i.e. were you running successfully for 1 week and then these pods died? Or,
was there some other action that occurred before these pods reported errors?
-Dom
From: [email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Bharath Thiruveedula via
Lists.Onap.Org
Sent: Wednesday, January 16, 2019 8:23 AM
To: Michael O'Brien <[email protected]<mailto:[email protected]>>;
onap-discuss <[email protected]<mailto:[email protected]>>
Subject: [onap-discuss] [oom] DMAAP pg-0 pod failing in Casablanca
Hi,
I have been running Casablanca in my setup for a week. From today I can see
postgres containers are not in running state. Especialy -dmaap-dbc-pg-0 pod.
I got the below logs from the pod.
INFO: Setting PGROOT to /usr/pgsql-10.
INFO: Cleaning up the old postmaster.pid file..
INFO: User ID is set to uid=26(postgres) gid=26(postgres) groups=26(postgres).
INFO: Ordinal is set to 0.
pod "dev-dmaap-dbc-pg-0" not labeled
pod "dev-dmaap-dbc-pg-0" not labeled
INFO: Setting PG_MODE to primary.
INFO: Working on primary..
INFO: Initializing the primary database..
INFO: PGDATA already contains a database.
INFO: Starting PostgreSQL..
[75] LOG: skipping missing configuration file
"/pgdata/dev-dmaap-dbc-pg-0/postgresql.auto.conf"
[2-1] user=,db=,app=,client=FATAL: "/pgdata/dev-dmaap-dbc-pg-0" is not a valid
data directory
[3-1] user=,db=,app=,client=DETAIL: File
"/pgdata/dev-dmaap-dbc-pg-0/PG_VERSION" is missing.
INFO: PostgreSQL is shutting down. Exiting..
Couldn't figure out the exact reason. Anyone came across the issue.
Best Regards
Bharath T
This email and the information contained herein is proprietary and confidential
and subject to the Amdocs Email Terms of Service, which you may review at
https://www.amdocs.com/about/email-terms-of-service
<https://www.amdocs.com/about/email-terms-of-service>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#15036): https://lists.onap.org/g/onap-discuss/message/15036
Mute This Topic: https://lists.onap.org/mt/29155546/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
