Hi Michael, Apologies for late reply. I was on vacation.
After couple of deployments and cleaning NFS, OOF pod started running. As you said, seems like race condition. Thanks for the help. Best Regards Bharath T On Wed, Jan 9, 2019 at 9:44 PM Michael O'Brien <[email protected]> wrote: > Bharath, > > > > Hi, see my comments on sequencing, docker preload for aaf – same issue > – a matter of timing/resources until helm hooks are in > > 3.0.0-ONAP this morning on my latest test run for > > https://gerrit.onap.org/r/#/c/75422 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.onap.org_r_-23_c_75422&d=DwMGaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=UTP8N06L2trD3jawJkElFfA6CQYtkCWzTnn4GsHkmJw&m=BMuaRj6g1__FpgVEd_-6-bmvzvu4GkrsCvtNzqlxWoo&s=P1gp0VgrI4bRuIZwuI9TdruJnYrk1nmABK6kKCFj-Lk&e=> > > > > that particular container seems to be very sensitive to > resource/timing factors and may need to have its liveness/readiness limits > and the job rechecked – however it does start 2 out of 3 times on my last 3 > runs for 3.0.0-ONAP > > > > yesterday - note that even a crashed pod will say 1/1 Running for a couple > seconds on each retry - so I pasted 2 copies - one at 6 and 12h > > - different run in this case > > > > 3rd last > > onap > onap-oof-oof-5b8c698f4d-4st95 1/1 > Running 0 6h > > > > 2nd last > > onap onap-oof-cmso-db-0 > 1/1 Running 0 12h > > onap onap-oof-music-cassandra-0 > 1/1 Running 0 12h > > onap onap-oof-music-cassandra-1 > 1/1 Running 0 12h > > onap onap-oof-music-cassandra-2 > 1/1 Running 0 12h > > onap onap-oof-music-cassandra-job-config-dp2hr > 0/1 Completed 0 12h > > onap onap-oof-music-tomcat-c7b8f6488-bc6ph > 1/1 Running 0 12h > > onap onap-oof-music-tomcat-c7b8f6488-c2zcw > 1/1 Running 0 12h > > onap onap-oof-music-tomcat-c7b8f6488-jk74z > 1/1 Running 0 12h > > onap onap-oof-oof-5b8c698f4d-k8prv > 1/1 Running 0 12h > > onap onap-oof-oof-cmso-service-6454cf5994-krkjz > 1/1 Running 0 12h > > onap onap-oof-oof-has-api-748978b8d4-stvvn > 1/1 Running 0 12h > > onap onap-oof-oof-has-controller-797dc5f77f-68fxc > 1/1 Running 0 12h > > onap onap-oof-oof-has-data-678ccbb6c9-wcjx5 > 1/1 Running 0 12h > > onap onap-oof-oof-has-healthcheck-vphql > 0/1 Completed 0 12h > > onap onap-oof-oof-has-onboard-w4ghs > 0/1 Completed 0 12h > > onap onap-oof-oof-has-reservation-765c68fc45-gc4rf > 1/1 Running 0 12h > > onap onap-oof-oof-has-solver-54fb478dfb-r7fgk > 1/1 Running 0 12h > > onap onap-oof-zookeeper-0 > 1/1 Running 0 12h > > onap onap-oof-zookeeper-1 > 1/1 Running 0 12h > > onap onap-oof-zookeeper-2 > 1/1 Running 0 12h > > > > > > today (no image change on the 3.0.0-ONAP tags) > > onap onap-oof-cmso-db-0 1/1 > Running 0 58m > > onap onap-oof-music-cassandra-0 1/1 > Running 0 58m > > onap onap-oof-music-cassandra-1 1/1 > Running 0 57m > > onap onap-oof-music-cassandra-2 1/1 > Running 0 56m > > onap onap-oof-music-cassandra-job-config-vp9sd 0/1 > Completed 0 58m > > onap onap-oof-music-tomcat-c7b8f6488-4lxjd 1/1 > Running 0 58m > > onap onap-oof-music-tomcat-c7b8f6488-ncjml 1/1 > Running 0 58m > > onap onap-oof-music-tomcat-c7b8f6488-vvdgd 1/1 > Running 0 58m > > onap onap-oof-oof-5b8c698f4d-ph4hq 0/1 > Init:0/2 5 58m > > onap onap-oof-oof-cmso-service-6454cf5994-f57g7 1/1 > Running 0 58m > > onap onap-oof-oof-has-api-748978b8d4-48w9j 1/1 > Running 0 58m > > onap onap-oof-oof-has-controller-797dc5f77f-jgcbf 1/1 > Running 0 58m > > onap onap-oof-oof-has-data-678ccbb6c9-pvsfn 1/1 > Running 0 58m > > onap onap-oof-oof-has-healthcheck-xqh7r 0/1 > Completed 0 58m > > onap onap-oof-oof-has-onboard-jcvxt 0/1 > Completed 0 58m > > onap onap-oof-oof-has-reservation-765c68fc45-p285r 1/1 > Running 0 58m > > onap onap-oof-oof-has-solver-54fb478dfb-q47tz 1/1 > Running 0 58m > > > > > > *From:* [email protected] <[email protected]> *On > Behalf Of *Bharath Thiruveedula via Lists.Onap.Org > *Sent:* Wednesday, January 9, 2019 6:30 AM > *To:* onap-discuss <[email protected]>; Michael O'Brien < > [email protected]> > *Subject:* [onap-discuss] [OOF][OOM] Unable to bring OOF Pod in Casablanca > > > > Hi, > > > > I am trying to bring Casablanca setup to test OOF PCI usecase. All the > pods are up except the OOF pod. The status of the oof pods looks like below: > > > > dev-oof-cmso-db-0 1/1 Running > 0 1h > > dev-oof-music-cassandra-0 1/1 Running > 3 1h > > dev-oof-music-cassandra-1 1/1 Running > 0 1h > > dev-oof-music-cassandra-2 1/1 Running > 0 1h > > dev-oof-music-cassandra-job-config-wd5g9 0/1 Completed > 0 1h > > dev-oof-music-tomcat-66f77966b7-cwp64 1/1 Running > 0 1h > > dev-oof-music-tomcat-66f77966b7-klrlm 1/1 Running > 0 1h > > dev-oof-music-tomcat-66f77966b7-thplm 1/1 Running > 0 1h > > dev-oof-oof-788f484fd9-spn4z 0/1 > CrashLoopBackOff 17 1h > > dev-oof-oof-cmso-service-768ff6596c-kbtmw 1/1 Running > 0 1h > > dev-oof-oof-has-api-5cb7fdcb44-tl5xb 1/1 Running > 0 1h > > dev-oof-oof-has-controller-74666787b9-w92ml 1/1 Running > 0 1h > > dev-oof-oof-has-data-7c755bb546-xv7wl 1/1 Running > 0 1h > > dev-oof-oof-has-healthcheck-hghwk 0/1 Completed > 0 1h > > dev-oof-oof-has-onboard-sxbs6 0/1 Completed > 0 1h > > dev-oof-oof-has-reservation-6c6ff78cb8-84cn9 1/1 Running > 0 1h > > dev-oof-oof-has-solver-57d48c596c-k9djs 1/1 Running > 0 1h > > dev-oof-zookeeper-0 1/1 Running > 0 1h > > dev-oof-zookeeper-1 1/1 Running > 0 1h > > dev-oof-zookeeper-2 1/1 Running > 0 1h > > > > > > The only logs i can see in dev-oof-oof-788f484fd9-spn4z pod is > > > > *Updating certificates in /etc/ssl/certs...* > > *1 added, 0 removed; done.* > > *Running hooks in /etc/ca-certificates/update.d...* > > *done.* > > > > and then the pod is going down. Even I removed liveness/readiness probe > if that is causing any problem. But didn't solve the issue. > > > > Following is the description of the pod. > > > > Events: > > Type Reason Age From Message > > ---- ------ ---- ---- ------- > > Normal Scheduled 1m default-scheduler > Successfully assigned onap/dev-oof-oof-788f484fd9-z2sgv to onap-oof-pci > > Normal Pulled 1m kubelet, onap-oof-pci Container > image "oomk8s/readiness-check:2.0.0" already present on machine > > Normal Created 1m kubelet, onap-oof-pci Created > container > > Normal Started 1m kubelet, onap-oof-pci Started > container > > Normal Pulled 1m kubelet, onap-oof-pci Container > image "oomk8s/readiness-check:2.0.0" already present on machine > > Normal Created 1m kubelet, onap-oof-pci Created > container > > Normal Started 1m kubelet, onap-oof-pci Started > container > > Normal Pulled 34s (x4 over 1m) kubelet, onap-oof-pci Container > image "nexus3.onap.org:10001/onap/optf-osdf:1.2.4 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_optf-2Dosdf-3A1.2.4&d=DwMGaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=UTP8N06L2trD3jawJkElFfA6CQYtkCWzTnn4GsHkmJw&m=BMuaRj6g1__FpgVEd_-6-bmvzvu4GkrsCvtNzqlxWoo&s=gotjQ4h3kwaQmGMYuscoKNbqJXOtfFronYay8jweWLM&e=>" > already present on machine > > Normal Created 34s (x4 over 1m) kubelet, onap-oof-pci Created > container > > Normal Started 34s (x4 over 1m) kubelet, onap-oof-pci Started > container > > Warning BackOff 14s (x5 over 1m) kubelet, onap-oof-pci Back-off > restarting failed container > > > > As you see I am using 1.2.4 version of oof docker image. Is that the > correct one? > > > > Best Regards > > Bharath T > > > > *This email and the information contained herein is proprietary and > confidential and subject to the Amdocs Email Terms of Service, which you > may review at* *https://www.amdocs.com/about/email-terms-of-service* > <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Dterms-2Dof-2Dservice&d=DwMGaQ&c=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ&r=UTP8N06L2trD3jawJkElFfA6CQYtkCWzTnn4GsHkmJw&m=BMuaRj6g1__FpgVEd_-6-bmvzvu4GkrsCvtNzqlxWoo&s=of45ENrZjspFb7UgdQsOIoEfiBvsnBKBPzbKZUQoCAY&e=> > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14990): https://lists.onap.org/g/onap-discuss/message/14990 Mute This Topic: https://lists.onap.org/mt/29146517/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
