Marco, Yes, not related to HEAT or OOM deployment – looks like nexus3/gerritt timeouts and moved/renamed docker images on nexus3 – as model-loader has similar issues since the 16th in the branch. Any ONAP deployment lately seems very fragile – as I have worked with a fully working stack just last Wed on HEAT
I tried several timing related workarounds, rebooting, delete/recreate stack, running parts of deploy_vm1.sh commented – also 2 of the 6 containers are pointing to openecomp nexus namespace not the new onap – like 4 of the newer services in aai1 – so the issue might be intermittent nexus3 access – which I have seen in the past In the other reference OOM deployment – all 9 containers come up ok – looking at differences in image versions between HEAT and OOM ubuntu@ip-172-31-31-252:~$ kubectl get pods --all-namespaces | grep aai onap-aai aai-resources-898583818-dnw9l 2/2 Running 0 24m onap-aai aai-service-3470292893-40dfh 1/1 Running 0 24m onap-aai aai-traversal-2843093465-sqgn5 2/2 Running 0 24m onap-aai data-router-2254557428-5z2sm 1/1 Running 0 24m onap-aai elasticsearch-622738319-88x11 1/1 Running 0 24m onap-aai hbase-1949550546-fxl7f 1/1 Running 0 24m onap-aai model-loader-service-1260517459-3wqmq 2/2 Running 0 24m onap-aai search-data-service-2243336541-btbdh 2/2 Running 0 24m onap-aai sparky-be-1292339831-xhmpx 2/2 Running 0 24m Also note that the unrelated model-loader builds have been failing since the 16th – so likely this is a nexus branching issue (images moving over to 1.1.0) https://jira.onap.org/browse/AAI-511 Started getting refused connections later today to gerrit see root@onap-aai-inst1:/opt/test-config# cat deploy_vm1.sh RESOURCES_CONTAINER_NAME=$($DOCKER_COMPOSE_CMD up -d aai-resources.api.simpledemo.openecomp.org 2>&1 | grep 'Creating' | grep -v 'volume' | grep -v 'network' | awk '{ print $2; }' | head -1); wait_for_container $RESOURCES_CONTAINER_NAME '0.0.0.0:8447'; intermittent in open-lab Started also failing on contacting gerrit root@onap-aai-inst1:/opt# ./aai_vm_init.sh fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/': Could not resolve host: gerrit.onap.org root@onap-aai-inst1:/opt# ./aai_vm_init.sh fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/': Could not resolve host: gerrit.onap.org Error response from daemon: Get https://nexus3.onap.org:10001/v1/users/: dial tcp: lookup nexus3.onap.org on 10.0.100.1:53: read udp 10.0.1.1:34203->10.0.100.1:53: read: connection refused repos working https://gerrit.onap.org/r/#/admin/projects/aai/test-config From: PLATANIA, MARCO (MARCO) [mailto:[email protected]] Sent: Sunday, November 26, 2017 11:39 To: Michael O'Brien <[email protected]>; [email protected] Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Michael, We saw similar intermittent failures in cloud init. This is not related to heat in particular, but I agree that a component installation is incomplete if cloud init fails. We had to re-run the install script to bring up an affected component. Marco From: <[email protected]<mailto:[email protected]>> on behalf of Michael O'Brien <[email protected]<mailto:[email protected]>> Date: Sunday, November 26, 2017 at 1:13 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Team, I am experiencing the aai-vm1 startup errors we previously saw last week in HEAT. I deployed the latest yaml to open-lab and am having an issue manually bringing up the 6 containers in aai1 – specifically the resources one again. I understand there may be a set of scripts developed to work around the intermittent issue where vm1 is deployed before vm2. After 1 hour aai1 had not even pulled docker images until I manually started the install scripts off the /opt dir – shouldn’t these normally be bootstrapped? Aai-vm2’s hbase and es are up fine. It is an intermittent issue because I have seen deployments from last week where AAI fully works. The current workaround is to delete/recreate the entire HEAT stack. Let us know if there is another workaround. https://jira.onap.org/browse/AAI-513<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_AAI-2D513&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=RCSGr1kIZ6BqOK6Fre_hPnaJFiCVvFEKo8z4JavSqC8&s=uhcYhh0UeGZVza9c1ds7d94RfwZJynfvvou2-CxuZZY&e=> /michael This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=RCSGr1kIZ6BqOK6Fre_hPnaJFiCVvFEKo8z4JavSqC8&s=X8u990ICHnre1qgOXNoN432RLLT0VMc100wqrKdh21k&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer <https://www.amdocs.com/about/email-disclaimer>
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
