Michael
Following things - Can you try nc to run as a client and server and check if that communication is Ok . - nslookup or ip dig < fqdn > what is the result . - We are working on 16th Nov Master branch . - thanks gaurav ________________________________ From: [email protected] <[email protected]> on behalf of Michael O'Brien <[email protected]> Sent: 27 November 2017 10:38:07 To: [email protected] Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Discussed with Venkata, Can I get someone who has deployed aai1 on heat to validate my env settings – specifically my heat settings. After I actually looked at the failed docker log as well as Venkata suggested it does look like my DNS settings are to blame – I am having a hard time finding the recommended settings on the wiki or the PPT. and would like to document it as we go. aai.hbase.simpledemo.openecomp.org: forward host lookup failed: Unknown host Unable to connect to hbase after 200 retries, please check if hbase server is properly configured and be able to connect Unable to run init chef script, please check logs Onap-openstack.env dns_list: 8.8.4.4 external_dns: 8.8.4.4 dns_forwarder: 10.0.100.1 (was missing the dns-server ip) testing this now – but my current restack stops the /opt file downloads before the test-config folder is in where the deploy scripts are. From: [email protected] [mailto:[email protected]] On Behalf Of Michael O'Brien Sent: Sunday, November 26, 2017 13:33 To: PLATANIA, MARCO (MARCO) <[email protected]>; [email protected] Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images 30 min later on a re-stack we get connectivity again to gerrit/nexus – when manually bringing up the containers – will see if resources loads Narrows down the image:version issue – as non-issue – was worried about the dns setting in onap-openstack.env https://wiki.onap.org/display/DW/Vetted+vFirewall+Demo+-+Full+draft+how-to+for+F2F+and+ReadTheDocs#VettedvFirewallDemo-Fulldrafthow-toforF2FandReadTheDocs-OpenlabVNCandCLI<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_Vetted-2BvFirewall-2BDemo-2B-2D-2BFull-2Bdraft-2Bhow-2Dto-2Bfor-2BF2F-2Band-2BReadTheDocs-23VettedvFirewallDemo-2DFulldrafthow-2DtoforF2FandReadTheDocs-2DOpenlabVNCandCLI&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=ebJjFMpXijqZjbZCcbF7yJIq2ES6jM0Q-DEcP-qjjeI&m=LIlANk-U7HCH8ndymom-KmuBXT0D9cV5FRnZUNw53lI&s=u0KH25dXb8LDutjY97pnYO7YwcuxWGeIfQhYLpo0NIg&e=> – was using google’s 8.8.4.4 for the three dns* properties – need to vet these on Monday. root@onap-aai-inst1:/opt/test-config# ./deploy_vm1.sh Login Succeeded v1.1.0: Pulling from openecomp/aai-resources c62795f78da9: Already exists d4fceeeb758e: Already exists 5c9125a401ae: Already exists 0062f774e994: Already exists 6b33fd031fac: Already exists 9a115de3c55d: Already exists 55300a0e3724: Already exists 371557fd7d66: Downloading [=========================================> ] 137.4MB/165.8MB From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Michael O'Brien Sent: Sunday, November 26, 2017 13:04 To: PLATANIA, MARCO (MARCO) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; *_Amdocs DTV CONV CORE PMO <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Marco, Yes, not related to HEAT or OOM deployment – looks like nexus3/gerritt timeouts and moved/renamed docker images on nexus3 – as model-loader has similar issues since the 16th in the branch. Any ONAP deployment lately seems very fragile – as I have worked with a fully working stack just last Wed on HEAT I tried several timing related workarounds, rebooting, delete/recreate stack, running parts of deploy_vm1.sh commented – also 2 of the 6 containers are pointing to openecomp nexus namespace not the new onap – like 4 of the newer services in aai1 – so the issue might be intermittent nexus3 access – which I have seen in the past In the other reference OOM deployment – all 9 containers come up ok – looking at differences in image versions between HEAT and OOM ubuntu@ip-172-31-31-252:~$ kubectl get pods --all-namespaces | grep aai onap-aai aai-resources-898583818-dnw9l 2/2 Running 0 24m onap-aai aai-service-3470292893-40dfh 1/1 Running 0 24m onap-aai aai-traversal-2843093465-sqgn5 2/2 Running 0 24m onap-aai data-router-2254557428-5z2sm 1/1 Running 0 24m onap-aai elasticsearch-622738319-88x11 1/1 Running 0 24m onap-aai hbase-1949550546-fxl7f 1/1 Running 0 24m onap-aai model-loader-service-1260517459-3wqmq 2/2 Running 0 24m onap-aai search-data-service-2243336541-btbdh 2/2 Running 0 24m onap-aai sparky-be-1292339831-xhmpx 2/2 Running 0 24m Also note that the unrelated model-loader builds have been failing since the 16th – so likely this is a nexus branching issue (images moving over to 1.1.0) https://jira.onap.org/browse/AAI-511<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_AAI-2D511&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=ebJjFMpXijqZjbZCcbF7yJIq2ES6jM0Q-DEcP-qjjeI&m=LIlANk-U7HCH8ndymom-KmuBXT0D9cV5FRnZUNw53lI&s=4MrJOFcudUq6ISWeAj86wUYHM3RjIuYWxh-KAlG67wk&e=> Started getting refused connections later today to gerrit see root@onap-aai-inst1:/opt/test-config# cat deploy_vm1.sh RESOURCES_CONTAINER_NAME=$($DOCKER_COMPOSE_CMD up -d aai-resources.api.simpledemo.openecomp.org 2>&1 | grep 'Creating' | grep -v 'volume' | grep -v 'network' | awk '{ print $2; }' | head -1); wait_for_container $RESOURCES_CONTAINER_NAME '0.0.0.0:8447'; intermittent in open-lab Started also failing on contacting gerrit root@onap-aai-inst1:/opt# ./aai_vm_init.sh fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/': Could not resolve host: gerrit.onap.org root@onap-aai-inst1:/opt# ./aai_vm_init.sh fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/': Could not resolve host: gerrit.onap.org Error response from daemon: Get https://nexus3.onap.org:10001/v1/users/<https://urldefense.proofpoint.com/v2/url?u=https-3A__nexus3.onap.org-3A10001_v1_users_&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=ebJjFMpXijqZjbZCcbF7yJIq2ES6jM0Q-DEcP-qjjeI&m=LIlANk-U7HCH8ndymom-KmuBXT0D9cV5FRnZUNw53lI&s=URPkmSIUkpJMSUYu3bmyuCXWXrs56vhWfPDoIMTfqUA&e=>: dial tcp: lookup nexus3.onap.org on 10.0.100.1:53: read udp 10.0.1.1:34203->10.0.100.1:53: read: connection refused repos working https://gerrit.onap.org/r/#/admin/projects/aai/test-config<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.onap.org_r_-23_admin_projects_aai_test-2Dconfig&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=ebJjFMpXijqZjbZCcbF7yJIq2ES6jM0Q-DEcP-qjjeI&m=LIlANk-U7HCH8ndymom-KmuBXT0D9cV5FRnZUNw53lI&s=5pdMVZnKyyQQMA0ZsgNZla0Z3DjgyIJjaqJDxdm677I&e=> From: PLATANIA, MARCO (MARCO) [mailto:[email protected]] Sent: Sunday, November 26, 2017 11:39 To: Michael O'Brien <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Michael, We saw similar intermittent failures in cloud init. This is not related to heat in particular, but I agree that a component installation is incomplete if cloud init fails. We had to re-run the install script to bring up an affected component. Marco From: <[email protected]<mailto:[email protected]>> on behalf of Michael O'Brien <[email protected]<mailto:[email protected]>> Date: Sunday, November 26, 2017 at 1:13 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Team, I am experiencing the aai-vm1 startup errors we previously saw last week in HEAT. I deployed the latest yaml to open-lab and am having an issue manually bringing up the 6 containers in aai1 – specifically the resources one again. I understand there may be a set of scripts developed to work around the intermittent issue where vm1 is deployed before vm2. After 1 hour aai1 had not even pulled docker images until I manually started the install scripts off the /opt dir – shouldn’t these normally be bootstrapped? Aai-vm2’s hbase and es are up fine. It is an intermittent issue because I have seen deployments from last week where AAI fully works. The current workaround is to delete/recreate the entire HEAT stack. Let us know if there is another workaround. https://jira.onap.org/browse/AAI-513<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_AAI-2D513&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=RCSGr1kIZ6BqOK6Fre_hPnaJFiCVvFEKo8z4JavSqC8&s=uhcYhh0UeGZVza9c1ds7d94RfwZJynfvvou2-CxuZZY&e=> /michael This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=RCSGr1kIZ6BqOK6Fre_hPnaJFiCVvFEKo8z4JavSqC8&s=X8u990ICHnre1qgOXNoN432RLLT0VMc100wqrKdh21k&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=ebJjFMpXijqZjbZCcbF7yJIq2ES6jM0Q-DEcP-qjjeI&m=LIlANk-U7HCH8ndymom-KmuBXT0D9cV5FRnZUNw53lI&s=p5SYKSoDd8rQ34mfYf5b8Md0zG549LdE1ND11V2I1oA&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=ebJjFMpXijqZjbZCcbF7yJIq2ES6jM0Q-DEcP-qjjeI&m=LIlANk-U7HCH8ndymom-KmuBXT0D9cV5FRnZUNw53lI&s=p5SYKSoDd8rQ34mfYf5b8Md0zG549LdE1ND11V2I1oA&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=ebJjFMpXijqZjbZCcbF7yJIq2ES6jM0Q-DEcP-qjjeI&m=LIlANk-U7HCH8ndymom-KmuBXT0D9cV5FRnZUNw53lI&s=p5SYKSoDd8rQ34mfYf5b8Md0zG549LdE1ND11V2I1oA&e=>
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
