You probably should log into the dns server and make sure bind loaded the zone files for both openecomp.org and onap.org. The zone files are created from the .env parameters so if your overall ONAP install is not correct there can be blank lines in the zone file which will make it fail to load.
nslookup simpledemo.openecomp.org is the command I use to check that the internal dns is functioning. We have both zone files so that we can transition to onap.org like we have for the portal apps. Brian root@vm1-dns-server:~# nslookup simpledemo.openecomp.org @10.0.100.1 nslookup: couldn't get address for '@10.0.100.1': not found root@vm1-dns-server:~# nslookup simpledemo.openecomp.org Server: 10.0.100.1 Address: 10.0.100.1#53 Name: simpledemo.openecomp.org Address: 10.0.100.1 root@vm1-dns-server:~# nslookup simpledemo.onap.org Server: 10.0.100.1 Address: 10.0.100.1#53 Name: simpledemo.onap.org Address: 10.0.100.1 From: onap-discuss-boun...@lists.onap.org [mailto:onap-discuss-boun...@lists.onap.org] On Behalf Of BARSKY, GEORA Sent: Monday, November 27, 2017 9:46 AM To: OBRIEN, FRANK MICHAEL <frank.obr...@amdocs.com>; onap-discuss@lists.onap.org Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Hi Michael, We had the same issue in Logging tenant, aa2 was not reachable . Restarting DNS server itself didn’t help, but full re-deploy apparently worked. These are the setting from env file: dns_list: ["10.12.25.5", "8.8.8.8"] external_dns: 8.8.8.8 dns_forwarder: 10.12.25.5 oam_network_cidr: 10.0.0.0/16 dns_ip_addr: 10.0.100.1 Geora Barsky Amdocs Technology 647-688-1039 [amdocs-a] Follow us on Facebook<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_amdocs_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=Q2rMIPEBhTZPnBcXvUx1R3l8kT2hkthGbBn-KXgIGT4&e=>, Twitter<https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_amdocs&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=T2MHsJbZAZwByOWY9SlKoo5BOUVvUq5D2If6wbLcFiQ&e=>, LinkedIn<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.linkedin.com_company_amdocs&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=IL8snTnKLapyPVq25RKIdQb8f2qvyta1sBv2j4dMZMs&e=>, YouTube<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.youtube.com_amdocsinc&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=zMZgweM77G9ZaKcoyQrD9Bvh0VBiVJGctysTEhIiCAA&e=>, Google+<https://urldefense.proofpoint.com/v2/url?u=https-3A__plus.google.com_105657940751678445194&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=7E4KA7a-NGx4hsJ3KSZ2DNtQKnYpzC_ekVjOcdcI-1E&e=> and the Amdocs blog network<https://urldefense.proofpoint.com/v2/url?u=http-3A__blogs.amdocs.com_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=V3Q7LcPk7h1tmpVL8tcaJ48fHW7MFZFodv6kSj8gfhI&e=>. From: onap-discuss-boun...@lists.onap.org<mailto:onap-discuss-boun...@lists.onap.org> [mailto:onap-discuss-boun...@lists.onap.org] On Behalf Of Michael O'Brien Sent: Monday, November 27, 2017 12:08 AM To: onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org> Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Discussed with Venkata, Can I get someone who has deployed aai1 on heat to validate my env settings – specifically my heat settings. After I actually looked at the failed docker log as well as Venkata suggested it does look like my DNS settings are to blame – I am having a hard time finding the recommended settings on the wiki or the PPT. and would like to document it as we go. aai.hbase.simpledemo.openecomp.org: forward host lookup failed: Unknown host Unable to connect to hbase after 200 retries, please check if hbase server is properly configured and be able to connect Unable to run init chef script, please check logs Onap-openstack.env dns_list: 8.8.4.4 external_dns: 8.8.4.4 dns_forwarder: 10.0.100.1 (was missing the dns-server ip) testing this now – but my current restack stops the /opt file downloads before the test-config folder is in where the deploy scripts are. From: onap-discuss-boun...@lists.onap.org<mailto:onap-discuss-boun...@lists.onap.org> [mailto:onap-discuss-boun...@lists.onap.org] On Behalf Of Michael O'Brien Sent: Sunday, November 26, 2017 13:33 To: PLATANIA, MARCO (MARCO) <plata...@research.att.com<mailto:plata...@research.att.com>>; onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org> Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images 30 min later on a re-stack we get connectivity again to gerrit/nexus – when manually bringing up the containers – will see if resources loads Narrows down the image:version issue – as non-issue – was worried about the dns setting in onap-openstack.env https://wiki.onap.org/display/DW/Vetted+vFirewall+Demo+-+Full+draft+how-to+for+F2F+and+ReadTheDocs#VettedvFirewallDemo-Fulldrafthow-toforF2FandReadTheDocs-OpenlabVNCandCLI<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_Vetted-2BvFirewall-2BDemo-2B-2D-2BFull-2Bdraft-2Bhow-2Dto-2Bfor-2BF2F-2Band-2BReadTheDocs-23VettedvFirewallDemo-2DFulldrafthow-2DtoforF2FandReadTheDocs-2DOpenlabVNCandCLI&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=zAY7RKb_X8Cj1gHwhYpStqAzuAkvB7Wh7sBzSW9JpkA&e=> – was using google’s 8.8.4.4 for the three dns* properties – need to vet these on Monday. root@onap-aai-inst1:/opt/test-config# ./deploy_vm1.sh Login Succeeded v1.1.0: Pulling from openecomp/aai-resources c62795f78da9: Already exists d4fceeeb758e: Already exists 5c9125a401ae: Already exists 0062f774e994: Already exists 6b33fd031fac: Already exists 9a115de3c55d: Already exists 55300a0e3724: Already exists 371557fd7d66: Downloading [=========================================> ] 137.4MB/165.8MB From: onap-discuss-boun...@lists.onap.org<mailto:onap-discuss-boun...@lists.onap.org> [mailto:onap-discuss-boun...@lists.onap.org] On Behalf Of Michael O'Brien Sent: Sunday, November 26, 2017 13:04 To: PLATANIA, MARCO (MARCO) <plata...@research.att.com<mailto:plata...@research.att.com>>; onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>; *_Amdocs DTV CONV CORE PMO <_amdocsdtvconvcore...@int.amdocs.com<mailto:_amdocsdtvconvcore...@int.amdocs.com>> Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Marco, Yes, not related to HEAT or OOM deployment – looks like nexus3/gerritt timeouts and moved/renamed docker images on nexus3 – as model-loader has similar issues since the 16th in the branch. Any ONAP deployment lately seems very fragile – as I have worked with a fully working stack just last Wed on HEAT I tried several timing related workarounds, rebooting, delete/recreate stack, running parts of deploy_vm1.sh commented – also 2 of the 6 containers are pointing to openecomp nexus namespace not the new onap – like 4 of the newer services in aai1 – so the issue might be intermittent nexus3 access – which I have seen in the past In the other reference OOM deployment – all 9 containers come up ok – looking at differences in image versions between HEAT and OOM ubuntu@ip-172-31-31-252:~$ kubectl get pods --all-namespaces | grep aai onap-aai aai-resources-898583818-dnw9l 2/2 Running 0 24m onap-aai aai-service-3470292893-40dfh 1/1 Running 0 24m onap-aai aai-traversal-2843093465-sqgn5 2/2 Running 0 24m onap-aai data-router-2254557428-5z2sm 1/1 Running 0 24m onap-aai elasticsearch-622738319-88x11 1/1 Running 0 24m onap-aai hbase-1949550546-fxl7f 1/1 Running 0 24m onap-aai model-loader-service-1260517459-3wqmq 2/2 Running 0 24m onap-aai search-data-service-2243336541-btbdh 2/2 Running 0 24m onap-aai sparky-be-1292339831-xhmpx 2/2 Running 0 24m Also note that the unrelated model-loader builds have been failing since the 16th – so likely this is a nexus branching issue (images moving over to 1.1.0) https://jira.onap.org/browse/AAI-511<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_AAI-2D511&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=yIP0yYWJFUTNJ0uC7lVx5gY5JlvdJ1GhCtU0YIUhzSo&e=> Started getting refused connections later today to gerrit see root@onap-aai-inst1:/opt/test-config# cat deploy_vm1.sh RESOURCES_CONTAINER_NAME=$($DOCKER_COMPOSE_CMD up -d aai-resources.api.simpledemo.openecomp.org 2>&1 | grep 'Creating' | grep -v 'volume' | grep -v 'network' | awk '{ print $2; }' | head -1); wait_for_container $RESOURCES_CONTAINER_NAME '0.0.0.0:8447'; intermittent in open-lab Started also failing on contacting gerrit root@onap-aai-inst1:/opt# ./aai_vm_init.sh fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/':<https://urldefense.proofpoint.com/v2/url?u=http-3A__gerrit.onap.org_r_aai_test-2Dconfig_-27-3A&d=DwQGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=NLBQMIm9-R4e0JGj1KrJ-aQs_yivkliqjLOTafGA1rQ&e=> Could not resolve host: gerrit.onap.org root@onap-aai-inst1:/opt# ./aai_vm_init.sh fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/':<https://urldefense.proofpoint.com/v2/url?u=http-3A__gerrit.onap.org_r_aai_test-2Dconfig_-27-3A&d=DwQGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=NLBQMIm9-R4e0JGj1KrJ-aQs_yivkliqjLOTafGA1rQ&e=> Could not resolve host: gerrit.onap.org Error response from daemon: Get https://nexus3.onap.org:10001/v1/users/<https://urldefense.proofpoint.com/v2/url?u=https-3A__nexus3.onap.org-3A10001_v1_users_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=3ELA_tlqsBWkDJRYW02XY9z7w5GKluTz2Swso81-IOg&e=>: dial tcp: lookup nexus3.onap.org on 10.0.100.1:53: read udp 10.0.1.1:34203->10.0.100.1:53: read: connection refused repos working https://gerrit.onap.org/r/#/admin/projects/aai/test-config<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.onap.org_r_-23_admin_projects_aai_test-2Dconfig&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=qmHqTUO5BzSEAMdtQq8SuFjJy5X3pkRMpPEmDYu-dgM&e=> From: PLATANIA, MARCO (MARCO) [mailto:plata...@research.att.com] Sent: Sunday, November 26, 2017 11:39 To: Michael O'Brien <frank.obr...@amdocs.com<mailto:frank.obr...@amdocs.com>>; onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org> Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Michael, We saw similar intermittent failures in cloud init. This is not related to heat in particular, but I agree that a component installation is incomplete if cloud init fails. We had to re-run the install script to bring up an affected component. Marco From: <onap-discuss-boun...@lists.onap.org<mailto:onap-discuss-boun...@lists.onap.org>> on behalf of Michael O'Brien <frank.obr...@amdocs.com<mailto:frank.obr...@amdocs.com>> Date: Sunday, November 26, 2017 at 1:13 AM To: "onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>" <onap-discuss@lists.onap.org<mailto:onap-discuss@lists.onap.org>> Subject: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup failures in HEAT - manual /opt/aai_install.sh fails but installs docker and pulls images Team, I am experiencing the aai-vm1 startup errors we previously saw last week in HEAT. I deployed the latest yaml to open-lab and am having an issue manually bringing up the 6 containers in aai1 – specifically the resources one again. I understand there may be a set of scripts developed to work around the intermittent issue where vm1 is deployed before vm2. After 1 hour aai1 had not even pulled docker images until I manually started the install scripts off the /opt dir – shouldn’t these normally be bootstrapped? Aai-vm2’s hbase and es are up fine. It is an intermittent issue because I have seen deployments from last week where AAI fully works. The current workaround is to delete/recreate the entire HEAT stack. Let us know if there is another workaround. https://jira.onap.org/browse/AAI-513<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_AAI-2D513&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=RCSGr1kIZ6BqOK6Fre_hPnaJFiCVvFEKo8z4JavSqC8&s=uhcYhh0UeGZVza9c1ds7d94RfwZJynfvvou2-CxuZZY&e=> /michael This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=RCSGr1kIZ6BqOK6Fre_hPnaJFiCVvFEKo8z4JavSqC8&s=X8u990ICHnre1qgOXNoN432RLLT0VMc100wqrKdh21k&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=0Y6invrR9-WgC1jnIY67N3D_eqpyleViymKtECCE2xQ&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=0Y6invrR9-WgC1jnIY67N3D_eqpyleViymKtECCE2xQ&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=0Y6invrR9-WgC1jnIY67N3D_eqpyleViymKtECCE2xQ&e=> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Xy-7Md8IGmsdCNcJv5DpjXQLOniyt6StK-7zt9Fvuq8&s=0Y6invrR9-WgC1jnIY67N3D_eqpyleViymKtECCE2xQ&e=>
_______________________________________________ onap-discuss mailing list onap-discuss@lists.onap.org https://lists.onap.org/mailman/listinfo/onap-discuss
