Looking better, I hacked my /etc/hosts on aai1 (tried the internal dns of the
dns vm – but aai2 did not resolve from aa1 without the temp workaround – until
I figure out the real value for this new entry in the env template for R1 – all
containers coming up – closing aai-513.
root@onap-aai-inst1:/opt/test-config# docker ps
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS
NAMES
603e85af586f nexus3.onap.org:10001/onap/model-loader
"/opt/app/model-lo..." About a minute ago Up About a minute
testconfig_model-loader_1
9826995b7ad5 nexus3.onap.org:10001/onap/data-router
"/opt/app/data-rou..." About a minute ago Up About a minute
0.0.0.0:9502->9502/tcp testconfig_datarouter_1
19dd8614b767 nexus3.onap.org:10001/onap/search-data-service
"/opt/app/search-d..." About a minute ago Up About a minute
0.0.0.0:9509->9509/tcp
testconfig_aai.searchservice.simpledemo.openecomp.org_1
89b93577733f nexus3.onap.org:10001/onap/sparky-be "/bin/sh
-c /opt/a..." About a minute ago Up About a minute 8000/tcp,
0.0.0.0:9517->9517/tcp testconfig_sparky-be_1
c13e604e1fdc aaionap/haproxy:1.1.0
"/docker-entrypoin..." About a minute ago Up About a minute
0.0.0.0:8443->8443/tcp testconfig_aai.api.simpledemo.openecomp.org_1
00aa79860bd5 nexus3.onap.org:10001/openecomp/aai-traversal "/bin/bash
/opt/ap..." 4 minutes ago Up 4 minutes 0.0.0.0:8446->8446/tcp,
8447/tcp testconfig_aai-traversal.api.simpledemo.openecomp.org_1
54747c3594fc nexus3.onap.org:10001/openecomp/aai-resources "/bin/bash
/opt/ap..." 7 minutes ago Up 7 minutes 0.0.0.0:8447->8447/tcp
testconfig_aai-resources.api.simpledemo.openecomp.org_1
rerunning aai1 install sh
Processing triggers for ureadahead (0.100.0-16) ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 617 0 617 0 0 3151 0 --:--:-- --:--:-- --:--:-- 3164
100 7857k 100 7857k 0 0 2180k 0 0:00:03 0:00:03 --:--:-- 2850k
docker stop/waiting
docker start/running, process 9460
Cloning into 'test-config'...
still can't ping - hardcoding to test
10.0.1.2 aai.hbase.simpledemo.openecomp.org
10.12.5.226 aai.hbase.simpledemo.openecomp.org
root@onap-aai-inst1:/opt/test-config# ./deploy_vm1.sh
Warning: Unable to find the volume directory /opt/aai/logroot/AAI-SEARCH so
creating it as regular directory
Warning: Unable to find the volume directory /opt/aai/logroot/AAI-DATA-ROUTER
so creating it as regular directory
Warning: Unable to find the volume directory /opt/aai/logroot/AAI-MODEL-LOADER
so creating it as regular directory
Login Succeeded
680d14a1d355: Pull complete
a48e250250b3: Pull complete
Digest: sha256:10b920dde66a0c61b153a569b7ca87ba531afeafd3f07c28a591f903d87c3ef1
Status: Downloaded newer image for nexus3.onap.org:10001/onap/data-router:v1.1.0
4f23d6fd6171: Pull complete
032366839203: Pull complete
Digest: sha256:8c850a34decf38c02f1d0e0976659cab0bc39a65275b0714bf9f207cdceff8c2
Status: Downloaded newer image for nexus3.onap.org:10001/onap/sparky-be:v1.1.0
WARNING: The USER_ID variable is not set. Defaulting to a blank string.
WARNING: The USER_ID variable is not set. Defaulting to a blank string.
No stopped containers
Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1'
deployment to finish ...
Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1'
deployment to finish ...
Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1'
deployment to finish ...
Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1'
deployment to finish ...
Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1'
deployment to finish ...
Waiting for 'testconfig_aai-resources.api.simpledemo.openecomp.org_1'
deployment to finish ...
looking better with /etc/hosts hack until I get a correct internal dns entry in
my env file
root@onap-aai-inst1:/opt/test-config# docker ps -a
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS
NAMES
54747c3594fc nexus3.onap.org:10001/openecomp/aai-resources "/bin/bash
/opt/ap..." About a minute ago Up About a minute 0.0.0.0:8447->8447/tcp
testconfig_aai-resources.api.simpledemo.openecomp.org_1
root@onap-aai-inst1:/opt/test-config# docker ps
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS
NAMES
54747c3594fc nexus3.onap.org:10001/openecomp/aai-resources "/bin/bash
/opt/ap..." About a minute ago Up About a minute 0.0.0.0:8447->8447/tcp
testconfig_aai-resources.api.simpledemo.openecomp.org_1
root@onap-aai-inst1:/opt/test-config#
From: [email protected]
[mailto:[email protected]] On Behalf Of Michael O'Brien
Sent: Monday, November 27, 2017 00:08
To: [email protected]
Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup
failures in HEAT - manual /opt/aai_install.sh fails but installs docker and
pulls images
Discussed with Venkata, Can I get someone who has deployed aai1 on heat to
validate my env settings – specifically my heat settings.
After I actually looked at the failed docker log as well as Venkata suggested
it does look like my DNS settings are to blame – I am having a hard time
finding the recommended settings on the wiki or the PPT. and would like to
document it as we go.
aai.hbase.simpledemo.openecomp.org: forward host lookup failed: Unknown host
Unable to connect to hbase after 200 retries, please check if hbase server is
properly configured and be able to connect
Unable to run init chef script, please check logs
Onap-openstack.env
dns_list: 8.8.4.4
external_dns: 8.8.4.4
dns_forwarder: 10.0.100.1 (was missing the dns-server ip)
testing this now – but my current restack stops the /opt file downloads before
the test-config folder is in where the deploy scripts are.
From:
[email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Michael O'Brien
Sent: Sunday, November 26, 2017 13:33
To: PLATANIA, MARCO (MARCO)
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup
failures in HEAT - manual /opt/aai_install.sh fails but installs docker and
pulls images
30 min later on a re-stack we get connectivity again to gerrit/nexus – when
manually bringing up the containers – will see if resources loads
Narrows down the image:version issue – as non-issue – was worried about the dns
setting in onap-openstack.env
https://wiki.onap.org/display/DW/Vetted+vFirewall+Demo+-+Full+draft+how-to+for+F2F+and+ReadTheDocs#VettedvFirewallDemo-Fulldrafthow-toforF2FandReadTheDocs-OpenlabVNCandCLI
– was using google’s 8.8.4.4 for the three dns* properties – need to vet these
on Monday.
root@onap-aai-inst1:/opt/test-config# ./deploy_vm1.sh
Login Succeeded
v1.1.0: Pulling from openecomp/aai-resources
c62795f78da9: Already exists
d4fceeeb758e: Already exists
5c9125a401ae: Already exists
0062f774e994: Already exists
6b33fd031fac: Already exists
9a115de3c55d: Already exists
55300a0e3724: Already exists
371557fd7d66: Downloading [=========================================> ]
137.4MB/165.8MB
From:
[email protected]<mailto:[email protected]>
[mailto:[email protected]] On Behalf Of Michael O'Brien
Sent: Sunday, November 26, 2017 13:04
To: PLATANIA, MARCO (MARCO)
<[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>; *_Amdocs DTV
CONV CORE PMO
<[email protected]<mailto:[email protected]>>
Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup
failures in HEAT - manual /opt/aai_install.sh fails but installs docker and
pulls images
Marco,
Yes, not related to HEAT or OOM deployment – looks like nexus3/gerritt
timeouts and moved/renamed docker images on nexus3 – as model-loader has
similar issues since the 16th in the branch.
Any ONAP deployment lately seems very fragile – as I have worked with a
fully working stack just last Wed on HEAT
I tried several timing related workarounds, rebooting, delete/recreate stack,
running parts of deploy_vm1.sh commented – also 2 of the 6 containers are
pointing to openecomp nexus namespace not the new onap – like 4 of the newer
services in aai1 – so the issue might be intermittent nexus3 access – which I
have seen in the past
In the other reference OOM deployment – all 9 containers come up ok – looking
at differences in image versions between HEAT and OOM
ubuntu@ip-172-31-31-252:~$ kubectl get pods --all-namespaces | grep aai
onap-aai aai-resources-898583818-dnw9l 2/2
Running 0 24m
onap-aai aai-service-3470292893-40dfh 1/1
Running 0 24m
onap-aai aai-traversal-2843093465-sqgn5 2/2
Running 0 24m
onap-aai data-router-2254557428-5z2sm 1/1
Running 0 24m
onap-aai elasticsearch-622738319-88x11 1/1
Running 0 24m
onap-aai hbase-1949550546-fxl7f 1/1
Running 0 24m
onap-aai model-loader-service-1260517459-3wqmq 2/2
Running 0 24m
onap-aai search-data-service-2243336541-btbdh 2/2
Running 0 24m
onap-aai sparky-be-1292339831-xhmpx 2/2
Running 0 24m
Also note that the unrelated model-loader builds have been failing since the
16th – so likely this is a nexus branching issue (images moving over to 1.1.0)
https://jira.onap.org/browse/AAI-511
Started getting refused connections later today to gerrit
see
root@onap-aai-inst1:/opt/test-config# cat deploy_vm1.sh
RESOURCES_CONTAINER_NAME=$($DOCKER_COMPOSE_CMD up -d
aai-resources.api.simpledemo.openecomp.org 2>&1 | grep 'Creating' | grep -v
'volume' | grep -v 'network' | awk '{ print $2; }' | head -1);
wait_for_container $RESOURCES_CONTAINER_NAME '0.0.0.0:8447';
intermittent in open-lab
Started also failing on contacting gerrit
root@onap-aai-inst1:/opt# ./aai_vm_init.sh
fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/': Could not
resolve host: gerrit.onap.org
root@onap-aai-inst1:/opt# ./aai_vm_init.sh
fatal: unable to access 'http://gerrit.onap.org/r/aai/test-config/': Could not
resolve host: gerrit.onap.org
Error response from daemon: Get https://nexus3.onap.org:10001/v1/users/: dial
tcp: lookup nexus3.onap.org on 10.0.100.1:53: read udp
10.0.1.1:34203->10.0.100.1:53: read: connection refused
repos working
https://gerrit.onap.org/r/#/admin/projects/aai/test-config
From: PLATANIA, MARCO (MARCO) [mailto:[email protected]]
Sent: Sunday, November 26, 2017 11:39
To: Michael O'Brien <[email protected]<mailto:[email protected]>>;
[email protected]<mailto:[email protected]>
Subject: Re: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup
failures in HEAT - manual /opt/aai_install.sh fails but installs docker and
pulls images
Michael,
We saw similar intermittent failures in cloud init. This is not related to heat
in particular, but I agree that a component installation is incomplete if cloud
init fails. We had to re-run the install script to bring up an affected
component.
Marco
From:
<[email protected]<mailto:[email protected]>>
on behalf of Michael O'Brien
<[email protected]<mailto:[email protected]>>
Date: Sunday, November 26, 2017 at 1:13 AM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: [onap-discuss] R1 Blitz: Intermittent AAI-vm1 container startup
failures in HEAT - manual /opt/aai_install.sh fails but installs docker and
pulls images
Team,
I am experiencing the aai-vm1 startup errors we previously saw last week in
HEAT. I deployed the latest yaml to open-lab and am having an issue manually
bringing up the 6 containers in aai1 – specifically the resources one again. I
understand there may be a set of scripts developed to work around the
intermittent issue where vm1 is deployed before vm2.
After 1 hour aai1 had not even pulled docker images until I manually started
the install scripts off the /opt dir – shouldn’t these normally be bootstrapped?
Aai-vm2’s hbase and es are up fine.
It is an intermittent issue because I have seen deployments from last week
where AAI fully works. The current workaround is to delete/recreate the entire
HEAT stack.
Let us know if there is another workaround.
https://jira.onap.org/browse/AAI-513<https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.onap.org_browse_AAI-2D513&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=RCSGr1kIZ6BqOK6Fre_hPnaJFiCVvFEKo8z4JavSqC8&s=uhcYhh0UeGZVza9c1ds7d94RfwZJynfvvou2-CxuZZY&e=>
/michael
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at
https://www.amdocs.com/about/email-disclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amdocs.com_about_email-2Ddisclaimer&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=KgFIQiUJzSC0gUhJaQxg8eC3w16GC3sKgWIcs4iIee0&m=RCSGr1kIZ6BqOK6Fre_hPnaJFiCVvFEKo8z4JavSqC8&s=X8u990ICHnre1qgOXNoN432RLLT0VMc100wqrKdh21k&e=>
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at https://www.amdocs.com/about/email-disclaimer
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at https://www.amdocs.com/about/email-disclaimer
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at https://www.amdocs.com/about/email-disclaimer
This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,
you may review at https://www.amdocs.com/about/email-disclaimer
<https://www.amdocs.com/about/email-disclaimer>
_______________________________________________
onap-discuss mailing list
[email protected]
https://lists.onap.org/mailman/listinfo/onap-discuss
