Team, Since the removal of the IPs/Ports across the intel openlab 2 days ago, I am having an issue near the end of bringing up DCAE to return my environment back to working state. I am running out of IPs for my private network. However if I manually allocate IPs we are good. The cloudify blueprint manages to successfully associate with the first 5 VMs but is failing on the last 8. In the past this was due to left over nodes or zones - these are all deleted prior to bringing up dcae-bootstrap. Just wondering if any other tenant is having this issue for the last 2 days.
2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap06_02e83.create] Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network e39327a5-76c0-409a-b4e4-6ccf047c4630. [status_code=409] 2018-02-16T04:38:12 CFY <cdap7> 'install' workflow execution failed: RuntimeError: Workflow failed: Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network e39327a5-76c0-409a-b4e4-6ccf047c4630. [status_code=409] I was not getting this behaviour before the openlab router deletion issue 2 days ago https://jira.onap.org/browse/OPENLABS-175 ubuntu@dcae-dcae-bootstrap:~$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 59524098086d nexus3.onap.org:10001/onap/org.onap.dcaegen2.deployments.bootstrap:v1.1.1 "/bin/sh -c 'exec ..." 33 minutes ago Up 33 minutes boot ubuntu@dcae-dcae-bootstrap:~$ sudo docker logs -f boot Finished executing workflow install on deployment DockerComponent * Run 'cfy events list --include-logs --execution-id b899c896-6353-4efb-ab64-a64041fd1e72' to retrieve the execution's events/logs + cfy install -p ./blueprints/cdap/cdapbp7.yaml -b cdapbp7 -d cdap7 -i ../config/cdapinputs.yaml -i location_id=4R4x Uploading blueprint ./blueprints/cdap/cdapbp7.yaml... Blueprint uploaded. The blueprint's id is cdapbp7 Processing inputs source: ../config/cdapinputs.yaml Processing inputs source: location_id=4R4x Creating new deployment from blueprint cdapbp7... Deployment created. The deployment's id is cdap7 Executing workflow install on deployment cdap7 [timeout=900 seconds] Deployment environment creation is in progress... 2018-02-16T04:37:55 CFY <cdap7> Starting 'create_deployment_environment' workflow execution 2018-02-16T04:37:55 CFY <cdap7> Installing deployment plugins 2018-02-16T04:37:55 CFY <cdap7> Sending task 'cloudify_agent.operations.install_plugins' 2018-02-16T04:37:55 CFY <cdap7> Task started 'cloudify_agent.operations.install_plugins' 2018-02-16T04:37:56 CFY <cdap7> Task succeeded 'cloudify_agent.operations.install_plugins' 2018-02-16T04:37:56 CFY <cdap7> Skipping starting deployment policy engine core - no policies defined 2018-02-16T04:37:56 CFY <cdap7> Creating deployment work directory 2018-02-16T04:37:57 CFY <cdap7> 'create_deployment_environment' workflow execution succeeded 2018-02-16T04:38:02 CFY <cdap7> Starting 'install' workflow execution 2018-02-16T04:38:02 CFY <cdap7> [key_pair_df6d4] Creating node 2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap04_2cfed] Creating node 2018-02-16T04:38:02 CFY <cdap7> [private_net_d15fa] Creating node 2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap03_4808a] Creating node 2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap04_2cfed.create] Sending task 'neutron_plugin.floatingip.create' 2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap06_81dc1] Creating node 2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap03_4808a.create] Sending task 'neutron_plugin.floatingip.create' 2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap05_17385] Creating node 2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap04_2cfed.create] Task started 'neutron_plugin.floatingip.create' 2018-02-16T04:38:02 CFY <cdap7> [floatingip_cdap03_4808a.create] Task started 'neutron_plugin.floatingip.create' 2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap06_81dc1.create] Sending task 'neutron_plugin.floatingip.create' 2018-02-16T04:38:03 CFY <cdap7> [key_pair_df6d4.create] Sending task 'nova_plugin.keypair.create' 2018-02-16T04:38:03 CFY <cdap7> [private_net_d15fa.create] Sending task 'neutron_plugin.network.create' 2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap00_89e4a] Creating node 2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap06_81dc1.create] Task started 'neutron_plugin.floatingip.create' 2018-02-16T04:38:03 CFY <cdap7> [security_group_3e25d] Creating node 2018-02-16T04:38:03 CFY <cdap7> [key_pair_df6d4.create] Task started 'nova_plugin.keypair.create' 2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap05_17385.create] Sending task 'neutron_plugin.floatingip.create' 2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap01_5452d] Creating node 2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap00_89e4a.create] Sending task 'neutron_plugin.floatingip.create' 2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap02_c05d6] Creating node 2018-02-16T04:38:03 CFY <cdap7> [private_net_d15fa.create] Task started 'neutron_plugin.network.create' 2018-02-16T04:38:03 CFY <cdap7> [sharedsshkey_cdap_9efda] Creating node 2018-02-16T04:38:03 CFY <cdap7> [sharedsshkey_cdap_9efda.create] Sending task 'sshkeyshare.keyshare_plugin.generate' 2018-02-16T04:38:03 CFY <cdap7> [security_group_3e25d.create] Sending task 'neutron_plugin.security_group.create' 2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap02_c05d6.create] Sending task 'neutron_plugin.floatingip.create' 2018-02-16T04:38:03 CFY <cdap7> [floatingip_cdap01_5452d.create] Sending task 'neutron_plugin.floatingip.create' 2018-02-16T04:38:05 CFY <cdap7> [key_pair_df6d4.create] Task succeeded 'nova_plugin.keypair.create' 2018-02-16T04:38:05 CFY <cdap7> [floatingip_cdap05_17385.create] Task started 'neutron_plugin.floatingip.create' 2018-02-16T04:38:05 CFY <cdap7> [private_net_d15fa.create] Task succeeded 'neutron_plugin.network.create' 2018-02-16T04:38:05 CFY <cdap7> [floatingip_cdap00_89e4a.create] Task started 'neutron_plugin.floatingip.create' 2018-02-16T04:38:05 CFY <cdap7> [key_pair_df6d4] Configuring node 2018-02-16T04:38:05 CFY <cdap7> [floatingip_cdap04_2cfed.create] Task succeeded 'neutron_plugin.floatingip.create' 2018-02-16T04:38:05 CFY <cdap7> [sharedsshkey_cdap_9efda.create] Task started 'sshkeyshare.keyshare_plugin.generate' 2018-02-16T04:38:06 CFY <cdap7> [private_net_d15fa] Configuring node 2018-02-16T04:38:06 CFY <cdap7> [key_pair_df6d4] Starting node 2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap04_2cfed] Configuring node 2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap06_81dc1.create] Task succeeded 'neutron_plugin.floatingip.create' 2018-02-16T04:38:06 CFY <cdap7> [security_group_3e25d.create] Task started 'neutron_plugin.security_group.create' 2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap03_4808a.create] Task succeeded 'neutron_plugin.floatingip.create' 2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap02_c05d6.create] Task started 'neutron_plugin.floatingip.create' 2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap06_81dc1] Configuring node 2018-02-16T04:38:06 CFY <cdap7> [private_net_d15fa] Starting node 2018-02-16T04:38:06 CFY <cdap7> [floatingip_cdap04_2cfed] Starting node 2018-02-16T04:38:07 CFY <cdap7> [floatingip_cdap03_4808a] Configuring node 2018-02-16T04:38:07 CFY <cdap7> [floatingip_cdap06_81dc1] Starting node 2018-02-16T04:38:07 CFY <cdap7> [sharedsshkey_cdap_9efda.create] Task succeeded 'sshkeyshare.keyshare_plugin.generate' 2018-02-16T04:38:07 CFY <cdap7> [floatingip_cdap01_5452d.create] Task started 'neutron_plugin.floatingip.create' 2018-02-16T04:38:07 CFY <cdap7> [dns_cdap04_c80a9] Creating node 2018-02-16T04:38:07 CFY <cdap7> [dns_cdap04_c80a9.create] Sending task 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:07 CFY <cdap7> [fixedip_cdap00_2934e] Creating node 2018-02-16T04:38:07 CFY <cdap7> [floatingip_cdap05_17385.create] Task succeeded 'neutron_plugin.floatingip.create' 2018-02-16T04:38:07 CFY <cdap7> [dns_cdap04_c80a9.create] Task started 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:07 CFY <cdap7> [fixedip_cdap02_9697b] Creating node 2018-02-16T04:38:07 CFY <cdap7> [fixedip_cdap01_7f78b] Creating node 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap04_42278] Creating node 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap05_433fb] Creating node 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap01_7f78b.create] Sending task 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap03_4808a] Starting node 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap02_9697b.create] Sending task 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap00_89e4a.create] Task succeeded 'neutron_plugin.floatingip.create' 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap01_7f78b.create] Task started 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap03_1c4c2] Creating node 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap00_2934e.create] Sending task 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap04_42278.create] Sending task 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap05_433fb.create] Sending task 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap06_02e83] Creating node 2018-02-16T04:38:08 CFY <cdap7> [security_group_3e25d.create] Task succeeded 'neutron_plugin.security_group.create' 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap02_9697b.create] Task started 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap03_1c4c2.create] Sending task 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [sharedsshkey_cdap_9efda] Configuring node 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap06_02e83.create] Sending task 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap05_17385] Configuring node 2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap00_89e4a] Configuring node 2018-02-16T04:38:08 CFY <cdap7> [dns_cdap06_2d407] Creating node 2018-02-16T04:38:08 CFY <cdap7> [security_group_3e25d] Configuring node 2018-02-16T04:38:08 CFY <cdap7> [floatingip_cdap02_c05d6.create] Task succeeded 'neutron_plugin.floatingip.create' 2018-02-16T04:38:08 CFY <cdap7> [fixedip_cdap00_2934e.create] Task started 'neutron_plugin.port.create' 2018-02-16T04:38:08 CFY <cdap7> [dns_cdap06_2d407.create] Sending task 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:09 CFY <cdap7> [sharedsshkey_cdap_9efda] Starting node 2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap00_89e4a] Starting node 2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap05_17385] Starting node 2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap02_c05d6] Configuring node 2018-02-16T04:38:09 CFY <cdap7> [dns_cdap03_f3c7d] Creating node 2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap01_5452d.create] Task succeeded 'neutron_plugin.floatingip.create' 2018-02-16T04:38:09 CFY <cdap7> [dns_cdap03_f3c7d.create] Sending task 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:09 CFY <cdap7> [security_group_3e25d] Starting node 2018-02-16T04:38:09 CFY <cdap7> [fixedip_cdap04_42278.create] Task started 'neutron_plugin.port.create' 2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap01_5452d] Configuring node 2018-02-16T04:38:09 CFY <cdap7> [floatingip_cdap02_c05d6] Starting node 2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap01_7f78b.create] Task succeeded 'neutron_plugin.port.create' 2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap05_433fb.create] Task started 'neutron_plugin.port.create' 2018-02-16T04:38:10 CFY <cdap7> [dns_cdap00_e22b0.create] Sending task 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:10 CFY <cdap7> [dns_cdap00_e22b0] Creating node 2018-02-16T04:38:10 CFY <cdap7> [dns_cdap05_9f71a] Creating node 2018-02-16T04:38:10 CFY <cdap7> [floatingip_cdap01_5452d] Starting node 2018-02-16T04:38:10 CFY <cdap7> [dns_cdap04_c80a9.create] Task succeeded 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap03_1c4c2.create] Task started 'neutron_plugin.port.create' 2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap06_02e83.create] Task started 'neutron_plugin.port.create' 2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap02_9697b.create] Task succeeded 'neutron_plugin.port.create' 2018-02-16T04:38:10 CFY <cdap7> [dns_cdap05_9f71a.create] Sending task 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap01_7f78b] Configuring node 2018-02-16T04:38:10 CFY <cdap7> [dns_cdap06_2d407.create] Task started 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:10 CFY <cdap7> [fixedip_cdap00_2934e.create] Task succeeded 'neutron_plugin.port.create' 2018-02-16T04:38:10 CFY <cdap7> [dns_cdap02_a5bef] Creating node 2018-02-16T04:38:11 CFY <cdap7> [dns_cdap04_c80a9] Configuring node 2018-02-16T04:38:11 CFY <cdap7> [dns_cdap02_a5bef.create] Sending task 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap02_9697b] Configuring node 2018-02-16T04:38:11 CFY <cdap7> [dns_cdap01_65c42] Creating node 2018-02-16T04:38:11 CFY <cdap7> [dns_cdap03_f3c7d.create] Task started 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap04_42278.create] Task succeeded 'neutron_plugin.port.create' 2018-02-16T04:38:11 CFY <cdap7> [dns_cdap01_65c42.create] Sending task 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap01_7f78b] Starting node 2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap00_2934e] Configuring node 2018-02-16T04:38:11 CFY <cdap7> [dns_cdap04_c80a9] Starting node 2018-02-16T04:38:11 CFY <cdap7> [fixedip_cdap02_9697b] Starting node 2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap04_42278] Configuring node 2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap05_433fb.create] Task succeeded 'neutron_plugin.port.create' 2018-02-16T04:38:12 CFY <cdap7> [dns_cdap00_e22b0.create] Task started 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap00_2934e] Starting node 2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap04_42278] Starting node 2018-02-16T04:38:12 CFY <cdap7> [dns_cdap02_a5bef.create] Task started 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:12 CFY <cdap7> [dns_cdap05_9f71a.create] Task started 'dnsdesig.dns_plugin.aneeded' 2018-02-16T04:38:12 CFY <cdap7> [fixedip_cdap06_02e83.create] Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network e39327a5-76c0-409a-b4e4-6ccf047c4630. [status_code=409] 2018-02-16T04:38:12 CFY <cdap7> 'install' workflow execution failed: RuntimeError: Workflow failed: Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network e39327a5-76c0-409a-b4e4-6ccf047c4630. [status_code=409] Execution of workflow install for deployment cdap7 failed. [error=Traceback (most recent call last): File "/tmp/pip-build-U0y0vb/cloudify-plugins-common/cloudify/dispatch.py", line 472, in _remote_workflow_child_thread File "/tmp/pip-build-U0y0vb/cloudify-plugins-common/cloudify/dispatch.py", line 504, in _execute_workflow_function File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/plugins/workflows.py", line 27, in install node_instances=set(ctx.node_instances)) File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/plugins/lifecycle.py", line 28, in install_node_instances processor.install() File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/plugins/lifecycle.py", line 83, in install graph_finisher_func=self._finish_install) File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/plugins/lifecycle.py", line 103, in _process_node_instances self.graph.execute() File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks_graph.py", line 133, in execute self._handle_terminated_task(task) File "/opt/mgmtworker/env/lib/python2.7/site-packages/cloudify/workflows/tasks_graph.py", line 207, in _handle_terminated_task raise RuntimeError(message) RuntimeError: Workflow failed: Task failed 'neutron_plugin.port.create' -> No more IP addresses available on network e39327a5-76c0-409a-b4e4-6ccf047c4630. [status_code=409] ] * Run 'cfy events list --include-logs --execution-id 7d34383e-4558-4475-9ddf-e3aeabb5a665' to retrieve the execution's events/logs + cfy install -p ./blueprints/cbs/config_binding_service.yaml -b config_binding_service -d config_binding_service -i location_id=4R4x Uploading blueprint ./blueprints/cbs/config_binding_service.yaml... Blueprint uploaded. The blueprint's id is config_binding_service Processing inputs source: location_id=4R4x Creating new deployment from blueprint config_binding_service... Deployment created. The deployment's id is config_binding_service Executing workflow install on deployment config_binding_service [timeout=900 seconds] Deployment environment creation is in progress... 2018-02-16T04:38:22 CFY <config_binding_service> Starting 'create_deployment_environment' workflow execution 2018-02-16T04:38:22 CFY <config_binding_service> Installing deployment plugins 2018-02-16T04:38:22 CFY <config_binding_service> Sending task 'cloudify_agent.operations.install_plugins' 2018-02-16T04:38:22 CFY <config_binding_service> Task started 'cloudify_agent.operations.install_plugins' 2018-02-16T04:38:23 CFY <config_binding_service> Task succeeded 'cloudify_agent.operations.install_plugins' 2018-02-16T04:38:23 CFY <config_binding_service> Skipping starting deployment policy engine core - no policies defined 2018-02-16T04:38:23 CFY <config_binding_service> Creating deployment work directory 2018-02-16T04:38:24 CFY <config_binding_service> 'create_deployment_environment' workflow execution succeeded * * Attachments Drop files to attach, or browse. * Add Link<https://jira.onap.org/secure/LinkJiraIssue!default.jspa?id=19561> Issue Links Michael O'Brien Amdocs Technology 16135955268 55268 [amdocs-a] This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer <https://www.amdocs.com/about/email-disclaimer>
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
