Public bug reported:
Neutron tries to bind port on compute where instance is launched. It
doesn't make sense when hypervisor_type is ironic, since VM does not
live on hypervisor in this case. Furthermore it leads to failed
provisioning of baremetal node, when neutron is not configured on ironic
compute node.
When binding fails, nova starts cleanup and then throws exception. As
part of cleanup ironic port is deleted. This should not be the case when
virt-driver being used is Ironic.
Nova versions: liberty on-wards.
Setup:
node-1: controller with neutron
node-2: ironic-compute without neutron/neutron agents
Register a BM node and perform nova boot.
>From Nova:
=============================================================
nova-compute.log:21147:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
[-] Instance failed network setup after 1 attempt(s)
nova-compute.log:21148:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
Traceback (most recent call last):
nova-compute.log:21149:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 1564, in _allocate_network_async
nova-compute.log:21150:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
dhcp_options=dhcp_options)
nova-compute.log:21151:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 727, in allocate_for_instance
nova-compute.log:21152:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
self._delete_ports(neutron, instance, created_port_ids)
nova-compute.log:21153:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 204, in __exit__
nova-compute.log:21154:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
six.reraise(self.type_, self.value, self.tb)
nova-compute.log:21155:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 719, in allocate_for_instance
nova-compute.log:21156:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
security_group_ids, available_macs, dhcp_opts)
nova-compute.log:21157:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 342, in _create_port
nova-compute.log:21158:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
raise exception.PortBindingFailed(port_id=port_id)
nova-compute.log:21159:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
PortBindingFailed: Binding failed for port
0514b7e8-0408-4d5e-9b48-8292e686494f, please check neutron logs for more
information.
nova-compute.log:21160:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
2015-12-05 02:57:43.375 7649 DEBUG nova.virt.ironic.driver
[req-25ba4c81-4333-43e4-b8d9-e460827435e0 a989fe7fc89e4825b986666d7e6584cc
1308f1a382ef43c79eeed0ebf8a9db3b] unplug:
instance_uuid=d9295d4d-1104-47ab-8e75-5b30d0a3838b vif=[] _unp
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager
[req-25ba4c81-4333-43e4-b8d9-e460827435e0 a989fe7fc89e4825b986666d7e6584cc
1308f1a382ef43c79eeed0ebf8a9db3b] [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] Instance failed to spawn
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] Traceback (most recent call last):
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 2157, in _build_resources
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] yield resources
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 2011, in _build_and_run_inst
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] block_device_info=block_device_info)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/virt/ironic/driver.py",
line 791, in spawn
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] flavor=flavor)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 204, in __exit__
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] six.reraise(self.type_, self.value,
self.tb)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/virt/ironic/driver.py",
line 782, in spawn
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] self._plug_vifs(node, instance,
network_info)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/virt/ironic/driver.py",
line 1068, in _plug_vifs
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] network_info_str = str(network_info)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/model.py",
line 515, in __str__
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] return self._sync_wrapper(fn, *args,
**kwargs)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/model.py",
line 498, in _sync_wrapper
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] self.wait()
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/model.py",
line 530, in wait
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] self[:] = self._gt.wait()
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/eventlet/greenthread.py",
line 175, in wait
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] return self._exit_event.wait()
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/eventlet/event.py",
line 125, in wait
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] current.throw(*self._exc)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/eventlet/greenthread.py",
line 214, in main
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] result = function(*args, **kwargs)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 1581, in _allocate_network_a
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] six.reraise(*exc_info)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 1564, in _allocate_network_a
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] dhcp_options=dhcp_options)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 727, in allocate_for_i
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] self._delete_ports(neutron, instance,
created_port_ids)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 204, in __exit__
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] six.reraise(self.type_, self.value,
self.tb)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 719, in allocate_for_i
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] security_group_ids, available_macs,
dhcp_opts)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 342, in _create_port
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] raise
exception.PortBindingFailed(port_id=port_id)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] PortBindingFailed: Binding failed for
port 0514b7e8-0408-4d5e-9b48-8292e686494f, please check neutron logs for more
information.
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b]
=============================================================
>From Neutron:
=============================================================
f 95bfe8c1d4fe49e38b599e801bc94f1c - - -] Failed to bind port
0514b7e8-0408-4d5e-9b48-8292e686494f on host helion-cp1-ir-compute0001-mgmt
2015-12-05 02:57:37.144 845 ERROR neutron.plugins.ml2.managers
[req-53993456-ea74-452a-8299-149505d99bb2 0a8461f0be414de584e666aae579cc8f
95bfe8c1d4fe49e38b599e801bc94f1c - - -] Failed to bind port
0514b7e8-0408-4d5e-9b48-8292e686494f on host helion-cp1-ir-compute0001-mgmt
2015-12-05 02:57:40.520 845 DEBUG keystoneclient.session [-] Request returned
failure status: 404 request
/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py:400
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova [-] Failed to notify
nova on events: [{'tag': u'0514b7e8-0408-4d5e-9b48-8292e686494f', 'name':
'network-vif-deleted', 'server_uuid': u'd9295d4d-1104-47ab-8e75-5b30d0a3838b'}]
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova Traceback (most recent
call last):
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/neutron/notifiers/nova.py",
line 247, in send_events
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova batched_events)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py",
line 39, in create
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return_raw=True)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/novaclient/base.py",
line 169, in _create
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova _resp, body =
self.api.client.post(url, body=body)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/adapter.py",
line 176, in post
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
self.request(url, 'POST', **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/novaclient/client.py",
line 91, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/adapter.py",
line 206, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova resp =
super(LegacyJsonAdapter, self).request(*args, **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/adapter.py",
line 95, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
self.session.request(url, method, **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/utils.py",
line 337, in inner
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return func(*args,
**kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py",
line 304, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova auth_headers =
self.get_auth_headers(auth)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py",
line 617, in get_auth_headers
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
auth.get_headers(self, **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/base.py",
line 142, in get_headers
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova token =
self.get_token(session)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
line 200, in get_token
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
self.get_access(session).auth_token
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
line 240, in get_access
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova self.auth_ref =
self.get_auth_ref(session)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py",
line 88, in get_auth_ref
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova
authenticated=False, log=False)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py",
line 501, in post
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
self.request(url, 'POST', **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/utils.py",
line 337, in inner
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return func(*args,
**kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py",
line 401, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova raise
exceptions.from_response(resp, method, url)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova NotFound: The resource
could not be found. (HTTP 404) (Request-ID:
req-1bdb3964-e31f-41bc-b503-116e3628b6a4)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova
** Affects: nova
Importance: Undecided
Status: New
** Tags: ironic
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1527061
Title:
Nova should not throw exception when port binding fails for Ironic
Status in OpenStack Compute (nova):
New
Bug description:
Neutron tries to bind port on compute where instance is launched. It
doesn't make sense when hypervisor_type is ironic, since VM does not
live on hypervisor in this case. Furthermore it leads to failed
provisioning of baremetal node, when neutron is not configured on
ironic compute node.
When binding fails, nova starts cleanup and then throws exception. As
part of cleanup ironic port is deleted. This should not be the case
when virt-driver being used is Ironic.
Nova versions: liberty on-wards.
Setup:
node-1: controller with neutron
node-2: ironic-compute without neutron/neutron agents
Register a BM node and perform nova boot.
From Nova:
=============================================================
nova-compute.log:21147:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
nova-compute.log:21148:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager Traceback (most recent call last):
nova-compute.log:21149:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 1564, in _allocate_network_async
nova-compute.log:21150:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager dhcp_options=dhcp_options)
nova-compute.log:21151:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 727, in allocate_for_instance
nova-compute.log:21152:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager self._delete_ports(neutron, instance, created_port_ids)
nova-compute.log:21153:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 204, in __exit__
nova-compute.log:21154:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager six.reraise(self.type_, self.value, self.tb)
nova-compute.log:21155:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 719, in allocate_for_instance
nova-compute.log:21156:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager security_group_ids, available_macs, dhcp_opts)
nova-compute.log:21157:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 342, in _create_port
nova-compute.log:21158:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager raise exception.PortBindingFailed(port_id=port_id)
nova-compute.log:21159:2015-12-05 02:57:38.542 7649 ERROR
nova.compute.manager PortBindingFailed: Binding failed for port
0514b7e8-0408-4d5e-9b48-8292e686494f, please check neutron logs for more
information.
nova-compute.log:21160:2015-12-05 02:57:38.542 7649 ERROR nova.compute.manager
2015-12-05 02:57:43.375 7649 DEBUG nova.virt.ironic.driver
[req-25ba4c81-4333-43e4-b8d9-e460827435e0 a989fe7fc89e4825b986666d7e6584cc
1308f1a382ef43c79eeed0ebf8a9db3b] unplug:
instance_uuid=d9295d4d-1104-47ab-8e75-5b30d0a3838b vif=[] _unp
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager
[req-25ba4c81-4333-43e4-b8d9-e460827435e0 a989fe7fc89e4825b986666d7e6584cc
1308f1a382ef43c79eeed0ebf8a9db3b] [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] Instance failed to spawn
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] Traceback (most recent call last):
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 2157, in _build_resources
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] yield resources
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 2011, in _build_and_run_inst
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] block_device_info=block_device_info)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/virt/ironic/driver.py",
line 791, in spawn
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] flavor=flavor)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 204, in __exit__
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] six.reraise(self.type_, self.value,
self.tb)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/virt/ironic/driver.py",
line 782, in spawn
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] self._plug_vifs(node, instance,
network_info)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/virt/ironic/driver.py",
line 1068, in _plug_vifs
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] network_info_str = str(network_info)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/model.py",
line 515, in __str__
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] return self._sync_wrapper(fn, *args,
**kwargs)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/model.py",
line 498, in _sync_wrapper
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] self.wait()
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/model.py",
line 530, in wait
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] self[:] = self._gt.wait()
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/eventlet/greenthread.py",
line 175, in wait
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] return self._exit_event.wait()
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/eventlet/event.py",
line 125, in wait
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] current.throw(*self._exc)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/eventlet/greenthread.py",
line 214, in main
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] result = function(*args, **kwargs)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 1581, in _allocate_network_a
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] six.reraise(*exc_info)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/compute/manager.py",
line 1564, in _allocate_network_a
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] dhcp_options=dhcp_options)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 727, in allocate_for_i
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] self._delete_ports(neutron, instance,
created_port_ids)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/oslo_utils/excutils.py",
line 204, in __exit__
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] six.reraise(self.type_, self.value,
self.tb)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 719, in allocate_for_i
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] security_group_ids, available_macs,
dhcp_opts)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] File
"/opt/stack/venv/nova-20151204T044743Z/lib/python2.7/site-packages/nova/network/neutronv2/api.py",
line 342, in _create_port
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] raise
exception.PortBindingFailed(port_id=port_id)
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b] PortBindingFailed: Binding failed for
port 0514b7e8-0408-4d5e-9b48-8292e686494f, please check neutron logs for more
information.
2015-12-05 02:57:43.376 7649 ERROR nova.compute.manager [instance:
d9295d4d-1104-47ab-8e75-5b30d0a3838b]
=============================================================
From Neutron:
=============================================================
f 95bfe8c1d4fe49e38b599e801bc94f1c - - -] Failed to bind port
0514b7e8-0408-4d5e-9b48-8292e686494f on host helion-cp1-ir-compute0001-mgmt
2015-12-05 02:57:37.144 845 ERROR neutron.plugins.ml2.managers
[req-53993456-ea74-452a-8299-149505d99bb2 0a8461f0be414de584e666aae579cc8f
95bfe8c1d4fe49e38b599e801bc94f1c - - -] Failed to bind port
0514b7e8-0408-4d5e-9b48-8292e686494f on host helion-cp1-ir-compute0001-mgmt
2015-12-05 02:57:40.520 845 DEBUG keystoneclient.session [-] Request returned
failure status: 404 request
/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py:400
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova [-] Failed to notify
nova on events: [{'tag': u'0514b7e8-0408-4d5e-9b48-8292e686494f', 'name':
'network-vif-deleted', 'server_uuid': u'd9295d4d-1104-47ab-8e75-5b30d0a3838b'}]
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova Traceback (most
recent call last):
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/neutron/notifiers/nova.py",
line 247, in send_events
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova batched_events)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py",
line 39, in create
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return_raw=True)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/novaclient/base.py",
line 169, in _create
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova _resp, body =
self.api.client.post(url, body=body)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/adapter.py",
line 176, in post
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
self.request(url, 'POST', **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/novaclient/client.py",
line 91, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/adapter.py",
line 206, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova resp =
super(LegacyJsonAdapter, self).request(*args, **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/adapter.py",
line 95, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
self.session.request(url, method, **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/utils.py",
line 337, in inner
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
func(*args, **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py",
line 304, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova auth_headers =
self.get_auth_headers(auth)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py",
line 617, in get_auth_headers
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
auth.get_headers(self, **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/base.py",
line 142, in get_headers
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova token =
self.get_token(session)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
line 200, in get_token
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
self.get_access(session).auth_token
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
line 240, in get_access
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova self.auth_ref =
self.get_auth_ref(session)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py",
line 88, in get_auth_ref
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova
authenticated=False, log=False)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py",
line 501, in post
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
self.request(url, 'POST', **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/utils.py",
line 337, in inner
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova return
func(*args, **kwargs)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova File
"/opt/stack/venv/neutron-20151204T044743Z/lib/python2.7/site-packages/keystoneclient/session.py",
line 401, in request
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova raise
exceptions.from_response(resp, method, url)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova NotFound: The
resource could not be found. (HTTP 404) (Request-ID:
req-1bdb3964-e31f-41bc-b503-116e3628b6a4)
2015-12-05 02:57:40.521 845 ERROR neutron.notifiers.nova
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1527061/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
