This goes back a long time but I'll just target to the non-extended-
maintenance branches.
** Also affects: nova/rocky
Importance: Undecided
Status: New
** Also affects: nova/stein
Importance: Undecided
Status: New
** Also affects: nova/train
Importance: Undecided
Status: New
** Changed in: nova/rocky
Status: New => Confirmed
** Changed in: nova/train
Status: New => Confirmed
** Changed in: nova/rocky
Importance: Undecided => Low
** Changed in: nova/stein
Status: New => Confirmed
** Changed in: nova/train
Importance: Undecided => Low
** Changed in: nova/stein
Importance: Undecided => Low
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1852465
Title:
ExternalNetworkAttachForbidden should result in BuildAbortException,
not reschedule
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) rocky series:
Confirmed
Status in OpenStack Compute (nova) stein series:
Confirmed
Status in OpenStack Compute (nova) train series:
Confirmed
Bug description:
I saw this in a CI run where creating a server on an external network
as a non-admin user failed and was rescheduled. Here are the failures
from the compute logs:
https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n-cpu.txt.gz#28774
https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/compute1/logs/screen-n-cpu.txt.gz#37415
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [None req-41117798-8a4e-469f-bfbb-8bdfdea1a83f demo
demo] [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Instance failed to
spawn: nova.exception.ExternalNetworkAttachForbidden: It is not allowed to
create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
Traceback (most recent call last):
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/compute/manager.py", line 2659, in _build_resources
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
yield resources
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/compute/manager.py", line 2433, in
_build_and_run_instance
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
block_device_info=block_device_info)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3467, in spawn
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
mdevs=mdevs)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6019, in _get_guest_xml
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
network_info_str = str(network_info)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/network/model.py", line 601, in __str__
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
return self._sync_wrapper(fn, *args, **kwargs)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/network/model.py", line 584, in _sync_wrapper
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
self.wait()
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/network/model.py", line 616, in wait
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
self[:] = self._gt.wait()
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line
181, in wait
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
return self._exit_event.wait()
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in
wait
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
current.throw(*self._exc)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line
221, in main
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
result = function(*args, **kwargs)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/utils.py", line 674, in context_wrapper
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
return func(*args, **kwargs)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/compute/manager.py", line 1704, in
_allocate_network_async
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
six.reraise(*exc_info)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
raise value
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/compute/manager.py", line 1687, in
_allocate_network_async
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
resource_provider_mapping=resource_provider_mapping)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/network/neutronv2/api.py", line 990, in
allocate_for_instance
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
context, instance, neutron, requested_networks, ordered_networks)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/network/neutronv2/api.py", line 868, in
_validate_requested_network_ids
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
self._check_external_network_attach(context, nets)
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
File "/opt/stack/nova/nova/network/neutronv2/api.py", line 542, in
_check_external_network_attach
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
network_uuid=net['id'])
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an
interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0
Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]:
ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd]
And eventually that results in MaxRetriesExceeded in conductor:
https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n
-cond-cell1.txt.gz#878
Nov 13 15:35:47.968308 ubuntu-bionic-rax-ord-0012778423 nova-
conductor[25967]: WARNING nova.scheduler.utils [None req-41117798
-8a4e-469f-bfbb-8bdfdea1a83f demo demo] Failed to
compute_task_build_instances: Exceeded maximum number of retries.
Exhausted all hosts available for retrying build failures for instance
be6eb09a-f0d9-4c04-9c55-29230a253fbd.:
nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries.
Exhausted all hosts available for retrying build failures for instance
be6eb09a-f0d9-4c04-9c55-29230a253fbd.
ExternalNetworkAttachForbidden is not something that a reschedule is
going to fix, it's essentially a bad request from the user and needs
to just abort.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1852465/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
