Reviewed: https://review.opendev.org/694179 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=36b110dfc5f8f04234b189a50e96de8dde5f3471 Submitter: Zuul Branch: master
commit 36b110dfc5f8f04234b189a50e96de8dde5f3471 Author: Matt Riedemann <[email protected]> Date: Wed Nov 13 18:27:36 2019 -0500 Do not reschedule on ExternalNetworkAttachForbidden When creating a server, if network setup fails with ExternalNetworkAttachForbidden then we should abort the build rather than reschedule because it's not going to work on another host either. A functional test is added for this since it's hard to reason about whether or not it works properly due to the async nature of network setup during server create and the _build_resources context manager. It's probably not a bad idea to have a functional test for a scenario that uses external networks either (and we can build on this). Change-Id: I6dd5b3f75e2746f53f9758f1a186794a98dba9bf Closes-Bug: #1852465 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1852465 Title: ExternalNetworkAttachForbidden should result in BuildAbortException, not reschedule Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Status in OpenStack Compute (nova) train series: Confirmed Bug description: I saw this in a CI run where creating a server on an external network as a non-admin user failed and was rescheduled. Here are the failures from the compute logs: https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n-cpu.txt.gz#28774 https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/compute1/logs/screen-n-cpu.txt.gz#37415 Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [None req-41117798-8a4e-469f-bfbb-8bdfdea1a83f demo demo] [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Instance failed to spawn: nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0 Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] Traceback (most recent call last): Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 2659, in _build_resources Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] yield resources Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 2433, in _build_and_run_instance Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] block_device_info=block_device_info) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3467, in spawn Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] mdevs=mdevs) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6019, in _get_guest_xml Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] network_info_str = str(network_info) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 601, in __str__ Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] return self._sync_wrapper(fn, *args, **kwargs) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 584, in _sync_wrapper Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] self.wait() Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/model.py", line 616, in wait Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] self[:] = self._gt.wait() Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 181, in wait Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] return self._exit_event.wait() Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 132, in wait Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] current.throw(*self._exc) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/eventlet/greenthread.py", line 221, in main Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] result = function(*args, **kwargs) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/utils.py", line 674, in context_wrapper Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] return func(*args, **kwargs) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 1704, in _allocate_network_async Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] six.reraise(*exc_info) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] raise value Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/compute/manager.py", line 1687, in _allocate_network_async Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] resource_provider_mapping=resource_provider_mapping) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 990, in allocate_for_instance Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] context, instance, neutron, requested_networks, ordered_networks) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 868, in _validate_requested_network_ids Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] self._check_external_network_attach(context, nets) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] File "/opt/stack/nova/nova/network/neutronv2/api.py", line 542, in _check_external_network_attach Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] network_uuid=net['id']) Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] nova.exception.ExternalNetworkAttachForbidden: It is not allowed to create an interface on external network 29715f6f-24ab-49b7-abff-60d3f97596a0 Nov 13 15:35:44.312293 ubuntu-bionic-rax-ord-0012778423 nova-compute[26791]: ERROR nova.compute.manager [instance: be6eb09a-f0d9-4c04-9c55-29230a253fbd] And eventually that results in MaxRetriesExceeded in conductor: https://zuul.opendev.org/t/openstack/build/540a9fc0dbc64abb92d3f3e513573307/log/controller/logs/screen-n -cond-cell1.txt.gz#878 Nov 13 15:35:47.968308 ubuntu-bionic-rax-ord-0012778423 nova- conductor[25967]: WARNING nova.scheduler.utils [None req-41117798 -8a4e-469f-bfbb-8bdfdea1a83f demo demo] Failed to compute_task_build_instances: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance be6eb09a-f0d9-4c04-9c55-29230a253fbd.: nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance be6eb09a-f0d9-4c04-9c55-29230a253fbd. ExternalNetworkAttachForbidden is not something that a reschedule is going to fix, it's essentially a bad request from the user and needs to just abort. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1852465/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
