Reviewed: https://review.openstack.org/349014 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a3b3e8d8314b0cedc2604be509f0f4d523a35ed5 Submitter: Jenkins Branch: master
commit a3b3e8d8314b0cedc2604be509f0f4d523a35ed5 Author: Matt Riedemann <[email protected]> Date: Thu Feb 9 15:54:41 2017 -0500 libvirt: wait for interface detach from the guest The test_reassign_port_between_servers test in Tempest creates a port in neutron and two servers. It attaches the port to the first server and then quickly detaches it and waits for the port.device_id to be unbound from the server. Then it repeats that for the second server. The interface detach from the guest is asynchronous and happens before nova unbinds the port, so there is a race where the port's device_id is unset but the interface is still on the first guest when we try to attach to the second guest, which fails. This is a latent bug, but apparently has been tickled by the move to our neutron CI jobs to use ubuntu xenial nodes. The fix is to add a detach and retry loop on the interface detach on the guest so that we can wait until the interface is gone from the guest before nova unbinds the port in neutron, which is what the Tempest test is waiting for. Then the device should be available for attaching to the second guest. This is similar to what we do with detaching volumes. Closes-Bug: #1607714 Change-Id: Ic04aad8923ea2edf1d16e32c208cd41fdf898834 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1607714 Title: test_reassign_port_between_servers failing with tap device is busy errors in neutron xenial jobs since 7/28 Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: Confirmed Status in OpenStack Compute (nova) ocata series: In Progress Bug description: Recent failures showing up in the tempest tests for 'test_reassign_port_between_servers'. From n-cpu.log (http://logs.openstack.org/43/298443/3/check/gate- tempest-dsvm-neutron-full-ubuntu- xenial/5acdcca/logs/screen-n-cpu.txt.gz#_2016-07-29_07_25_47_948): 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [req-572c16fd-696d-44ab-a633-49e6625b8f9c tempest-AttachInterfacesTestJSON-1497713114 tempest-AttachInterfacesTestJSON-1497713114] [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] attaching network adapter failed. 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] Traceback (most recent call last): 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1389, in attach_interface 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] guest.attach_device(cfg, persistent=True, live=live) 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 295, in attach_device 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] self._domain.attachDeviceFlags(device_xml, flags=flags) 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] result = proxy_call(self._autowrap, f, *args, **kwargs) 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] rv = execute(f, *args, **kwargs) 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] six.reraise(c, e, tb) 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] rv = meth(*args, **kwargs) 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 560, in attachDeviceFlags 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] libvirtError: Unable to create tap device tap9dd515f0-ac: Device or resource busy 2016-07-29 07:25:47.948 20203 ERROR nova.virt.libvirt.driver [instance: 73323063-7cc3-4645-9a68-662bf80d9e42] To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1607714/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
