Public bug reported: I've started noticing a lot of these in the neutron job logs:
http://logs.openstack.org/67/269867/4/check/gate-tempest-dsvm-neutron- src-os-brick/7617a9f/logs/screen-n-cpu.txt.gz#_2016-01-21_05_38_48_667 2016-01-21 05:38:48.667 ERROR nova.virt.libvirt.driver [req-c8971f87-303e-460b-894b-7e2ccde9944f nova service] [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] detaching network adapter failed. 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] Traceback (most recent call last): 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1354, in detach_interface 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] guest.detach_device(cfg, persistent=True, live=live) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 341, in detach_device 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] self._domain.detachDeviceFlags(conf.to_xml(), flags=flags) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] result = proxy_call(self._autowrap, f, *args, **kwargs) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] rv = execute(f, *args, **kwargs) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] six.reraise(c, e, tb) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] rv = meth(*args, **kwargs) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 985, in detachDeviceFlags 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] libvirtError: operation failed: no matching network device was found 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] Following the request ID, it's coming from a neutron vif deleted event: http://logs.openstack.org/67/269867/4/check/gate-tempest-dsvm-neutron- src-os-brick/7617a9f/logs/screen-n-cpu.txt.gz#_2016-01-21_05_38_48_361 It looks like we should just handle and not log that network device not found as an error, but libvirt doesn't provide a specific error code for that case: http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_driver.c;h=e04a32841807851fd1386ab8a7fd91672f114dce;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l6764 And the error message is translatable so we can't just parse the message either. It seems there should be a way to verify that the network device exists before we try to detach it from the config so we can avoid this error which doesn't actually result in job failures (it's probably happening during an instance delete asynchronously where we're deleting the ports in neutron). http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Caught%20error%3A%20%3Ctype%20'exceptions.AttributeError'%3E%20'module'%20object%20has%20no%20attribute%20'ServiceList'%5C%22%20AND%20tags%3A%5C%22screen-c-api.txt%5C%22&from=7d ** Affects: nova Importance: Medium Status: Confirmed ** Tags: libvirt network neutron ** Changed in: nova Status: New => Confirmed ** Changed in: nova Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1536671 Title: libvirt detach_interface logs errors for network device not found after neutron network-vif-deleted event Status in OpenStack Compute (nova): Confirmed Bug description: I've started noticing a lot of these in the neutron job logs: http://logs.openstack.org/67/269867/4/check/gate-tempest-dsvm-neutron- src-os-brick/7617a9f/logs/screen-n-cpu.txt.gz#_2016-01-21_05_38_48_667 2016-01-21 05:38:48.667 ERROR nova.virt.libvirt.driver [req-c8971f87-303e-460b-894b-7e2ccde9944f nova service] [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] detaching network adapter failed. 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] Traceback (most recent call last): 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1354, in detach_interface 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] guest.detach_device(cfg, persistent=True, live=live) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 341, in detach_device 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] self._domain.detachDeviceFlags(conf.to_xml(), flags=flags) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] result = proxy_call(self._autowrap, f, *args, **kwargs) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] rv = execute(f, *args, **kwargs) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] six.reraise(c, e, tb) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] rv = meth(*args, **kwargs) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 985, in detachDeviceFlags 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self) 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] libvirtError: operation failed: no matching network device was found 2016-01-21 05:38:48.667 12834 ERROR nova.virt.libvirt.driver [instance: d8d15c87-79cc-4b63-99bb-64dde4576b3d] Following the request ID, it's coming from a neutron vif deleted event: http://logs.openstack.org/67/269867/4/check/gate-tempest-dsvm-neutron- src-os-brick/7617a9f/logs/screen-n-cpu.txt.gz#_2016-01-21_05_38_48_361 It looks like we should just handle and not log that network device not found as an error, but libvirt doesn't provide a specific error code for that case: http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/qemu/qemu_driver.c;h=e04a32841807851fd1386ab8a7fd91672f114dce;hb=e8684eb541f01df9b45e87e0a8ce446c7bc90a17#l6764 And the error message is translatable so we can't just parse the message either. It seems there should be a way to verify that the network device exists before we try to detach it from the config so we can avoid this error which doesn't actually result in job failures (it's probably happening during an instance delete asynchronously where we're deleting the ports in neutron). http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Caught%20error%3A%20%3Ctype%20'exceptions.AttributeError'%3E%20'module'%20object%20has%20no%20attribute%20'ServiceList'%5C%22%20AND%20tags%3A%5C%22screen-c-api.txt%5C%22&from=7d To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1536671/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
