Public bug reported:
In my case, we had a chain of patches from
https://review.openstack.org/#/q/topic:bug/1686116 backported to ocata
downstream. Then, when detaching a ceph volume from a node, the
following happens:
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] Traceback
(most recent call last):
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4835, in
_driver_detach_volume
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
encryption=encryption)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1393, in
detach_volume
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] live=live)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 413, in
detach_device_with_retry
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
_try_detach_device(conf, persistent, live)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 407, in
_try_detach_device
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] raise
exception.DeviceNotFound(device=device)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in _exit_
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
self.force_reraise()
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
six.reraise(self.type_, self.value, self.tb)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 392, in
_try_detach_device
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
self.detach_device(conf, persistent=persistent, live=live)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 449, in
detach_device
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
self._domain.detachDeviceFlags(device_xml, flags=flags)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] result =
proxy_call(self._autowrap, f, *args, **kwargs)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv =
execute(f, *args, **kwargs)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
six.reraise(c, e, tb)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv =
meth(*args, **kwargs)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/libvirt.py", line 1190, in detachDeviceFlags
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] if ret ==
-1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
libvirtError: XML error: Invalid PCI address 0000:00:00, at least one of
domain, bus, or slot must be > 0
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
Here is the instance xml bit for the volume:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='volumes'>
<secret type='ceph' uuid='ce6d1549-4d63-476b-afb6-88f0b196414f'/>
</auth>
<source protocol='rbd'
name='volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc'>
<host name='192.168.90.22' port='6789'/>
</source>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</disk>
After inserting some debug logs, the following xml gets passed to the
libvirt:
<disk type="network" device="disk">
<driver name="qemu" type="raw" cache="none"/>
<source protocol="rbd"
name="volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc">
<host name="192.168.90.22" port="6789"/>
</source>
<target bus="virtio" dev="vdb"/>
<serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial>
<address type="pci"/>
</disk>
After introducing a proper format_dom method for
LibvirtConfigGuestDeviceAddressPCI, things seem to be back to normal.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1709319
Title:
LibvirtConfigGuestDeviceAddressPCI missing format_dom method
Status in OpenStack Compute (nova):
New
Bug description:
In my case, we had a chain of patches from
https://review.openstack.org/#/q/topic:bug/1686116 backported to ocata
downstream. Then, when detaching a ceph volume from a node, the
following happens:
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] Traceback
(most recent call last):
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4835, in
_driver_detach_volume
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
encryption=encryption)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1393, in
detach_volume
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] live=live)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 413, in
detach_device_with_retry
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
_try_detach_device(conf, persistent, live)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 407, in
_try_detach_device
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] raise
exception.DeviceNotFound(device=device)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in _exit_
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
self.force_reraise()
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
six.reraise(self.type_, self.value, self.tb)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 392, in
_try_detach_device
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
self.detach_device(conf, persistent=persistent, live=live)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 449, in
detach_device
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
self._domain.detachDeviceFlags(device_xml, flags=flags)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] result =
proxy_call(self._autowrap, f, *args, **kwargs)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv =
execute(f, *args, **kwargs)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
six.reraise(c, e, tb)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] rv =
meth(*args, **kwargs)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] File
"/usr/lib/python2.7/dist-packages/libvirt.py", line 1190, in detachDeviceFlags
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604] if ret ==
-1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
libvirtError: XML error: Invalid PCI address 0000:00:00, at least one of
domain, bus, or slot must be > 0
nova/nova-compute.log.1:2017-07-31 00:21:24.261 341396 ERROR
nova.compute.manager [instance: 43304a1b-bfcf-4e78-a9a0-eec1c6eff604]
Here is the instance xml bit for the volume:
<disk type='network' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<auth username='volumes'>
<secret type='ceph' uuid='ce6d1549-4d63-476b-afb6-88f0b196414f'/>
</auth>
<source protocol='rbd'
name='volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc'>
<host name='192.168.90.22' port='6789'/>
</source>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</disk>
After inserting some debug logs, the following xml gets passed to the
libvirt:
<disk type="network" device="disk">
<driver name="qemu" type="raw" cache="none"/>
<source protocol="rbd"
name="volumes/volume-49bd8d30-96cb-455c-9b73-8dc67bf0e7fc">
<host name="192.168.90.22" port="6789"/>
</source>
<target bus="virtio" dev="vdb"/>
<serial>49bd8d30-96cb-455c-9b73-8dc67bf0e7fc</serial>
<address type="pci"/>
</disk>
After introducing a proper format_dom method for
LibvirtConfigGuestDeviceAddressPCI, things seem to be back to normal.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1709319/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
