Public bug reported: Description =========== Volume status rollbacks to "in-use" after a failure to detach a volume due to DeviceDetachFailed, which is fixed in https://bugs.launchpad.net/nova/+bug/1786318. But the next detaching action will fail because of DeviceNotFound exception.
Steps to reproduce ================== Attempt to detach a volume while it is mounted and in-use within an instance. Wait volume status rollbacks to "in-use". Attempt to detach the volume again. Expected result =============== Volume detach succeeds. Actual result ============= Volume detach fails and the volume returns to in-use after the second detach. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ Queens-17.0.3 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? Libirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? N/A 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs ============== Logs of second detach after DeviceDetachFailed 2019-10-30 13:17:29.401 2109249 INFO nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Attempting to driver detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from mountpoint /dev/vdb 2019-10-30 13:17:29.407 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] Attempting initial detach for device vdb detach_device_with_retry /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:435 2019-10-30 13:17:29.408 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] detach device xml: <disk type="network" device="disk"> <driver name="qemu" type="raw" cache="writethrough" discard="unmap"/> <source protocol="rbd" name="volumes/volume-355d2619-cd59-4c97-80a9-b65ff1870961"> <host name="10.114.198.109" port="6789"/> <host name="10.114.198.110" port="6789"/> <host name="10.114.198.111" port="6789"/> <host name="10.114.198.116" port="6789"/> <host name="10.114.198.117" port="6789"/> </source> <target bus="virtio" dev="vdb"/> <serial>355d2619-cd59-4c97-80a9-b65ff1870961</serial> <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/> </disk> detach_device /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:486 2019-10-30 13:17:29.417 2109249 ERROR nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Failed to detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from /dev/vdb: libvirtError: device not found: no target device vdb ** Affects: nova Importance: Undecided Assignee: Fan Zhang (fanzhang) Status: New ** Changed in: nova Assignee: (unassigned) => Fan Zhang (fanzhang) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1850783 Title: Need to rollback persistent domain xml after DeviceDetachFailed exception Status in OpenStack Compute (nova): New Bug description: Description =========== Volume status rollbacks to "in-use" after a failure to detach a volume due to DeviceDetachFailed, which is fixed in https://bugs.launchpad.net/nova/+bug/1786318. But the next detaching action will fail because of DeviceNotFound exception. Steps to reproduce ================== Attempt to detach a volume while it is mounted and in-use within an instance. Wait volume status rollbacks to "in-use". Attempt to detach the volume again. Expected result =============== Volume detach succeeds. Actual result ============= Volume detach fails and the volume returns to in-use after the second detach. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ Queens-17.0.3 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? Libirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? N/A 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs ============== Logs of second detach after DeviceDetachFailed 2019-10-30 13:17:29.401 2109249 INFO nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Attempting to driver detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from mountpoint /dev/vdb 2019-10-30 13:17:29.407 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] Attempting initial detach for device vdb detach_device_with_retry /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:435 2019-10-30 13:17:29.408 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] detach device xml: <disk type="network" device="disk"> <driver name="qemu" type="raw" cache="writethrough" discard="unmap"/> <source protocol="rbd" name="volumes/volume-355d2619-cd59-4c97-80a9-b65ff1870961"> <host name="10.114.198.109" port="6789"/> <host name="10.114.198.110" port="6789"/> <host name="10.114.198.111" port="6789"/> <host name="10.114.198.116" port="6789"/> <host name="10.114.198.117" port="6789"/> </source> <target bus="virtio" dev="vdb"/> <serial>355d2619-cd59-4c97-80a9-b65ff1870961</serial> <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/> </disk> detach_device /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:486 2019-10-30 13:17:29.417 2109249 ERROR nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Failed to detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from /dev/vdb: libvirtError: device not found: no target device vdb To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1850783/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
