Nevermind about detaching the volume from cinder, the compute manager tries to detach in the virt driver first, and then when that is done the compute manager terminates the connection via cinder API and then detaches the volume via cinder API.
Following: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4785 Do you see anything in the libvirtd logs for the volume/instance uuid here? ** Changed in: cinder Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1565859 Title: Can't detach SVC volume from an instance; guest detach device times out Status in Cinder: Invalid Status in OpenStack Compute (nova): Incomplete Bug description: Steps to Reproduce: 1. Setup instance with a libvirt/KVM host and SVC storage 2. Spawn a VM 3. Attach a volume to the VM 4. Wait for volume attachment to complete successfully 5. Detach the volume Expected Result: 1. The volume is detached from the VM 2. The volume's status becomes "Available" 3. The volume can be deleted. Actual result: 1. Volume remains attached to the VM (waited over 10 minutes) 2. The volume's state stays "In-Use" Logs: 016-03-24 16:34:13.852 143842 INFO nova.compute.resource_tracker [-] Final resource view: name=C387f19U21-KVM phys_ram=260533MB used_ram=4608MB phys_disk=545GB used_disk=40GB total_vcpus=160 used_vcpus=2 pci_stats=[] 2016-03-24 16:34:14.081 143842 INFO nova.compute.resource_tracker [-] Compute_service record updated for C387f19U21_KVM:C387f19U21-KVM 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall [-] Dynamic interval looping call 'oslo_service.loopingcall._func' failed 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall Traceback (most recent call last): 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 136, in _run_loop 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw) 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 377, in _func 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall return self._sleep_time 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall self.force_reraise() 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb) 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 356, in _func 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall result = f(*args, **kwargs) 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 342, in _do_wait_and_retry_detach 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall reason=reason) 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain.) 2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [req-fab1608e-cffe-40f9-82d0-a4c7a9cebf10 3ebcf1a38bc7b4977b7f8da32faad97bdef843372a670bb2817f8a066f042b9b e10bc17f58d8499a8fab1b05687123e5 - - -] [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] Failed to detach volume 4221ccad-0f98-4f78-ad06-92ea4941afc1 from /dev/vdb 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] Traceback (most recent call last): 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4767, in _driver_detach_volume 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] encryption=encryption) 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1469, in detach_volume 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] wait_for_detach() 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 385, in func 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] return evt.wait() 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] return hubs.get_hub().switch() 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] return self.greenlet.switch() 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 136, in _run_loop 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] result = func(*self.args, **self.kw) 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 377, in _func 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] return self._sleep_time 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] self.force_reraise() 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] six.reraise(self.type_, self.value, self.tb) 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 356, in _func 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] result = f(*args, **kwargs) 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 342, in _do_wait_and_retry_detach 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] reason=reason) 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain.) 2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1565859/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
