Public bug reported: Description =========== When trying nova VMDK volume detach from an instance hosted on vCenter, the detach fail if the shadow VM used for vmdk volume is unregistered and registered back in vCenter.
Steps to reproduce ================== * Attach a VMDK volume to an instance * Unregister/delete the shadow VM. [A VM just contains the VMDK matching the cinder UUID] * register/create the VM back with the same name * Try detach Expected result =============== It should detach the volume from the instance Actual result ============= It fails to detach. Because in the vCenter, due to un-registering the vm MOid changed. In block_device_mapping table we store this information (connection_info) and use it to form the volume ref during detach in the driver. https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/volumeops.py#L305 I think we should not be relying on that, instead make the vCenter call and get the MOID by name. Logs ==== Cause: Server raised fault: 'The object has already been deleted or has not been completely created' Faults: [ManagedObjectNotFound] Details: {'obj': 'vm-1031'} 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher incoming.message)) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher payload) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 387, in decorated_function 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 375, in decorated_function 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4892, in detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher attachment_id=attachment_id) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4835, in _detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher connection_info = self._driver_detach_volume(context, instance, bdm) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4787, in _driver_detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.volume_api.roll_detaching(context, volume_id) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4775, in _driver_detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher encryption=encryption) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 391, in detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self._volumeops.detach_volume(connection_info, instance) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 582, in detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self._detach_volume_vmdk(connection_info, instance) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 530, in _detach_volume_vmdk 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher vmdk = "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 635, in get_vmdk_info 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher "config.hardware.device") 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 596, in _call_method 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self.invoke_api(module, method, self.vim, *args, **kwargs) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 348, in invoke_api 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return _invoke_api(module, method, *args, **kwargs) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 123, in func 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return evt.wait() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return hubs.get_hub().switch() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self.greenlet.switch() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 124, in _inner 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher idle = self.f(*self.args, **self.kw) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 96, in _func 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher result = f(*args, **kwargs) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 332, in _invoke_api 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher details=excep.details) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher ManagedObjectNotFoundException: The object has already been deleted or has not been completely created 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Cause: Server raised fault: 'The object has already been deleted or has not been completely created' 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Faults: [ManagedObjectNotFound] 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Details: {'obj': 'vm-1031'} 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Environment =========== nova driver: VMWareVCDriver cinder driver: VMDkdriver Observing the issue on "stable/mitaka". I believe this issue persists in master as well. ** Affects: nova Importance: Undecided Assignee: Chinmaya Bharadwaj (acbharadwaj) Status: New ** Changed in: nova Assignee: (unassigned) => Chinmaya Bharadwaj (acbharadwaj) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1593742 Title: Detach VMDK volume fails if shadow VM unregistered and registered back Status in OpenStack Compute (nova): New Bug description: Description =========== When trying nova VMDK volume detach from an instance hosted on vCenter, the detach fail if the shadow VM used for vmdk volume is unregistered and registered back in vCenter. Steps to reproduce ================== * Attach a VMDK volume to an instance * Unregister/delete the shadow VM. [A VM just contains the VMDK matching the cinder UUID] * register/create the VM back with the same name * Try detach Expected result =============== It should detach the volume from the instance Actual result ============= It fails to detach. Because in the vCenter, due to un-registering the vm MOid changed. In block_device_mapping table we store this information (connection_info) and use it to form the volume ref during detach in the driver. https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/volumeops.py#L305 I think we should not be relying on that, instead make the vCenter call and get the MOID by name. Logs ==== Cause: Server raised fault: 'The object has already been deleted or has not been completely created' Faults: [ManagedObjectNotFound] Details: {'obj': 'vm-1031'} 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher incoming.message)) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 110, in wrapped 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher payload) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 387, in decorated_function 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info()) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 375, in decorated_function 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4892, in detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher attachment_id=attachment_id) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4835, in _detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher connection_info = self._driver_detach_volume(context, instance, bdm) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4787, in _driver_detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.volume_api.roll_detaching(context, volume_id) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__ 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self.force_reraise() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4775, in _driver_detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher encryption=encryption) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 391, in detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self._volumeops.detach_volume(connection_info, instance) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 582, in detach_volume 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher self._detach_volume_vmdk(connection_info, instance) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 530, in _detach_volume_vmdk 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher vmdk = "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 635, in get_vmdk_info 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher "config.hardware.device") 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 596, in _call_method 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self.invoke_api(module, method, self.vim, *args, **kwargs) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 348, in invoke_api 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return _invoke_api(module, method, *args, **kwargs) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 123, in func 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return evt.wait() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return hubs.get_hub().switch() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher return self.greenlet.switch() 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 124, in _inner 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher idle = self.f(*self.args, **self.kw) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 96, in _func 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher result = f(*args, **kwargs) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 332, in _invoke_api 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher details=excep.details) 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher ManagedObjectNotFoundException: The object has already been deleted or has not been completely created 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Cause: Server raised fault: 'The object has already been deleted or has not been completely created' 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Faults: [ManagedObjectNotFound] 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Details: {'obj': 'vm-1031'} 2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Environment =========== nova driver: VMWareVCDriver cinder driver: VMDkdriver Observing the issue on "stable/mitaka". I believe this issue persists in master as well. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1593742/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
