Public bug reported: Description ===========
Attempting to delete a NFS volume snapshot (via c-api and the the os- assisted-volume-snapshots n-api) of a volume attached to a SHUTOFF instance currently results in n-cpu attempting to fire off a virDomainBlockCommit command even though the instance isn't running. Steps to reproduce ================== 1. Create multiple volume snapshots against a volume. 2. Attach the volume to an ACTIVE instance. 3. Stop the instance and ensure it is SHUTOFF. 4. Attempt to delete the latest snapshot. Expected result =============== qemu-img commit or qemu-img rebase should be used to handle this offline. Actual result ============= virDomainBlockCommit is called even though the domain isn't running. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ master 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? NFS c-vol 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs ============== Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server [req-570281c6-566e-44a3-9953-eeb634513778 req-0fbbe87f-fd1d-4861-9fb3-21b8eb011e55 service nova] Exception during message handling: libvirt.libvirtError: Requested operation is not valid: domain is not > Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 241, in inner Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 78, in wrapped Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server self.force_reraise() Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server raise value Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3916, in volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server snapshot_id, delete_info) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3129, in volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server context, snapshot_id, 'error_deleting') Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server self.force_reraise() Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server raise value Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3122, in volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server snapshot_id, delete_info=delete_info) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3107, in _volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server result = dev.commit(commit_base, commit_top, relative=True) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 809, in commit Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server self._disk, base, top, self.COMMIT_DEFAULT_BANDWIDTH, flags=flags) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 190, in doit Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 148, in proxy_call Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 129, in execute Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server raise value Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 83, in tworker Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.7/site-packages/libvirt.py", line 728, in blockCommit Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server libvirt.libvirtError: Requested operation is not valid: domain is not running ** Affects: nova Importance: Medium Assignee: Lee Yarwood (lyarwood) Status: New ** Tags: nfs volumes ** Changed in: nova Assignee: (unassigned) => Lee Yarwood (lyarwood) ** Changed in: nova Importance: Undecided => Medium -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1919487 Title: virDomainBlockCommit called when deleting a snapshot via os-assisted- volume-snapshots even when instance is shutoff Status in OpenStack Compute (nova): New Bug description: Description =========== Attempting to delete a NFS volume snapshot (via c-api and the the os- assisted-volume-snapshots n-api) of a volume attached to a SHUTOFF instance currently results in n-cpu attempting to fire off a virDomainBlockCommit command even though the instance isn't running. Steps to reproduce ================== 1. Create multiple volume snapshots against a volume. 2. Attach the volume to an ACTIVE instance. 3. Stop the instance and ensure it is SHUTOFF. 4. Attempt to delete the latest snapshot. Expected result =============== qemu-img commit or qemu-img rebase should be used to handle this offline. Actual result ============= virDomainBlockCommit is called even though the domain isn't running. Environment =========== 1. Exact version of OpenStack you are running. See the following list for all releases: http://docs.openstack.org/releases/ master 2. Which hypervisor did you use? (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...) What's the version of that? libvirt + KVM 2. Which storage type did you use? (For example: Ceph, LVM, GPFS, ...) What's the version of that? NFS c-vol 3. Which networking type did you use? (For example: nova-network, Neutron with OpenVSwitch, ...) N/A Logs & Configs ============== Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server [req-570281c6-566e-44a3-9953-eeb634513778 req-0fbbe87f-fd1d-4861-9fb3-21b8eb011e55 service nova] Exception during message handling: libvirt.libvirtError: Requested operation is not valid: domain is not > Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 273, in dispatch Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _do_dispatch Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_messaging/rpc/server.py", line 241, in inner Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server return func(*args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 78, in wrapped Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server self.force_reraise() Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server raise value Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 69, in wrapped Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3916, in volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server snapshot_id, delete_info) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3129, in volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server context, snapshot_id, 'error_deleting') Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server self.force_reraise() Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server raise value Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3122, in volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server snapshot_id, delete_info=delete_info) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3107, in _volume_snapshot_delete Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server result = dev.commit(commit_base, commit_top, relative=True) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 809, in commit Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server self._disk, base, top, self.COMMIT_DEFAULT_BANDWIDTH, flags=flags) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 190, in doit Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 148, in proxy_call Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 129, in execute Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/six.py", line 703, in reraise Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server raise value Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.7/site-packages/eventlet/tpool.py", line 83, in tworker Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server File "/usr/local/lib64/python3.7/site-packages/libvirt.py", line 728, in blockCommit Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) Jul 03 09:37:57 localhost.localdomain nova-compute[127223]: ERROR oslo_messaging.rpc.server libvirt.libvirtError: Requested operation is not valid: domain is not running To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1919487/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
