Reviewed: https://review.openstack.org/541660 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d5901f613cf98f61b5253a1568b22af1d9dd1a08 Submitter: Zuul Branch: master
commit d5901f613cf98f61b5253a1568b22af1d9dd1a08 Author: LeopardMa <[email protected]> Date: Wed Feb 7 10:21:08 2018 +0800 Fix shelving a paused instance It is possible to shelve a paused instance, but in that case the guest is already shutdown, and some hypervisors will fail when trying to perform a clean shutdown of a non-running guest. For example, attempting to shelve a paused libvirt instance will result in this error: libvirtError: Requested operation is not valid: domain is not running Therefore, if the instance is paused, we don't attempt a clean shutdown while shelving. Related Tempest test: https://review.openstack.org/564127/ Closes-Bug: #1745529 Change-Id: I8ca25d9847d50001fbe8513a6c1dba8b697c24e4 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1745529 Title: Failed to shelve paused instance with libvirt: "libvirtError: Requested operation is not valid: domain is not running" Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Confirmed Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Bug description: You can use the following steps to reproduce(This problem also exists in the Ocata version)： 1.nova pause 5e824732-c557-434f-9902-a6d9abce7bda 2.nova list +--------------------------------------+-------+--------+------------+-------------+---------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+---------------------+ | 5e824732-c557-434f-9902-a6d9abce7bda | test | PAUSED | - | Paused | network=192.168.0.9 | +--------------------------------------+-------+--------+------------+-------------+---------------------+ 3.nova shelve 5e824732-c557-434f-9902-a6d9abce7bda Error log is as follows： oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Lock "compute_resources" acquired by "nova.compute.resource_tracker.update_usage" :: waited 0.000s {{(pid=8304) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273}} Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: DEBUG oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Lock "compute_resources" released by "nova.compute.resource_tracker.update_usage" :: held 0.038s {{(pid=8304) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285}} Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: INFO nova.compute.manager [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] [instance: 5e824732-c557-434f-9902-a6d9abce7bda] Successfully reverted task state from shelving on failure for instance. Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Exception during message handling: libvirtError: Requested operation is not valid: domain is not running Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server Traceback (most recent call last): Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 76, in wrapped Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server function_name, call_dict, binary) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self.force_reraise() Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 67, in wrapped Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 186, in decorated_function Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self.force_reraise() Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 156, in decorated_function Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 944, in decorated_function Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 214, in decorated_function Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info()) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self.force_reraise() Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 202, in decorated_function Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 4678, in shelve_instance Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server do_shelve_instance() Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 4677, in do_shelve_instance Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self._shelve_instance(context, instance, image_id, clean_shutdown) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 4711, in _shelve_instance Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self._power_off_instance(context, instance, clean_shutdown) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 2318, in _power_off_instance Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self.driver.power_off(instance, timeout, retry_interval) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2684, in power_off Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self._clean_shutdown(instance, timeout, retry_interval) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2639, in _clean_shutdown Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server guest.shutdown() Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 589, in shutdown Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server self._domain.shutdown() Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2399, in shutdown Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainShutdown() failed', dom=self) Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server libvirtError: Requested operation is not valid: domain is not running Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR oslo_messaging.rpc.server Jan 26 10:13:16 localhost.localdomain nova-compute[8304]: DEBUG oslo_service.periodic_task [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin admin] Running periodic task ComputeManager._poll_rebooting_instances {{(pid=8304) run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215}} To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1745529/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
