** Changed in: nova
Assignee: Matt Riedemann (mriedem) => Leopard Ma (leopardma)
** Also affects: nova/ocata
Importance: Undecided
Status: New
** Changed in: nova/ocata
Status: New => Confirmed
** Changed in: nova/ocata
Importance: Undecided => Medium
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1745529
Title:
Failed to shelve paused instance with libvirt: "libvirtError:
Requested operation is not valid: domain is not running"
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) ocata series:
Confirmed
Status in OpenStack Compute (nova) pike series:
Confirmed
Status in OpenStack Compute (nova) queens series:
Confirmed
Bug description:
You can use the following steps to reproduce(This problem also exists in the
Ocata version)：
1.nova pause 5e824732-c557-434f-9902-a6d9abce7bda
2.nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power
State | Networks |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| 5e824732-c557-434f-9902-a6d9abce7bda | test | PAUSED | - | Paused
| network=192.168.0.9 |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
3.nova shelve 5e824732-c557-434f-9902-a6d9abce7bda
Error log is as follows：
oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169
admin admin] Lock "compute_resources" acquired by
"nova.compute.resource_tracker.update_usage" :: waited 0.000s {{(pid=8304)
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273}}
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: DEBUG
oslo_concurrency.lockutils [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin
admin] Lock "compute_resources" released by
"nova.compute.resource_tracker.update_usage" :: held 0.038s {{(pid=8304) inner
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285}}
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: INFO
nova.compute.manager [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin
admin] [instance: 5e824732-c557-434f-9902-a6d9abce7bda] Successfully reverted
task state from shelving on failure for instance.
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin
admin] Exception during message handling: libvirtError: Requested operation is
not valid: domain is not running
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server Traceback (most recent call last):
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in
_process_incoming
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220,
in dispatch
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt,
args)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190,
in _do_dispatch
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py",
line 76, in wrapped
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server function_name, call_dict, binary)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server self.force_reraise()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py",
line 67, in wrapped
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server return f(self, context, *args, **kw)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py",
line 186, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server "Error: %s", e, instance=instance)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server self.force_reraise()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py",
line 156, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line
944, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py",
line 214, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server self.force_reraise()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
force_reraise
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py",
line 202, in decorated_function
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py",
line 4678, in shelve_instance
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server do_shelve_instance()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in
inner
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server return f(*args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py",
line 4677, in do_shelve_instance
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server self._shelve_instance(context, instance,
image_id, clean_shutdown)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py",
line 4711, in _shelve_instance
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server self._power_off_instance(context, instance,
clean_shutdown)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py",
line 2318, in _power_off_instance
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server self.driver.power_off(instance, timeout,
retry_interval)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py",
line 2684, in power_off
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server self._clean_shutdown(instance, timeout,
retry_interval)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py",
line 2639, in _clean_shutdown
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server guest.shutdown()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/guest.py",
line 589, in shutdown
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server self._domain.shutdown()
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args,
**kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server six.reraise(c, e, tb)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server rv = meth(*args, **kwargs)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 2399, in shutdown
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server if ret == -1: raise libvirtError
('virDomainShutdown() failed', dom=self)
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server libvirtError: Requested operation is not valid:
domain is not running
Jan 26 10:13:10 localhost.localdomain nova-compute[8304]: ERROR
oslo_messaging.rpc.server
Jan 26 10:13:16 localhost.localdomain nova-compute[8304]: DEBUG
oslo_service.periodic_task [None req-54849f29-2fd5-4ece-976b-bf4b2c5fd169 admin
admin] Running periodic task ComputeManager._poll_rebooting_instances
{{(pid=8304) run_periodic_tasks
/usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215}}
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1745529/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
