** Also affects: nova/havana
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1270008
Title:
periodic tasks will be invalid if a qemu process becomes to defunct
status
Status in OpenStack Compute (Nova):
Fix Released
Status in OpenStack Compute (nova) havana series:
New
Bug description:
I am using stable havana nova.
I got this exception while I delete my kvm instance, but the qemu process of
this instance become to 'defunct' status by some unknown reason(may be a
qemu/kvm bug), and then the periodic task stopped unexpectly everytime, then
the resources of this compute node will never be reported, because of this
exception below, I think we should handle this exception while running periodic
task.
2014-01-16 15:53:28.421 47954 ERROR nova.openstack.common.periodic_task [-]
Error during ComputeManager.update_available_resource: cannot get CPU affinity
of process 62279: No such process
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
Traceback (most recent call last):
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py",
line 180, in run_periodic_tasks
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
task(self, context)
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5617, in
update_available_resource
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
rt.update_available_resource(context)
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
line 246, in inner
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
return f(*args, **kwargs)
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line
281, in update_available_resource
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
resources = self.driver.get_available_resource(self.nodename)
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4275,
in get_available_resource
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
stats = self.host_state.get_host_stats(refresh=True)
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5350,
in get_host_stats
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
self.update_status()
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5386,
in update_status
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
data["vcpus_used"] = self.driver.get_vcpu_used()
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3949,
in get_vcpu_used
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
vcpus = dom.vcpus()
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in
proxy_call
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
rv = execute(f,*args,**kwargs)
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
rv = meth(*args,**kwargs)
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2222, in vcpus
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
if ret == -1: raise libvirtError ('virDomainGetVcpus() failed', dom=self)
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
libvirtError: cannot get CPU affinity of process 62279: No such process
2014-01-16 15:53:28.421 47954 TRACE nova.openstack.common.periodic_task
and the exception while I delete this instance:
2014-01-16 15:13:26.640 47954 ERROR nova.openstack.common.rpc.amqp
[req-03ed9463-0740-4423-bf1e-2334ed29ee5c 9537af4d80e546409b670673f9a81388
3179fc9d69d747b4a06f27a6d2334050] Exception during message handling
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp Traceback
(most recent call last):
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461,
in _process_data
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp **args)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 172, in dispatch
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp result
= getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 400, in
decorated_function
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
payload)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp return
f(self, context, *args, **kw)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in
decorated_function
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp pass
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 276, in
decorated_function
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 341, in
decorated_function
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
function(self, context, *args, **kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 318, in
decorated_function
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp e,
sys.exc_info())
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 305, in
decorated_function
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2128, in
terminate_instance
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
do_terminate_instance(instance, bdms, clean_shutdown)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
246, in inner
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp return
f(*args, **kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2120, in
do_terminate_instance
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
reservations=reservations)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/hooks.py", line 105, in inner
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp rv =
f(*args, **kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2091, in
_delete_instance
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
user_id=user_id)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2063, in
_delete_instance
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
clean_shutdown=clean_shutdown)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1984, in
_shutdown_instance
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
requested_networks)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1974, in
_shutdown_instance
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
context=context, clean_shutdown=clean_shutdown)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 883, in
destroy
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
self._destroy(instance)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 838, in
_destroy
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
instance=instance)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 810, in
_destroy
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
virt_dom.destroy()
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp result
= proxy_call(self._autowrap, f, *args, **kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp rv =
execute(f,*args,**kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp rv =
meth(*args,**kwargs)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/libvirt.py", line 760, in destroy
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp if ret
== -1: raise libvirtError ('virDomainDestroy() failed', dom=self)
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
libvirtError: Failed to terminate process 61945 with SIGKILL: Device or
resource busy
2014-01-16 15:13:26.640 47954 TRACE nova.openstack.common.rpc.amqp
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1270008/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
