I'm having the same problem - I had a working Havana setup (with
Ceph/rbd as the cinder storage) and within the last few days, whammo,
this starts occuring. I had updated ubuntu 13.10 patches across parts of
the cluster.
nova-compute log from the node with the exception below
2013-11-03 08:37:03.487 1796 ERROR nova.openstack.common.rpc.amqp
[req-16923a1f-cf2b-43fb-bf81-3cdceb698fb0 f2b1d092d8144d\
17aa1c51eecfd067bb 305f4fa9adcc48b09b8d07df5b247df7] Exception during message
handling
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp Traceback
(most recent call last):
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/openstack/\
common/rpc/amqp.py, line 461, in _process_data
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp **args)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/openstack/\
common/rpc/dispatcher.py, line 172, in dispatch
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp result =
getattr(proxyobj, method)(ctxt, **kwargs)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/compute/ma\
nager.py, line 353, in decorated_function
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/exception.\
py, line 90, in wrapped
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp payload)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/exception.\
py, line 73, in wrapped
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return
f(self, context, *args, **kw)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/compute/ma\
nager.py, line 243, in decorated_function
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp pass
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/compute/ma\
nager.py, line 229, in decorated_function
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/compute/ma\
nager.py, line 271, in decorated_function
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp e,
sys.exc_info())
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/compute/ma\
nager.py, line 258, in decorated_function
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp return
function(self, context, *args, **kwargs)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/compute/ma\
nager.py, line 319, in decorated_function
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp %
image_id, instance=instance)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/compute/ma\
nager.py, line 309, in decorated_function
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp *args,
**kwargs)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/compute/ma\
nager.py, line 2293, in snapshot_instance
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp
task_states.IMAGE_SNAPSHOT)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/compute/ma\
nager.py, line 2324, in _snapshot_instance
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp
update_task_state)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/virt/libvi\
rt/driver.py, line 1397, in snapshot
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp
image_format)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/nova/virt/libvi\
rt/driver.py, line 1483, in _live_snapshot
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp
libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/eventlet/tpool.\
py, line 179, in doit
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp result =
proxy_call(self._autowrap, f, *args, **kwar\
gs)
2013-11-03 08:37:03.487 1796 TRACE nova.openstack.common.rpc.amqp File
/usr/lib/python2.7/dist-packages/eventlet/tpool.\
py, line 139, in proxy_call
2013-11-03 08:37:03.487 1796