Looks like I misread that patch below, it's adding back the channel
error check, not the connection error check.
This may be due to a bad patch on our end, sorry for the noise.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1296972
Title:
RPC code in Havana doesn't handle connection errors
Status in OpenStack Compute (Nova):
Invalid
Bug description:
We've got an HA controller setup using pacemaker and were stress-
testing it by doing multiple controlled switchovers while doing other
activity. Generally this works okay, but last night we ran into a
problem where nova-compute got into a state where it was unable to
reconnect with the AMQP server. Logs are at the bottom, they repeat
every minute and did this for 7+ hours until the system was manually
cleaned up.
I've found something in the code that looks a bit suspicious. The
"Unexpected exception occurred 61 time(s)... retrying." message comes from
forever_retry_uncaught_exceptions() in excutils.py. It looks like we're raising
RecoverableConnectionError: connection already closed
down in /usr/lib64/python2.7/site-packages/amqp/abstract_channel.py,
but nothing handles it.
It looks like the most likely place that should be handling it is
nova.openstack.common.rpc.impl_kombu.Connection.ensure().
In the current oslo.messaging code the ensure() routine explicitly
handles connection errors (which RecoverableConnectionError is) and
socket timeouts--the ensure() routine in Havana doesn't do this.
Maybe we should look at porting
https://github.com/openstack/oslo.messaging/commit/0400cbf4f83cf8d58076c7e65e08a156ec3508a8
to the Havana RPC code?
Logs showing the start of the problem and the first few iterations of
the repeating issue:
2014-03-24 09:24:33.566 6620 AUDIT nova.compute.resource_tracker [-] Auditing
locally available compute resources
2014-03-24 09:24:34.126 6620 INFO nova.compute.resource_tracker [-] DETAIL:
instance: name=u'sgw-4', vm_state=u'active', task_state=None, vcpus=2,
cpuset=0x180, cpulist=[7, 8] pinned, nodelist=[0], node=0
2014-03-24 09:24:34.126 6620 INFO nova.compute.resource_tracker [-] DETAIL:
instance: name=u'sgw-1', vm_state=u'active', task_state=None, vcpus=2,
cpuset=0x60, cpulist=[5, 6] pinned, nodelist=[0], node=0
2014-03-24 09:24:34.126 6620 INFO nova.compute.resource_tracker [-] DETAIL:
instance: name=u'load_balancer', vm_state=u'active', task_state=None, vcpus=3,
cpuset=0x1c00, cpulist=[10, 11, 12] pinned, nodelist=[1], node=1
2014-03-24 09:24:34.182 6620 AUDIT nova.compute.resource_tracker [-] Free ram
(MB): 111290, per-node: [52286, 59304], numa nodes:2
2014-03-24 09:24:34.183 6620 AUDIT nova.compute.resource_tracker [-] Free
disk (GB): 29
2014-03-24 09:24:34.183 6620 AUDIT nova.compute.resource_tracker [-] Free
vcpus: 170, free per-node float vcpus: [48, 112], free per-node pinned vcpus:
[3, 7]
2014-03-24 09:24:34.183 6620 INFO nova.compute.resource_tracker [-] DETAIL:
vcpus:20, Free vcpus:170, 16.0x overcommit, per-cpu float cpulist: [3, 4, 9,
13, 14, 15, 16, 17, 18, 19]
2014-03-24 09:24:34.244 6620 INFO nova.compute.resource_tracker [-]
Compute_service record updated for compute-0:compute-0
2014-03-24 09:25:36.564 6620 AUDIT nova.compute.resource_tracker [-] Auditing
locally available compute resources
2014-03-24 09:25:37.122 6620 INFO nova.compute.resource_tracker [-] DETAIL:
instance: name=u'sgw-4', vm_state=u'active', task_state=None, vcpus=2,
cpuset=0x180, cpulist=[7, 8] pinned, nodelist=[0], node=0
2014-03-24 09:25:37.122 6620 INFO nova.compute.resource_tracker [-] DETAIL:
instance: name=u'sgw-1', vm_state=u'active', task_state=None, vcpus=2,
cpuset=0x60, cpulist=[5, 6] pinned, nodelist=[0], node=0
2014-03-24 09:25:37.122 6620 INFO nova.compute.resource_tracker [-] DETAIL:
instance: name=u'load_balancer', vm_state=u'active', task_state=None, vcpus=3,
cpuset=0x1c00, cpulist=[10, 11, 12] pinned, nodelist=[1], node=1
2014-03-24 09:25:37.182 6620 AUDIT nova.compute.resource_tracker [-] Free ram
(MB): 111290, per-node: [52286, 59304], numa nodes:2
2014-03-24 09:25:37.182 6620 AUDIT nova.compute.resource_tracker [-] Free
disk (GB): 29
2014-03-24 09:25:37.183 6620 AUDIT nova.compute.resource_tracker [-] Free
vcpus: 170, free per-node float vcpus: [48, 112], free per-node pinned vcpus:
[3, 7]
2014-03-24 09:25:37.183 6620 INFO nova.compute.resource_tracker [-] DETAIL:
vcpus:20, Free vcpus:170, 16.0x overcommit, per-cpu float cpulist: [3, 4, 9,
13, 14, 15, 16, 17, 18, 19]
2014-03-24 09:25:37.245 6620 INFO nova.compute.resource_tracker [-]
Compute_service record updated for compute-0:compute-0
2014-03-24 09:26:47.324 6620 ERROR root [-] Unexpected exception occurred 1
time(s)... retrying.
2014-03-24 09:26:47.324 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:26:47.324 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line
78, in inner_func
2014-03-24 09:26:47.324 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 745, in _consumer_thread
2014-03-24 09:26:47.324 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 736, in consume
2014-03-24 09:26:47.324 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 663, in iterconsume
2014-03-24 09:26:47.324 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 578, in ensure
2014-03-24 09:26:47.324 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 658, in _consume
2014-03-24 09:26:47.324 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/kombu/connection.py", line 279, in
drain_events
2014-03-24 09:26:47.324 6620 TRACE root return
self.transport.drain_events(self.connection, **kwargs)
2014-03-24 09:26:47.324 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/kombu/transport/pyamqp.py", line 90, in
drain_events
2014-03-24 09:26:47.324 6620 TRACE root return
connection.drain_events(**kwargs)
2014-03-24 09:26:47.324 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/connection.py", line 303, in
drain_events
2014-03-24 09:26:47.324 6620 TRACE root return amqp_method(channel, args)
2014-03-24 09:26:47.324 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/connection.py", line 509, in _close
2014-03-24 09:26:47.324 6620 TRACE root (class_id, method_id),
ConnectionError)
2014-03-24 09:26:47.324 6620 TRACE root ConnectionForced: (0, 0): (320)
CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'
2014-03-24 09:26:47.324 6620 TRACE root
2014-03-24 09:26:47.366 6620 ERROR nova.openstack.common.rpc.common [-]
Failed to consume message from queue: [Errno 104] Connection reset by peer
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common Traceback
(most recent call last):
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 578, in ensure
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 658, in _consume
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/kombu/connection.py", line 279, in
drain_events
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
return self.transport.drain_events(self.connection, **kwargs)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/kombu/transport/pyamqp.py", line 90, in
drain_events
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
return connection.drain_events(**kwargs)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/connection.py", line 303, in
drain_events
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
return amqp_method(channel, args)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/connection.py", line 506, in _close
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
self._x_close_ok()
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/connection.py", line 534, in
_x_close_ok
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
self._send_method((10, 51))
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 62, in
_send_method
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
self.channel_id, method_sig, args, content,
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/method_framing.py", line 227, in
write_method
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
write_frame(1, channel, payload)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/transport.py", line 183, in write_frame
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
frame_type, channel, size, payload, 0xce,
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 309, in sendall
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common tail
= self.send(data, flags)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 295, in send
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
total_sent += fd.send(data[total_sent:], flags)
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common error:
[Errno 104] Connection reset by peer
2014-03-24 09:26:47.366 6620 TRACE nova.openstack.common.rpc.common
2014-03-24 09:26:47.368 6620 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:47.378 6620 ERROR nova.openstack.common.rpc.common [-] AMQP
server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying
again in 1 seconds.
2014-03-24 09:26:48.103 6620 ERROR nova.openstack.common.rpc.common [-]
Failed to publish message to topic 'conductor': [Errno 104] Connection reset by
peer
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common Traceback
(most recent call last):
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 578, in ensure
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 689, in _publish
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 369, in __init__
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 316, in __init__
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 324, in reconnect
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 82, in __init__
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
self.revive(self._channel)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 216, in revive
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
self.declare()
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/kombu/messaging.py", line 102, in declare
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
self.exchange.declare()
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/kombu/entity.py", line 166, in declare
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
nowait=nowait, passive=passive,
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/channel.py", line 604, in
exchange_declare
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
self._send_method((40, 10), args)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 62, in
_send_method
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
self.channel_id, method_sig, args, content,
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/method_framing.py", line 227, in
write_method
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
write_frame(1, channel, payload)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/amqp/transport.py", line 183, in write_frame
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
frame_type, channel, size, payload, 0xce,
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 309, in sendall
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common tail
= self.send(data, flags)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common File
"/usr/lib64/python2.7/site-packages/eventlet/greenio.py", line 295, in send
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
total_sent += fd.send(data[total_sent:], flags)
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common error:
[Errno 104] Connection reset by peer
2014-03-24 09:26:48.103 6620 TRACE nova.openstack.common.rpc.common
2014-03-24 09:26:48.105 6620 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:48.112 6620 ERROR nova.openstack.common.rpc.common [-] AMQP
server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying
again in 1 seconds.
2014-03-24 09:26:48.327 6620 ERROR root [-] Unexpected exception occurred 1
time(s)... retrying.
2014-03-24 09:26:48.327 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:26:48.327 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line
78, in inner_func
2014-03-24 09:26:48.327 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 745, in _consumer_thread
2014-03-24 09:26:48.327 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 736, in consume
2014-03-24 09:26:48.327 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 663, in iterconsume
2014-03-24 09:26:48.327 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 578, in ensure
2014-03-24 09:26:48.327 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 656, in _consume
2014-03-24 09:26:48.327 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 191, in consume
2014-03-24 09:26:48.327 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/kombu/entity.py", line 609, in consume
2014-03-24 09:26:48.327 6620 TRACE root nowait=nowait)
2014-03-24 09:26:48.327 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/channel.py", line 1775, in
basic_consume
2014-03-24 09:26:48.327 6620 TRACE root self._send_method((60, 20), args)
2014-03-24 09:26:48.327 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in
_send_method
2014-03-24 09:26:48.327 6620 TRACE root raise
RecoverableConnectionError('connection already closed')
2014-03-24 09:26:48.327 6620 TRACE root RecoverableConnectionError:
connection already closed
2014-03-24 09:26:48.327 6620 TRACE root
2014-03-24 09:26:48.379 6620 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:48.386 6620 ERROR nova.openstack.common.rpc.common [-] AMQP
server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying
again in 3 seconds.
2014-03-24 09:26:49.112 6620 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:49.120 6620 ERROR nova.openstack.common.rpc.common [-] AMQP
server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying
again in 3 seconds.
2014-03-24 09:26:51.386 6620 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:51.402 6620 ERROR nova.openstack.common.rpc.common [-] AMQP
server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying
again in 5 seconds.
2014-03-24 09:26:52.120 6620 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:52.127 6620 ERROR nova.openstack.common.rpc.common [-] AMQP
server on 192.168.204.2:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying
again in 5 seconds.
2014-03-24 09:26:56.408 6620 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:56.454 6620 INFO nova.openstack.common.rpc.common [-]
Connected to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:57.128 6620 INFO nova.openstack.common.rpc.common [-]
Reconnecting to AMQP server on 192.168.204.2:5672
2014-03-24 09:26:57.137 6620 INFO nova.openstack.common.rpc.common [-]
Connected to AMQP server on 192.168.204.2:5672
2014-03-24 09:27:39.807 6620 ERROR nova.openstack.common.periodic_task [-]
Error during ComputeManager._instance_usage_audit: Timeout while waiting on RPC
response - topic: "conductor", RPC method: "task_log_get" info: "<unknown>"
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task
Traceback (most recent call last):
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py",
line 180, in run_periodic_tasks
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 4623, in
_instance_usage_audit
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/compute/utils.py", line 363, in
has_audit_been_run
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/conductor/api.py", line 273, in
task_log_get
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 403, in
task_log_get
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 85, in call
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/proxy.py", line
130, in call
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task
Timeout: Timeout while waiting on RPC response - topic: "conductor", RPC
method: "task_log_get" info: "<unknown>"
2014-03-24 09:27:39.807 6620 TRACE nova.openstack.common.periodic_task
2014-03-24 09:27:39.809 6620 AUDIT nova.compute.resource_tracker [-] Auditing
locally available compute resources
2014-03-24 09:27:49.335 6620 ERROR root [-] Unexpected exception occurred 61
time(s)... retrying.
2014-03-24 09:27:49.335 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:27:49.335 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line
78, in inner_func
2014-03-24 09:27:49.335 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 745, in _consumer_thread
2014-03-24 09:27:49.335 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 736, in consume
2014-03-24 09:27:49.335 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 663, in iterconsume
2014-03-24 09:27:49.335 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 578, in ensure
2014-03-24 09:27:49.335 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 656, in _consume
2014-03-24 09:27:49.335 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 191, in consume
2014-03-24 09:27:49.335 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/kombu/entity.py", line 609, in consume
2014-03-24 09:27:49.335 6620 TRACE root nowait=nowait)
2014-03-24 09:27:49.335 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/channel.py", line 1775, in
basic_consume
2014-03-24 09:27:49.335 6620 TRACE root self._send_method((60, 20), args)
2014-03-24 09:27:49.335 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in
_send_method
2014-03-24 09:27:49.335 6620 TRACE root raise
RecoverableConnectionError('connection already closed')
2014-03-24 09:27:49.335 6620 TRACE root RecoverableConnectionError:
connection already closed
2014-03-24 09:27:49.335 6620 TRACE root
2014-03-24 09:27:57.139 6620 ERROR nova.servicegroup.drivers.db [-] model
server went away
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db Traceback
(most recent call last):
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File
"./usr/lib64/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 98,
in _report_state
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File
"./usr/lib64/python2.7/site-packages/nova/conductor/api.py", line 269, in
service_update
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File
"./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 397, in
service_update
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File
"./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 85, in call
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File
"./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/proxy.py", line
130, in call
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db Timeout:
Timeout while waiting on RPC response - topic: "conductor", RPC method:
"service_update" info: "<unknown>"
2014-03-24 09:27:57.139 6620 TRACE nova.servicegroup.drivers.db
2014-03-24 09:27:57.141 6620 WARNING nova.openstack.common.loopingcall [-]
task run outlasted interval by 59.038519 sec
2014-03-24 09:28:40.346 6620 ERROR nova.openstack.common.periodic_task [-]
Error during ComputeManager.update_available_resource: Timeout while waiting on
RPC response - topic: "conductor", RPC method: "object_class_action" info:
"<unknown>"
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task
Traceback (most recent call last):
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py",
line 180, in run_periodic_tasks
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 5024, in
update_available_resource
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/lockutils.py", line
246, in inner
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/compute/resource_tracker.py", line
495, in update_available_resource
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 106, in wrapper
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 492, in
object_class_action
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 85, in call
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/proxy.py", line
130, in call
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task
Timeout: Timeout while waiting on RPC response - topic: "conductor", RPC
method: "object_class_action" info: "<unknown>"
2014-03-24 09:28:40.346 6620 TRACE nova.openstack.common.periodic_task
2014-03-24 09:28:50.342 6620 ERROR root [-] Unexpected exception occurred 61
time(s)... retrying.
2014-03-24 09:28:50.342 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:28:50.342 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line
78, in inner_func
2014-03-24 09:28:50.342 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 745, in _consumer_thread
2014-03-24 09:28:50.342 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 736, in consume
2014-03-24 09:28:50.342 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 663, in iterconsume
2014-03-24 09:28:50.342 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 578, in ensure
2014-03-24 09:28:50.342 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 656, in _consume
2014-03-24 09:28:50.342 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 191, in consume
2014-03-24 09:28:50.342 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/kombu/entity.py", line 609, in consume
2014-03-24 09:28:50.342 6620 TRACE root nowait=nowait)
2014-03-24 09:28:50.342 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/channel.py", line 1775, in
basic_consume
2014-03-24 09:28:50.342 6620 TRACE root self._send_method((60, 20), args)
2014-03-24 09:28:50.342 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in
_send_method
2014-03-24 09:28:50.342 6620 TRACE root raise
RecoverableConnectionError('connection already closed')
2014-03-24 09:28:50.342 6620 TRACE root RecoverableConnectionError:
connection already closed
2014-03-24 09:28:50.342 6620 TRACE root
2014-03-24 09:28:57.144 6620 WARNING nova.openstack.common.loopingcall [-]
task run outlasted interval by 50.00323 sec
2014-03-24 09:29:40.351 6620 ERROR nova.openstack.common.periodic_task [-]
Error during ComputeManager._run_pending_deletes: Timeout while waiting on RPC
response - topic: "conductor", RPC method: "object_class_action" info:
"<unknown>"
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task
Traceback (most recent call last):
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/periodic_task.py",
line 180, in run_periodic_tasks
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/compute/manager.py", line 5239, in
_run_pending_deletes
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/objects/base.py", line 106, in wrapper
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/conductor/rpcapi.py", line 492, in
object_class_action
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 85, in call
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/rpcclient.py", line 63, in _invoke
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/proxy.py", line
130, in call
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task
Timeout: Timeout while waiting on RPC response - topic: "conductor", RPC
method: "object_class_action" info: "<unknown>"
2014-03-24 09:29:40.351 6620 TRACE nova.openstack.common.periodic_task
2014-03-24 09:29:51.349 6620 ERROR root [-] Unexpected exception occurred 61
time(s)... retrying.
2014-03-24 09:29:51.349 6620 TRACE root Traceback (most recent call last):
2014-03-24 09:29:51.349 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/excutils.py", line
78, in inner_func
2014-03-24 09:29:51.349 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 745, in _consumer_thread
2014-03-24 09:29:51.349 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 736, in consume
2014-03-24 09:29:51.349 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 663, in iterconsume
2014-03-24 09:29:51.349 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 578, in ensure
2014-03-24 09:29:51.349 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 656, in _consume
2014-03-24 09:29:51.349 6620 TRACE root File
"./usr/lib64/python2.7/site-packages/nova/openstack/common/rpc/impl_kombu.py",
line 191, in consume
2014-03-24 09:29:51.349 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/kombu/entity.py", line 609, in consume
2014-03-24 09:29:51.349 6620 TRACE root nowait=nowait)
2014-03-24 09:29:51.349 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/channel.py", line 1775, in
basic_consume
2014-03-24 09:29:51.349 6620 TRACE root self._send_method((60, 20), args)
2014-03-24 09:29:51.349 6620 TRACE root File
"/usr/lib64/python2.7/site-packages/amqp/abstract_channel.py", line 56, in
_send_method
2014-03-24 09:29:51.349 6620 TRACE root raise
RecoverableConnectionError('connection already closed')
2014-03-24 09:29:51.349 6620 TRACE root RecoverableConnectionError:
connection already closed
2014-03-24 09:29:51.349 6620 TRACE root
2014-03-24 09:29:57.147 6620 WARNING nova.openstack.common.loopingcall [-]
task run outlasted interval by 50.002704 sec
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1296972/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
