** Also affects: oslo.messaging
Importance: Undecided
Status: New
** No longer affects: nova
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1418790
Title:
rpc call is deadlocked when message broker is restarted
Status in Messaging API for OpenStack:
New
Bug description:
When I need to restart my message broker, qpidd, the rpc call from
nova-compute may be discrupted and blocked forever.
Sometimes in service_update and sometimes in periodic_tasks. The blocked call
trace of the greenthread is here:
Feb 06 09:28:50 computer01 nova-compute[21244]: ------
Green Thread ------
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py:78 in
_inner
Feb 06 09:28:50 computer01 nova-compute[21244]: `self.f(*self.args,
**self.kw)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py:115 in
_report_stat
Feb 06 09:28:50 computer01 nova-compute[21244]: `service.service_ref,
state_catalog)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/nova/conductor/api.py:218 in service_update
Feb 06 09:28:50 computer01 nova-compute[21244]: `return
self._manager.service_update(context, service, values)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py:330 in service_update
Feb 06 09:28:50 computer01 nova-compute[21244]: `service=service_p,
values=values)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py:150 in call
Feb 06 09:28:50 computer01 nova-compute[21244]: `wait_for_reply=True,
timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/transport.py:90 in _send
Feb 06 09:28:50 computer01 nova-compute[21244]: `timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:412 in
send
Feb 06 09:28:50 computer01 nova-compute[21244]: `return self._send(target,
ctxt, message, wait_for_reply, timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:403 in
_send
Feb 06 09:28:50 computer01 nova-compute[21244]: `result =
self._waiter.wait(msg_id, timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:267 in
wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `reply, ending =
self._poll_connection(msg_id, timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:213 in
_poll_
Feb 06 09:28:50 computer01 nova-compute[21244]: `self.conn.consume(limit=1,
timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:676 in
consume
Feb 06 09:28:50 computer01 nova-compute[21244]: `six.next(it)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:608 in
itercon
Feb 06 09:28:50 computer01 nova-compute[21244]: `yield
self.ensure(_error_callback, _consume)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:546 in
ensure
Feb 06 09:28:50 computer01 nova-compute[21244]: `return method(*args,
**kwargs)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:599 in
_consum
Feb 06 09:28:50 computer01 nova-compute[21244]: `nxt_receiver =
self.session.next_receiver(timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]: <string>:6 in next_receiver
Feb 06 09:28:50 computer01 nova-compute[21244]: (source not found)
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in
next_receiver
Feb 06 09:28:50 computer01 nova-compute[21244]: `if self._ecwait(lambda:
self.incoming, timeout):`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait
Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self._ewait(lambda:
self.closed or predicate(), timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait
Feb 06 09:28:50 computer01 nova-compute[21244]: `result =
self.connection._ewait(lambda: self.error or predicate(), timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait
Feb 06 09:28:50 computer01 nova-compute[21244]: `result = self._wait(lambda:
self.error or predicate(), timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `return
self._waiter.wait(predicate, timeout=timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/qpid/concurrency.py:59 in wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `self.condition.wait(timeout
- passed)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `sw.wait(timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait
Feb 06 09:28:50 computer01 nova-compute[21244]: `ready, _, _ = select([self],
[], [], timeout)`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/eventlet/green/select.py:79 in select
Feb 06 09:28:50 computer01 nova-compute[21244]: `return hub.switch()`
Feb 06 09:28:50 computer01 nova-compute[21244]:
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:187 in switch
Feb 06 09:28:50 computer01 nova-compute[21244]: `return
self.greenlet.switch()`
Feb 05 15:46:29 computer02 nova-compute[20264]: ------
Green Thread ------
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/eventlet/greenthread.py:194 in main
Feb 05 15:46:29 computer02 nova-compute[20264]: `result = function(*args,
**kwargs)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py:125 in
_inner
Feb 05 15:46:29 computer02 nova-compute[20264]: `idle = self.f(*self.args,
**self.kw)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/nova/service.py:308 in periodic_tasks
Feb 05 15:46:29 computer02 nova-compute[20264]: `return
self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/nova/manager.py:81 in periodic_tasks
Feb 05 15:46:29 computer02 nova-compute[20264]: `return
self.run_periodic_tasks(context, raise_on_error=raise_on_error)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py:182 in
run_p
Feb 05 15:46:29 computer02 nova-compute[20264]: `task(self, context)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/nova/compute/manager.py:5322 in
_sync_power_states
Feb 05 15:46:29 computer02 nova-compute[20264]: `use_slave=True)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/nova/objects/base.py:110 in wrapper
Feb 05 15:46:29 computer02 nova-compute[20264]: `args, kwargs)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py:425 in
object_class_action
Feb 05 15:46:29 computer02 nova-compute[20264]: `objver=objver, args=args,
kwargs=kwargs)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py:150 in call
Feb 05 15:46:29 computer02 nova-compute[20264]: `wait_for_reply=True,
timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/transport.py:90 in _send
Feb 05 15:46:29 computer02 nova-compute[20264]: `timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:412 in
send
Feb 05 15:46:29 computer02 nova-compute[20264]: `return self._send(target,
ctxt, message, wait_for_reply, timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:403 in
_send
Feb 05 15:46:29 computer02 nova-compute[20264]: `result =
self._waiter.wait(msg_id, timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:267 in
wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `reply, ending =
self._poll_connection(msg_id, timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:213 in
_poll_
Feb 05 15:46:29 computer02 nova-compute[20264]: `self.conn.consume(limit=1,
timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:676 in
consume
Feb 05 15:46:29 computer02 nova-compute[20264]: `six.next(it)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:608 in
itercon
Feb 05 15:46:29 computer02 nova-compute[20264]: `yield
self.ensure(_error_callback, _consume)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:546 in
ensure
Feb 05 15:46:29 computer02 nova-compute[20264]: `return method(*args,
**kwargs)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/impl_qpid.py:599 in
_consum
Feb 05 15:46:29 computer02 nova-compute[20264]: `nxt_receiver =
self.session.next_receiver(timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]: <string>:6 in next_receiver
Feb 05 15:46:29 computer02 nova-compute[20264]: (source not found)
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:660 in
next_receiver
Feb 05 15:46:29 computer02 nova-compute[20264]: `if self._ecwait(lambda:
self.incoming, timeout):`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:50 in _ecwait
Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self._ewait(lambda:
self.closed or predicate(), timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:566 in _ewait
Feb 05 15:46:29 computer02 nova-compute[20264]: `result =
self.connection._ewait(lambda: self.error or predicate(), timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:208 in _ewait
Feb 05 15:46:29 computer02 nova-compute[20264]: `result = self._wait(lambda:
self.error or predicate(), timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py:193 in _wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `return
self._waiter.wait(predicate, timeout=timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/qpid/concurrency.py:59 in wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `self.condition.wait(timeout
- passed)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/qpid/concurrency.py:96 in wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `sw.wait(timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/qpid/compat.py:53 in wait
Feb 05 15:46:29 computer02 nova-compute[20264]: `ready, _, _ = select([self],
[], [], timeout)`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/eventlet/green/select.py:79 in select
Feb 05 15:46:29 computer02 nova-compute[20264]: `return hub.switch()`
Feb 05 15:46:29 computer02 nova-compute[20264]:
/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py:187 in switch
Feb 05 15:46:29 computer02 nova-compute[20264]: `return
self.greenlet.switch()`
To manage notifications about this bug go to:
https://bugs.launchpad.net/oslo.messaging/+bug/1418790/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
