[Yahoo-eng-team] [Bug 1393391] [NEW] neutron-openvswitch-agent stuck on no queue 'q-agent-notifier-port-update_fanout..

Miguel Angel Ajo Mon, 17 Nov 2014 04:11:48 -0800

Public bug reported:

Under some conditions -TBD-, (port being created, and port being deleted?), 
neutron-openvswitch-agent can get stuck 
trying to subscribe a fanout queue,


It stops responding to any other messages, so it stops effectively
working at all.


2014-11-17 06:15:55.842 3042 ERROR root [-] Unexpected exception occurred 61 
time(s)... retrying.
2014-11-17 06:15:55.842 3042 TRACE root Traceback (most recent call last):
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 
92, in inner_func
2014-11-17 06:15:55.842 3042 TRACE root     return infunc(*args, **kwargs)
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 746, in _consumer_thread
2014-11-17 06:15:55.842 3042 TRACE root     self.consume()
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 737, in consume
2014-11-17 06:15:55.842 3042 TRACE root     six.next(it)
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 664, in iterconsume
2014-11-17 06:15:55.842 3042 TRACE root     yield self.ensure(_error_callback, 
_consume)
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 579, in ensure
2014-11-17 06:15:55.842 3042 TRACE root     return method(*args, **kwargs)
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 657, in _consume
2014-11-17 06:15:55.842 3042 TRACE root     queues_tail.consume(nowait=False)
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 190, in consume
2014-11-17 06:15:55.842 3042 TRACE root     self.queue.consume(*args, 
callback=_callback, **options)
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/kombu/entity.py", line 598, in consume
2014-11-17 06:15:55.842 3042 TRACE root     nowait=nowait)
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 1769, in basic_consume
2014-11-17 06:15:55.842 3042 TRACE root     (60, 21),  # 
Channel.basic_consume_ok
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 71, in wait
2014-11-17 06:15:55.842 3042 TRACE root     return 
self.dispatch_method(method_sig, args, content)
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 88, in 
dispatch_method
2014-11-17 06:15:55.842 3042 TRACE root     return amqp_method(self, args)
2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 224, in _close
2014-11-17 06:15:55.842 3042 TRACE root     raise ChannelError(reply_code, 
reply_text, (class_id, method_id))
2014-11-17 06:15:55.842 3042 TRACE root ChannelError: 404: (NOT_FOUND - no 
queue 'q-agent-notifier-port-update_fanout_c5de1f41f554475492301e03557cdd66' in 
vhost '/', (60, 20), None)
2014-11-17 06:15:55.842 3042 TRACE root
s /var/log/neutron/os /var/log/neutron/openvswitch-agent.log
[root@mrg-03 ~]# tail /var/log/neutron/openvswitch-agent.log &2014-11-17 
06:16:56.085 3042 ERROR root [-] Unexpected exception occurred 60 time(s)... 
retrying.
2014-11-17 06:16:56.085 3042 TRACE root Traceback (most recent call last):
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 
92, in inner_func
2014-11-17 06:16:56.085 3042 TRACE root     return infunc(*args, **kwargs)
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 746, in _consumer_thread
2014-11-17 06:16:56.085 3042 TRACE root     self.consume()
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 737, in consume
2014-11-17 06:16:56.085 3042 TRACE root     six.next(it)
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 664, in iterconsume
2014-11-17 06:16:56.085 3042 TRACE root     yield self.ensure(_error_callback, 
_consume)
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 579, in ensure
2014-11-17 06:16:56.085 3042 TRACE root     return method(*args, **kwargs)
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 657, in _consume
2014-11-17 06:16:56.085 3042 TRACE root     queues_tail.consume(nowait=False)
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 190, in consume
2014-11-17 06:16:56.085 3042 TRACE root     self.queue.consume(*args, 
callback=_callback, **options)
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/kombu/entity.py", line 598, in consume
2014-11-17 06:16:56.085 3042 TRACE root     nowait=nowait)
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 1769, in basic_consume
2014-11-17 06:16:56.085 3042 TRACE root     (60, 21),  # 
Channel.basic_consume_ok
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 71, in wait
2014-11-17 06:16:56.085 3042 TRACE root     return 
self.dispatch_method(method_sig, args, content)
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 88, in 
dispatch_method
2014-11-17 06:16:56.085 3042 TRACE root     return amqp_method(self, args)
2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 224, in _close
2014-11-17 06:16:56.085 3042 TRACE root     raise ChannelError(reply_code, 
reply_text, (class_id, method_id))
2014-11-17 06:16:56.085 3042 TRACE root ChannelError: 404: (NOT_FOUND - no 
queue 'q-agent-notifier-port-update_fanout_c5de1f41f554475492301e03557cdd66' in 
vhost '/', (60, 20), None)
2014-11-17 06:16:56.085 3042 TRACE root

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1393391

Title:
  neutron-openvswitch-agent stuck on no queue 'q-agent-notifier-port-
  update_fanout..

Status in OpenStack Neutron (virtual network service):
  New

Bug description:
  Under some conditions -TBD-, (port being created, and port being deleted?), 
neutron-openvswitch-agent can get stuck 
  trying to subscribe a fanout queue, 

  It stops responding to any other messages, so it stops effectively
  working at all.

  
  2014-11-17 06:15:55.842 3042 ERROR root [-] Unexpected exception occurred 61 
time(s)... retrying.
  2014-11-17 06:15:55.842 3042 TRACE root Traceback (most recent call last):
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 
92, in inner_func
  2014-11-17 06:15:55.842 3042 TRACE root     return infunc(*args, **kwargs)
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 746, in _consumer_thread
  2014-11-17 06:15:55.842 3042 TRACE root     self.consume()
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 737, in consume
  2014-11-17 06:15:55.842 3042 TRACE root     six.next(it)
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 664, in iterconsume
  2014-11-17 06:15:55.842 3042 TRACE root     yield 
self.ensure(_error_callback, _consume)
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 579, in ensure
  2014-11-17 06:15:55.842 3042 TRACE root     return method(*args, **kwargs)
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 657, in _consume
  2014-11-17 06:15:55.842 3042 TRACE root     queues_tail.consume(nowait=False)
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 190, in consume
  2014-11-17 06:15:55.842 3042 TRACE root     self.queue.consume(*args, 
callback=_callback, **options)
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/kombu/entity.py", line 598, in consume
  2014-11-17 06:15:55.842 3042 TRACE root     nowait=nowait)
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 1769, in basic_consume
  2014-11-17 06:15:55.842 3042 TRACE root     (60, 21),  # 
Channel.basic_consume_ok
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 71, in wait
  2014-11-17 06:15:55.842 3042 TRACE root     return 
self.dispatch_method(method_sig, args, content)
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 88, in 
dispatch_method
  2014-11-17 06:15:55.842 3042 TRACE root     return amqp_method(self, args)
  2014-11-17 06:15:55.842 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 224, in _close
  2014-11-17 06:15:55.842 3042 TRACE root     raise ChannelError(reply_code, 
reply_text, (class_id, method_id))
  2014-11-17 06:15:55.842 3042 TRACE root ChannelError: 404: (NOT_FOUND - no 
queue 'q-agent-notifier-port-update_fanout_c5de1f41f554475492301e03557cdd66' in 
vhost '/', (60, 20), None)
  2014-11-17 06:15:55.842 3042 TRACE root
  s /var/log/neutron/os /var/log/neutron/openvswitch-agent.log
  [root@mrg-03 ~]# tail /var/log/neutron/openvswitch-agent.log &2014-11-17 
06:16:56.085 3042 ERROR root [-] Unexpected exception occurred 60 time(s)... 
retrying.
  2014-11-17 06:16:56.085 3042 TRACE root Traceback (most recent call last):
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 
92, in inner_func
  2014-11-17 06:16:56.085 3042 TRACE root     return infunc(*args, **kwargs)
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 746, in _consumer_thread
  2014-11-17 06:16:56.085 3042 TRACE root     self.consume()
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 737, in consume
  2014-11-17 06:16:56.085 3042 TRACE root     six.next(it)
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 664, in iterconsume
  2014-11-17 06:16:56.085 3042 TRACE root     yield 
self.ensure(_error_callback, _consume)
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 579, in ensure
  2014-11-17 06:16:56.085 3042 TRACE root     return method(*args, **kwargs)
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 657, in _consume
  2014-11-17 06:16:56.085 3042 TRACE root     queues_tail.consume(nowait=False)
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", 
line 190, in consume
  2014-11-17 06:16:56.085 3042 TRACE root     self.queue.consume(*args, 
callback=_callback, **options)
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/kombu/entity.py", line 598, in consume
  2014-11-17 06:16:56.085 3042 TRACE root     nowait=nowait)
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 1769, in basic_consume
  2014-11-17 06:16:56.085 3042 TRACE root     (60, 21),  # 
Channel.basic_consume_ok
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 71, in wait
  2014-11-17 06:16:56.085 3042 TRACE root     return 
self.dispatch_method(method_sig, args, content)
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 88, in 
dispatch_method
  2014-11-17 06:16:56.085 3042 TRACE root     return amqp_method(self, args)
  2014-11-17 06:16:56.085 3042 TRACE root   File 
"/usr/lib/python2.7/site-packages/amqp/channel.py", line 224, in _close
  2014-11-17 06:16:56.085 3042 TRACE root     raise ChannelError(reply_code, 
reply_text, (class_id, method_id))
  2014-11-17 06:16:56.085 3042 TRACE root ChannelError: 404: (NOT_FOUND - no 
queue 'q-agent-notifier-port-update_fanout_c5de1f41f554475492301e03557cdd66' in 
vhost '/', (60, 20), None)
  2014-11-17 06:16:56.085 3042 TRACE root

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1393391/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1393391] [NEW] neutron-openvswitch-agent stuck on no queue 'q-agent-notifier-port-update_fanout..

Reply via email to