Public bug reported: There appears to be a scenario where amqp NotFound exceptions are being propagated up through oslo messaging to the Neutron RPC client attempting to perform a cast.
Since this appears in a grenade job, my guess is that it's due to the agents no longer listening on that particular exchange after they have been upgraded (the "port-delete" exchange is no longer used by agents in master). In neutron we probably need to ignore these exceptions on casts since it just means there are no longer agents interested in listening to the casts. However, this also seems like it may be an oslo messaging bug because in order for us to catch and ignore this in Neutron we would need to import AMQP directly for the exception. Directly depending on AMQP imports breaks the olso.messaging abstraction. >From http://logs.openstack.org/85/480185/4/check/gate-grenade-dsvm- ironic-inspector-ubuntu- xenial/6ef1e75/logs/new/screen-q-svc.txt.gz?level=ERROR#_2017-07-19_12_53_37_858 : 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation [req-8d272d85-8fc0-4aeb-9180-2bf92cfc67cc tempest-NetworksTest-1317585658 tempest-NetworksTest-1317585658] DELETE failed.: NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/' 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 683, in __call__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 574, in invoke_controller 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 70, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 109, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/resource.py", line 78, in delete 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return self.plugin_deleter(*deleter_args) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 929, in delete_network 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return super(Ml2Plugin, self).delete_network(context, id) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 442, in delete_network 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.delete_port(context.elevated(), port_id) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1479, in delete_port 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation context, port, router_ids, bound_mech_contexts) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1501, in _post_delete_port 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.notifier.port_delete(context, port['id']) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/rpc.py", line 412, in port_delete 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation cctxt.cast(context, 'port_delete', port_id=port_id) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 147, in cast 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.transport._send(self.target, msg_ctxt, msg, retry=self.retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation timeout=timeout, retry=retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation retry=retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 552, in _send 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation conn.fanout_send(target.topic, msg, retry=retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1286, in fanout_send 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self._ensure_publishing(self._publish, exchange, msg, retry=retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.ensure(method, retry=retry, error_callback=_error_callback) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ret, channel = autoretry_method() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 494, in _ensured 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return fun(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 570, in __call__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return fun(*args, channel=channels[0], **kwargs), channels[0] 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation method() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1192, in _publish 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation compression=self.kombu_compression) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 181, in publish 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation exchange_name, declare, 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in _publish 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation mandatory=mandatory, immediate=immediate, 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1759, in basic_publish_confirm 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.wait(spec.Basic.Ack) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in wait 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.connection.drain_events(timeout=timeout) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 485, in drain_events 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation while not self.blocking_read(timeout): 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 491, in blocking_read 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return self.on_inbound_frame(frame) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 69, in on_frame 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation callback(channel, method_sig, buf, None) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 495, in on_inbound_method 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation method_sig, payload, content, 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 142, in dispatch_method 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation listener(*args) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 293, in _on_close 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation reply_code, reply_text, (class_id, method_id), ChannelError, 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/' 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ** Affects: neutron Importance: High Status: Confirmed ** Affects: oslo.messaging Importance: Undecided Status: New ** Changed in: neutron Status: New => Confirmed ** Changed in: neutron Importance: Undecided => High ** Also affects: oslo.messaging Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1705351 Title: agent notifier getting amqp NotFound exceptions propagated up to it Status in neutron: Confirmed Status in oslo.messaging: New Bug description: There appears to be a scenario where amqp NotFound exceptions are being propagated up through oslo messaging to the Neutron RPC client attempting to perform a cast. Since this appears in a grenade job, my guess is that it's due to the agents no longer listening on that particular exchange after they have been upgraded (the "port-delete" exchange is no longer used by agents in master). In neutron we probably need to ignore these exceptions on casts since it just means there are no longer agents interested in listening to the casts. However, this also seems like it may be an oslo messaging bug because in order for us to catch and ignore this in Neutron we would need to import AMQP directly for the exception. Directly depending on AMQP imports breaks the olso.messaging abstraction. From http://logs.openstack.org/85/480185/4/check/gate-grenade-dsvm- ironic-inspector-ubuntu- xenial/6ef1e75/logs/new/screen-q-svc.txt.gz?level=ERROR#_2017-07-19_12_53_37_858 : 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation [req-8d272d85-8fc0-4aeb-9180-2bf92cfc67cc tempest-NetworksTest-1317585658 tempest-NetworksTest-1317585658] DELETE failed.: NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/' 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last): 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 683, in __call__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 574, in invoke_controller 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 70, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 109, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/resource.py", line 78, in delete 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return self.plugin_deleter(*deleter_args) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 929, in delete_network 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return super(Ml2Plugin, self).delete_network(context, id) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 442, in delete_network 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.delete_port(context.elevated(), port_id) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1479, in delete_port 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation context, port, router_ids, bound_mech_contexts) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1501, in _post_delete_port 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.notifier.port_delete(context, port['id']) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/rpc.py", line 412, in port_delete 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation cctxt.cast(context, 'port_delete', port_id=port_id) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 147, in cast 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.transport._send(self.target, msg_ctxt, msg, retry=self.retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation timeout=timeout, retry=retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation retry=retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 552, in _send 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation conn.fanout_send(target.topic, msg, retry=retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1286, in fanout_send 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self._ensure_publishing(self._publish, exchange, msg, retry=retry) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.ensure(method, retry=retry, error_callback=_error_callback) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ret, channel = autoretry_method() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 494, in _ensured 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return fun(*args, **kwargs) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 570, in __call__ 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return fun(*args, channel=channels[0], **kwargs), channels[0] 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation method() 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1192, in _publish 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation compression=self.kombu_compression) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 181, in publish 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation exchange_name, declare, 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in _publish 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation mandatory=mandatory, immediate=immediate, 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1759, in basic_publish_confirm 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.wait(spec.Basic.Ack) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in wait 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.connection.drain_events(timeout=timeout) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 485, in drain_events 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation while not self.blocking_read(timeout): 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 491, in blocking_read 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return self.on_inbound_frame(frame) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 69, in on_frame 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation callback(channel, method_sig, buf, None) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 495, in on_inbound_method 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation method_sig, payload, content, 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 142, in dispatch_method 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation listener(*args) 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 293, in _on_close 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation reply_code, reply_text, (class_id, method_id), ChannelError, 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/' 2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1705351/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
