Public bug reported: Package Versions ================ python-oslo.messaging/bionic,now 5.35.0-0ubuntu1 python-amqp/bionic,now 2.2.2-1
Description =========== We have a customer who keeps experiencing issues with Heat stack creations/deletions. After investigation, we found that Openstack services (neutron, nova, heat itself, etc) can encounter an error with RMQ message passing, where the service involved gets a MessageDeliveryFailure [0] and the stack fails. This failure can manifest at any point of the creation/deletion and can lead to instances being stuck forever in the BUILD state and other oddities. >From the error output, it looks a lot like a previously fixed issue for py-amqp where a connection closure causes channel issues [1]. Workaround ========== None found so far other than retrying the Heat stack creation/deletion. [0] from a nova-cloud-controller unit in nova-conductor.log: 2021-06-24 13:09:14.102 1252 ERROR oslo.messaging._drivers.impl_rabbit [req-5f22f6b9-7b96-4caa-80e2-780084603616 bd514b9cc0dd4395b8c4ea0aa251cf60 d7c94233a5144a4c96c1c78c06384616 - 7744427f95d7445bb90798103ac11564 7744427f95d7445bb90798103ac11564] Failed to publish message to topic 'nova': 'NoneType' object has no attribute '__getitem__' 2021-06-24 13:09:14.103 1252 ERROR oslo.messaging._drivers.impl_rabbit [req-5f22f6b9-7b96-4caa-80e2-780084603616 bd514b9cc0dd4395b8c4ea0aa251cf60 d7c94233a5144a4c96c1c78c06384616 - 7744427f95d7445bb90798103ac11564 7744427f95d7445bb90798103ac11564] Unable to connect to AMQP server on 172.29.101.62:5672 after None tries: 'NoneType' object has no attribute '__getitem__' 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server [req-5f22f6b9-7b96-4caa-80e2-780084603616 bd514b9cc0dd4395b8c4ea0aa251cf60 d7c94233a5144a4c96c1c78c06384616 - 7744427f95d7445bb90798103ac11564 7744427f95d7445bb90798103ac11564] Exception during message handling: MessageDeliveryFailure: Unable to connect to AMQP server on 172.29.101.62:5672 after None tries: 'NoneType' object has no attribute '__getitem__' 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 1302, in schedule_and_build_instances 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server limits=host.limits, host_list=host_list) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 1234, in build_and_run_instance 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server cctxt.cast(ctxt, 'build_and_run_instance', **kwargs) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 152, in cast 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server self.transport._send(self.target, msg_ctxt, msg, retry=self.retry) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, in _send 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server timeout=timeout, retry=retry) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server retry=retry) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 545, in _send 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server msg=msg, timeout=timeout, retry=retry) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1272, in topic_send 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server retry=retry) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1155, in _ensure_publishing 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server self.ensure(method, retry=retry, error_callback=_error_callback) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 825, in ensure 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server raise exceptions.MessageDeliveryFailure(msg) 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server MessageDeliveryFailure: Unable to connect to AMQP server on 172.29.101.62:5672 after None tries: 'NoneType' object has no attribute '__getitem__' 2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server [1] https://github.com/celery/py-amqp/pull/289 ** Affects: python-amqp (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1933539 Title: Intermittent messaging errors: 'NoneType' object has no attribute '__getitem__' To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-amqp/+bug/1933539/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
