Public bug reported: Hello,
Recently a node on my Openstack Cloud is going down periodically without nothing relevant happening, i have not found an explanation for this yet. I have the following environment: 1.- Version 13.1.2 2.- KVM 3.- Ceph. 4.- Neutron with VXlan tunneling. After reviewing nova logs i have seen these Error lines repeteadly: 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions [req-a88c0032-190f-4d44-a570-78e23714e761 5a2f3552384d4d6a9b45ddda6dd7b1a9 c7d1479503644cd0b793f2c81af5ee60 - - -] Unexpected exception in API method 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/remote_consoles.py", line 56, in get_vnc_console 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions console_type) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 171, in wrapped 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions return func(self, context, target, *args, **kwargs) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 151, in wrapped 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2933, in get_vnc_console 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions instance=instance, console_type=console_type) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 574, in get_vnc_console 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions instance=instance, console_type=console_type) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions retry=self.retry) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 91, in _send 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 512, in send 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions retry=retry) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 501, in _send 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 379, in wait 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 277, in get 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID c670a34162914270ad9ca6b0bd40b2fb 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions 2017-05-05 00:33:51.119 10983 INFO nova.api.openstack.wsgi [req-a88c0032-190f-4d44-a570-78e23714e761 5a2f3552384d4d6a9b45ddda6dd7b1a9 c7d1479503644cd0b793f2c81af5ee60 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_messaging.exceptions.MessagingTimeout' Thank you in advance for your response. Best regards, ** Affects: nova Importance: Undecided Status: New ** Attachment added: "nova-api.log-20170505-1493969701" https://bugs.launchpad.net/bugs/1689278/+attachment/4873430/+files/nova-api.log-20170505-1493969701 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1689278 Title: Compute Node goes down periodically Status in OpenStack Compute (nova): New Bug description: Hello, Recently a node on my Openstack Cloud is going down periodically without nothing relevant happening, i have not found an explanation for this yet. I have the following environment: 1.- Version 13.1.2 2.- KVM 3.- Ceph. 4.- Neutron with VXlan tunneling. After reviewing nova logs i have seen these Error lines repeteadly: 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions [req-a88c0032-190f-4d44-a570-78e23714e761 5a2f3552384d4d6a9b45ddda6dd7b1a9 c7d1479503644cd0b793f2c81af5ee60 - - -] Unexpected exception in API method 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/remote_consoles.py", line 56, in get_vnc_console 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions console_type) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 171, in wrapped 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions return func(self, context, target, *args, **kwargs) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 151, in wrapped 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2933, in get_vnc_console 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions instance=instance, console_type=console_type) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 574, in get_vnc_console 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions instance=instance, console_type=console_type) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions retry=self.retry) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 91, in _send 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions timeout=timeout, retry=retry) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 512, in send 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions retry=retry) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 501, in _send 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions result = self._waiter.wait(msg_id, timeout) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 379, in wait 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions message = self.waiters.get(msg_id, timeout=timeout) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 277, in get 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions 'to message ID %s' % msg_id) 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions MessagingTimeout: Timed out waiting for a reply to message ID c670a34162914270ad9ca6b0bd40b2fb 2017-05-05 00:33:51.117 10983 ERROR nova.api.openstack.extensions 2017-05-05 00:33:51.119 10983 INFO nova.api.openstack.wsgi [req-a88c0032-190f-4d44-a570-78e23714e761 5a2f3552384d4d6a9b45ddda6dd7b1a9 c7d1479503644cd0b793f2c81af5ee60 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'oslo_messaging.exceptions.MessagingTimeout' Thank you in advance for your response. Best regards, To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1689278/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
