Public bug reported: I am using openstack ussuri with Centos 8, installed using RDO packstack. I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue.
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8868aed7-265d-4e8e-aa37-d427b1b561cb) : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 2020-08-17 22:53:46.574 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager raise value 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager 2020-08-17 22:54:46.284 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager raise value 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager root@openstack-Karty nova(keystone_admin)]# openstack server list Unknown Error (HTTP 500) [root@openstack-Karty nova(keystone_admin)]# openstack server list Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'TypeError'> (HTTP 500) (Request-ID: req-7d3a09b6-2e6a-43b5-8b4b-03c1ac3d37dd) [root@openstack-Karty nova(keystone_admin)]# tail -f -n 100 nova-compute.log 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self, "Failed to establish a new connection: %s" % e 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred: 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager timeout=timeout 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred: 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager resp = self.session.request(method, url, **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager resp = self.send(prep, **send_kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager r = adapter.send(request, **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise ConnectionError(e, request=request) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred: 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise value 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 486, in get_providers_in_tree 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager global_request_id=context.global_id) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 237, in get 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager global_request_id=global_request_id) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 98, in request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return self.session.request(url, method, **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager resp = send(**kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1043, in _send_request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1020, in _send_request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise exceptions.ConnectFailure(msg) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.24.0.117:8778/placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:59:56.498 63567 WARNING nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Unknown base file: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943 2020-08-17 22:59:56.498 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Removable base files: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943 2020-08-17 22:59:56.499 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Base or swap file too young to remove: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943 2020-08-17 23:00:05.503 63567 INFO nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Running instance usage audit for host openstack-Karty from 2020-08-18 05:00:00 to 2020-08-18 06:00:00. 0 instances. 2020-08-17 23:02:10.221 63567 INFO nova.compute.claims [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Claim successful on node openstack-Karty 2020-08-17 23:02:10.621 63567 INFO nova.virt.libvirt.driver [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Creating image 2020-08-17 23:02:14.710 63567 INFO os_vif [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:15:cb:de,bridge_name='br-int',has_traffic_filtering=True,id=bb31ebbf-b1f4-48bd-9412-5b1a8e60fecd,network=Network(ca332b6f-a3c3-4c96-bc27-5a2243929fb9),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapbb31ebbf-b1') 2020-08-17 23:02:15.093 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Started (Lifecycle Event) 2020-08-17 23:02:15.131 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Paused (Lifecycle Event) 2020-08-17 23:02:15.199 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] During sync_power_state the instance has a pending task (spawning). Skip. 2020-08-17 23:02:51.316 63567 ERROR nova.compute.resource_tracker [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 2020-08-17 23:03:53.500 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager raise value 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager Gateway Timeout The gateway did not receive a timely response from the upstream server or application. I need to keep restarting service httpd restart inorder to overcome this issue. Please let us know a permanent solution. Thanks. ** Affects: nova Importance: Undecided Status: New ** Summary changed: - Getting HTTP 500 frequently in Openstack Ussuri + Getting HTTP 500 exception frequently in Openstack Ussuri ** Description changed: - I am using openstack ussuri with Centos 8, installed using RDO - packstack. + I am using openstack ussuri with Centos 8, installed using RDO packstack. + I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue. Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8868aed7-265d-4e8e-aa37-d427b1b561cb) - : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 2020-08-17 22:53:46.574 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> - <p>Please contact the server administrator at - [no address given] to inform them of the time this error occurred, - and the actions you performed just before this error.</p> + <p>Please contact the server administrator at + [no address given] to inform them of the time this error occurred, + and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager raise value 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 - 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager + 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager 2020-08-17 22:54:46.284 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> - <p>Please contact the server administrator at - [no address given] to inform them of the time this error occurred, - and the actions you performed just before this error.</p> + <p>Please contact the server administrator at + [no address given] to inform them of the time this error occurred, + and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager raise value 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 - 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager + 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Gateway Timeout The gateway did not receive a timely response from the upstream server or application. - - I need to keep restarting service httpd restart inorder to overcome this issue. Please let us know a permanent solution. Thanks. + I need to keep restarting service httpd restart inorder to overcome this + issue. Please let us know a permanent solution. Thanks. ** Description changed: I am using openstack ussuri with Centos 8, installed using RDO packstack. - I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue. + I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue. Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8868aed7-265d-4e8e-aa37-d427b1b561cb) : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 2020-08-17 22:53:46.574 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager raise value 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager 2020-08-17 22:54:46.284 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager raise value 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager + + root@openstack-Karty nova(keystone_admin)]# openstack server list + Unknown Error (HTTP 500) + [root@openstack-Karty nova(keystone_admin)]# openstack server list + Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. + <class 'TypeError'> (HTTP 500) (Request-ID: req-7d3a09b6-2e6a-43b5-8b4b-03c1ac3d37dd) + [root@openstack-Karty nova(keystone_admin)]# tail -f -n 100 nova-compute.log + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self, "Failed to establish a new connection: %s" % e + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred: + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last): + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager timeout=timeout + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise MaxRetryError(_pool, url, error or ResponseError(cause)) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred: + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last): + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager resp = self.session.request(method, url, **kwargs) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager resp = self.send(prep, **send_kwargs) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager r = adapter.send(request, **kwargs) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise ConnectionError(e, request=request) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred: + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last): + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager startup=startup) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return f(*args, **kwargs) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise value + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 486, in get_providers_in_tree + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager global_request_id=context.global_id) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 237, in get + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager global_request_id=global_request_id) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 98, in request + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager **kwargs) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return self.session.request(url, method, **kwargs) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager resp = send(**kwargs) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1043, in _send_request + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager **kwargs) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1020, in _send_request + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise exceptions.ConnectFailure(msg) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.24.0.117:8778/placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) + 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager + 2020-08-17 22:59:56.498 63567 WARNING nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Unknown base file: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943 + 2020-08-17 22:59:56.498 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Removable base files: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943 + 2020-08-17 22:59:56.499 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Base or swap file too young to remove: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943 + 2020-08-17 23:00:05.503 63567 INFO nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Running instance usage audit for host openstack-Karty from 2020-08-18 05:00:00 to 2020-08-18 06:00:00. 0 instances. + 2020-08-17 23:02:10.221 63567 INFO nova.compute.claims [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Claim successful on node openstack-Karty + 2020-08-17 23:02:10.621 63567 INFO nova.virt.libvirt.driver [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Creating image + 2020-08-17 23:02:14.710 63567 INFO os_vif [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:15:cb:de,bridge_name='br-int',has_traffic_filtering=True,id=bb31ebbf-b1f4-48bd-9412-5b1a8e60fecd,network=Network(ca332b6f-a3c3-4c96-bc27-5a2243929fb9),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapbb31ebbf-b1') + 2020-08-17 23:02:15.093 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Started (Lifecycle Event) + 2020-08-17 23:02:15.131 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Paused (Lifecycle Event) + 2020-08-17 23:02:15.199 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] During sync_power_state the instance has a pending task (spawning). Skip. + 2020-08-17 23:02:51.316 63567 ERROR nova.compute.resource_tracker [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> + <html><head> + <title>500 Internal Server Error</title> + </head><body> + <h1>Internal Server Error</h1> + <p>The server encountered an internal error or + misconfiguration and was unable to complete + your request.</p> + <p>Please contact the server administrator at + [no address given] to inform them of the time this error occurred, + and the actions you performed just before this error.</p> + <p>More information about this error may be available + in the server error log.</p> + </body></html> + : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> + 2020-08-17 23:03:53.500 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> + <html><head> + <title>500 Internal Server Error</title> + </head><body> + <h1>Internal Server Error</h1> + <p>The server encountered an internal error or + misconfiguration and was unable to complete + your request.</p> + <p>Please contact the server administrator at + [no address given] to inform them of the time this error occurred, + and the actions you performed just before this error.</p> + <p>More information about this error may be available + in the server error log.</p> + </body></html> + . + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager Traceback (most recent call last): + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager startup=startup) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager return f(*args, **kwargs) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager raise value + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 + 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager + + Gateway Timeout The gateway did not receive a timely response from the upstream server or application. I need to keep restarting service httpd restart inorder to overcome this issue. Please let us know a permanent solution. Thanks. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1891973 Title: Getting HTTP 500 exception frequently in Openstack Ussuri Status in OpenStack Compute (nova): New Bug description: I am using openstack ussuri with Centos 8, installed using RDO packstack. I frequently getting http 500 EXCEPTION ERROR when ever am doing some operation, i need to restart httpd service inorder to overcome this issue. Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8868aed7-265d-4e8e-aa37-d427b1b561cb) : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 2020-08-17 22:53:46.574 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager raise value 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:53:46.575 63567 ERROR nova.compute.manager 2020-08-17 22:54:46.284 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager raise value 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 22:54:46.285 63567 ERROR nova.compute.manager root@openstack-Karty nova(keystone_admin)]# openstack server list Unknown Error (HTTP 500) [root@openstack-Karty nova(keystone_admin)]# openstack server list Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'TypeError'> (HTTP 500) (Request-ID: req-7d3a09b6-2e6a-43b5-8b4b-03c1ac3d37dd) [root@openstack-Karty nova(keystone_admin)]# tail -f -n 100 nova-compute.log 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self, "Failed to establish a new connection: %s" % e 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred: 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager timeout=timeout 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred: 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager resp = self.session.request(method, url, **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager resp = self.send(prep, **send_kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager r = adapter.send(request, **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise ConnectionError(e, request=request) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager During handling of the above exception, another exception occurred: 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise value 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 486, in get_providers_in_tree 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager global_request_id=context.global_id) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 237, in get 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager global_request_id=global_request_id) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/openstack/proxy.py", line 98, in request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager return self.session.request(url, method, **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager resp = send(**kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1043, in _send_request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager **kwargs) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1020, in _send_request 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager raise exceptions.ConnectFailure(msg) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.24.0.117:8778/placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9: HTTPConnectionPool(host='10.24.0.117', port=8778): Max retries exceeded with url: /placement/resource_providers?in_tree=3a80ea71-91a6-4da7-a05b-682b501fe6c9 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff128340518>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2020-08-17 22:56:47.807 63567 ERROR nova.compute.manager 2020-08-17 22:59:56.498 63567 WARNING nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Unknown base file: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943 2020-08-17 22:59:56.498 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Removable base files: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943 2020-08-17 22:59:56.499 63567 INFO nova.virt.libvirt.imagecache [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Base or swap file too young to remove: /var/lib/nova/instances/_base/5a8a3c88277cdf03ea71a4448af35b8e684f7943 2020-08-17 23:00:05.503 63567 INFO nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Running instance usage audit for host openstack-Karty from 2020-08-18 05:00:00 to 2020-08-18 06:00:00. 0 instances. 2020-08-17 23:02:10.221 63567 INFO nova.compute.claims [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Claim successful on node openstack-Karty 2020-08-17 23:02:10.621 63567 INFO nova.virt.libvirt.driver [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] Creating image 2020-08-17 23:02:14.710 63567 INFO os_vif [req-1de4c903-89cf-4494-99d5-3e6237cd67bf 108551f604a24c23989a699b058e584b 9d3e3eadbc9c4d5fa6a3b27bfee40734 - default default] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:15:cb:de,bridge_name='br-int',has_traffic_filtering=True,id=bb31ebbf-b1f4-48bd-9412-5b1a8e60fecd,network=Network(ca332b6f-a3c3-4c96-bc27-5a2243929fb9),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tapbb31ebbf-b1') 2020-08-17 23:02:15.093 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Started (Lifecycle Event) 2020-08-17 23:02:15.131 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] VM Paused (Lifecycle Event) 2020-08-17 23:02:15.199 63567 INFO nova.compute.manager [req-b9522c87-a4bb-42ab-a9a9-a60d987e60a8 - - - - -] [instance: 1e296073-65af-4c9c-8574-8c150f539cec] During sync_power_state the instance has a pending task (spawning). Skip. 2020-08-17 23:02:51.316 63567 ERROR nova.compute.resource_tracker [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Skipping removal of allocations for deleted instances: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> : nova.exception.ResourceProviderAllocationRetrievalFailed: Failed to retrieve allocations for resource provider 3a80ea71-91a6-4da7-a05b-682b501fe6c9: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> 2020-08-17 23:03:53.500 63567 ERROR nova.scheduler.client.report [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] [None] Failed to retrieve resource provider tree from placement API for UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9. Got 500: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> . 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager [req-fb01576e-e540-4930-a4f0-5109a0df28c4 - - - - -] Error updating resources for node openstack-Karty.: nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager Traceback (most recent call last): 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 9685, in _update_available_resource_for_node 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager startup=startup) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 842, in update_available_resource 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 359, in inner 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager return f(*args, **kwargs) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 927, in _update_available_resource 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager self._update(context, cn, startup=startup) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1176, in _update 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager self._update_to_placement(context, compute_node, startup) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager return Retrying(*dargs, **dkw).call(f, *args, **kw) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager return attempt.get(self._wrap_exception) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager six.reraise(self.value[0], self.value[1], self.value[2]) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager raise value 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager attempt = Attempt(fn(*args, **kwargs), attempt_number, False) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1110, in _update_to_placement 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager context, compute_node.uuid, name=compute_node.hypervisor_hostname) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 857, in get_provider_tree_and_ensure_root 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager parent_provider_uuid=parent_provider_uuid) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 640, in _ensure_resource_provider 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager rps_to_refresh = self.get_providers_in_tree(context, uuid) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 503, in get_providers_in_tree 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager raise exception.ResourceProviderRetrievalFailed(uuid=uuid) 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager nova.exception.ResourceProviderRetrievalFailed: Failed to get resource provider with UUID 3a80ea71-91a6-4da7-a05b-682b501fe6c9 2020-08-17 23:03:53.501 63567 ERROR nova.compute.manager Gateway Timeout The gateway did not receive a timely response from the upstream server or application. I need to keep restarting service httpd restart inorder to overcome this issue. Please let us know a permanent solution. Thanks. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1891973/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
