Reviewed: https://review.opendev.org/c/openstack/neutron/+/914154 Committed: https://opendev.org/openstack/neutron/commit/6395b4fe8ed99855853587fa93cb59fd2691aed5 Submitter: "Zuul (22348)" Branch: master
commit 6395b4fe8ed99855853587fa93cb59fd2691aed5 Author: Anton Kurbatov <[email protected]> Date: Mon Mar 25 18:49:52 2024 +0000 Fixing the 500 HTTP code in the metadata service if Nova is down If the Nova metadata service is unavailable, the requests.request() function may raise a ConnectionError. This results in the upper code returning a 500 HTTP status code to the user along with a traceback. Let's handle this scenario and instead return a 503 HTTP status code (service unavailable). If the Nova service is down and is behind another proxy (such as Nginx), then instead of a ConnectionError, the request may result in receiving a 502 or 503 HTTP status code. Let's also consider this situation and add support for an additional 504 code. Closes-Bug: #2059032 Change-Id: I16be18c46a6796224b0793dc385b0ddec01739c4 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2059032 Title: Neutron metadata service returns http code 500 if nova metadata service is down Status in neutron: Fix Released Bug description: We discovered that if the nova metadata service is down, then the neutron metadata service starts printing stack traces with a 500 HTTP code to the user. Demo on a newly installed devstack $ systemctl stop [email protected] Then inside a VM: $ curl http://169.254.169.254/latest/meta-data/hostname <html> <head> <title>500 Internal Server Error</title> </head> <body> <h1>500 Internal Server Error</h1> An unknown error has occurred. Please try your request again.<br /><br /> </body> </html>$ Stack trace: ERROR neutron.agent.metadata.agent Traceback (most recent call last): ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 85, in __call__ ERROR neutron.agent.metadata.agent res = self._proxy_request(instance_id, tenant_id, req) ERROR neutron.agent.metadata.agent File "/opt/stack/neutron/neutron/agent/metadata/agent.py", line 249, in _proxy_request ERROR neutron.agent.metadata.agent resp = requests.request(method=req.method, url=url, ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request ERROR neutron.agent.metadata.agent return session.request(method=method, url=url, **kwargs) ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request ERROR neutron.agent.metadata.agent resp = self.send(prep, **send_kwargs) ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send ERROR neutron.agent.metadata.agent r = adapter.send(request, **kwargs) ERROR neutron.agent.metadata.agent File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send ERROR neutron.agent.metadata.agent raise ConnectionError(e, request=request) ERROR neutron.agent.metadata.agent requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.136.16.184', port=8775): Max retries exceeded with url: /latest/meta-data/hostname (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3ce93f38b0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED')) ERROR neutron.agent.metadata.agent INFO eventlet.wsgi.server [-] ::ffff:192.168.100.14,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500 len: 362 time: 0.1392403 Also, in our installation the nova service is behind nginx. And if we stop nova metadata service we also get 500 http code but with another traceback: 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent [-] Unexpected error.: Exception: Unexpected response code: 502 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Traceback (most recent call last): 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 93, in __call__ 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent res = self._proxy_request(instance_id, tenant_id, req) 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent File "/usr/lib/python3.6/site-packages/neutron/agent/metadata/agent.py", line 288, in _proxy_request 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent resp.status_code) 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent Exception: Unexpected response code: 502 2024-03-25 20:27:01.985 24 ERROR neutron.agent.metadata.agent 2024-03-25 20:27:01.988 24 INFO eventlet.wsgi.server [-] 10.197.115.207,<local> "GET /latest/meta-data/hostname HTTP/1.1" status: 500 len: 362 time: 0.1369441 It seems to me that it is also better to handle nginx-like gateway errors a bit more correctly. These 500 HTTP codes worries us because we are trying to create an alert system one of the criteria for which is 500 codes. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2059032/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
