Public bug reported: Description:-
So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:- 2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last): 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token( 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION): 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs), 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times] 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack 2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500 Environment Details:- OS: Fedora 28 python: 3.6.7 eventlet: 0.24.1 The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case. Expected result =============== Nova api should response successfully. Actual result ============= Nova api returns Recursion Error. [1] https://review.openstack.org/#/c/620561/ [2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32 [3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1808951 Title: python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object Status in OpenStack Compute (nova): New Bug description: Description:- So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:- 2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last): 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token( 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION): 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs), 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times] 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack 2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500 Environment Details:- OS: Fedora 28 python: 3.6.7 eventlet: 0.24.1 The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case. Expected result =============== Nova api should response successfully. Actual result ============= Nova api returns Recursion Error. [1] https://review.openstack.org/#/c/620561/ [2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32 [3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1808951/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
