Reviewed: https://review.openstack.org/626952 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3c5e2b0e9fac985294a949852bb8c83d4ed77e04 Submitter: Zuul Branch: master
commit 3c5e2b0e9fac985294a949852bb8c83d4ed77e04 Author: Matthew Booth <[email protected]> Date: Wed Jan 30 15:10:25 2019 +0000 Eventlet monkey patching should be as early as possible We were seeing infinite recursion opening an ssl socket when running various combinations of python3, eventlet, and urllib3. It is not clear exactly what combination of versions are affected, but for background there is an example of this issue documented here: https://github.com/eventlet/eventlet/issues/371 The immediate cause in nova's case was that we were calling eventlet.monkey_patch() after importing urllib3. Specifically, change Ie7bf5d012e2ccbcd63c262ddaf739782afcdaf56 introduced the nova.utils.monkey_patch() method to make monkey patching common between WSGI and non-WSGI services. Unfortunately, before executing this method you must first import nova.utils, which imports a large number of modules itself. Anything imported (transitively) by nova.utils would therefore be imported before monkey patching, which included urllib3. This triggers the infinite recursion problem described above if you have an affected combination of library versions. While this specific issue may eventually be worked around or fixed in eventlet or urllib3, it remains true that eventlet best practises are to monkey patch as early as possible, which we were not doing. To avoid this and hopefully future similar issues, this change ensures that monkey patching happens as early as possible, and only a minimum number of modules are imported first. This change fixes monkey patching for both non-wsgi and wsgi callers: * Non-WSGI services (nova/cmd) This is fixed by using the new monkey_patch module, which has minimal dependencies. * WSGI services (nova/api/openstack) This is fixed both by using the new monkey_patch module, and by moving the patching point up one level so that it is done before importing anything in nova/api/openstack/__init__.py. This move causes issues for some external tools which load this path from nova and now monkey patch where they previously did not. However, it is unfortunately unavoidable to enable monkey patching for the wsgi entry point without major restructuring. This change includes a workaround for sphinx to avoid this issue. This change has been through several iterations. I started with what seemed like the simplest and most obvious change, and moved on as I discovered more interactions which broke. It is clear that eventlet monkey patching is extremely fragile, especially when done implicitly at module load time as we do. I would advocate a code restructure to improve this situation, but I think the time would be better spent removing the eventlet dependency entirely. Co-authored-by: Lee Yarwood <[email protected]> Closes-Bug: #1808975 Closes-Bug: #1808951 Change-Id: Id46e76666b553a10ec4654d4418a9884975b5b95 ** Changed in: nova Status: In Progress => Fix Released ** Bug watch added: github.com/eventlet/eventlet/issues #371 https://github.com/eventlet/eventlet/issues/371 -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1808951 Title: python3 + Fedora + SSL + wsgi nova deployment, nova api returns RecursionError: maximum recursion depth exceeded while calling a Python object Status in OpenStack Compute (nova): Fix Released Status in tripleo: Triaged Bug description: Description:- So while testing python3 with Fedora in [1], Found an issue while running nova-api behind wsgi. It fails with below Traceback:- 2018-12-18 07:41:55.364 26870 INFO nova.api.openstack.requestlog [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] 127.0.0.1 "GET /v2.1/servers/detail?all_tenants=True&deleted=True" status: 500 len: 0 microversion: - time: 0.007297 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] Caught error: maximum recursion depth exceeded while calling a Python object: RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack Traceback (most recent call last): 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/__init__.py", line 94, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return req.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 92, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._log_req(req, res, start) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.force_reraise() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack six.reraise(self.type_, self.value, self.tb) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack raise value 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/nova/api/openstack/requestlog.py", line 87, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack res = req.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return resp(environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return request.get_response(self.application) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1313, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack application, catch_exc_info=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/request.py", line 1277, in call_application 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack app_iter = application(self.environ, start_response) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.func(req, *args, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 332, in __call__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack response = self.process_request(req) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 622, in process_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = super(AuthProtocol, self).process_request(request) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 405, in process_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 437, in _do_fetch_token 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack data = self.fetch_token(token, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 761, in fetch_token 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_expired=allow_expired) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack auth_ref = self._request_strategy.verify_token( 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 170, in _request_strategy 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack strategy_class = self._get_strategy_class() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/_identity.py", line 192, in _get_strategy_class 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack if self._adapter.get_endpoint(version=klass.AUTH_VERSION): 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.session.get_endpoint(auth or self.auth, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return auth.get_endpoint(self, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack allow_version_hack=allow_version_hack, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack service_catalog = self.get_access(session).service_catalog 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self.auth_ref = self.get_auth_ref(session) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._plugin = self._do_create_plugin(session) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=False) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack disc = Discover(session, url, authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__ 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = session.get(url, headers=headers, authenticated=authenticated) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack return self.request(url, 'GET', **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = send(**kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.session.request(method, url, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack resp = self.send(prep, **send_kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack r = adapter.send(request, **kwargs) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack timeout=timeout 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack chunked=chunked) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack self._validate_conn(conn) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack conn.connect() 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack cert_reqs=resolve_cert_reqs(self.cert_reqs), 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack context.options |= options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack [Previous line repeated 280 more times] 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 07:41:55.364 26870 ERROR nova.api.openstack 2018-12-18 07:41:55.371 26870 INFO nova.api.openstack [req-e1af4808-ecd8-47c7-9568-a5dd9691c2c9 - - - - -] https://127.0.0.1:8774/v2.1/servers/detail?all_tenants=True&deleted=True returned with HTTP 500 Environment Details:- OS: Fedora 28 python: 3.6.7 eventlet: 0.24.1 The issue is around using of eventlet.monkey_patch even if running under wsgi[2]. Eventlet Best practices suggest to not run(skip) monkey_patch when running behind wsgi[3] but with nova it's not the case. Expected result =============== Nova api should response successfully. Actual result ============= Nova api returns Recursion Error. [1] https://review.openstack.org/#/c/620561/ [2] https://github.com/openstack/nova/blob/a674241d0f026b546b38ec77ab472e916805c679/nova/api/openstack/wsgi_app.py#L32 [3] https://review.openstack.org/#/c/154642/2/specs/eventlet-best-practices.rst@54 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1808951/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
