Public bug reported: Description:- While Testing python3 Fedora deployment for nova in [1] got below Recursion Error in nova-compute:-
2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager [req-f908a9e0-e77a-4d35-9266-fc5e8d79dfde - - - - -] Error updating resources for node rdo-fedora-stable-rdo-cloud-0000358855.: RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager Traceback (most recent call last): 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7690, in _update_available_resource_for_node 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager rt.update_available_resource(context, nodename, startup=startup) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 738, in update_available_resource 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return f(*args, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 794, in _update_available_resource 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager instance_by_uuid) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1256, in _remove_deleted_instances_allocations 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager context, cn.uuid) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 2165, in get_allocations_for_resource_provider 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = self.get(url, global_request_id=context.global_id) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 297, in get 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self._client.get(url, microversion=version, headers=headers) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.session.request(url, method, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 684, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager auth_headers = self.get_auth_headers(auth) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return auth.get_headers(self, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager token = self.get_token(session) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.get_access(session).auth_token 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self.auth_ref = self.get_auth_ref(session) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self._plugin = self._do_create_plugin(session) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager authenticated=False) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager authenticated=authenticated) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1436, in get_discovery 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager disc = Discover(session, url, authenticated=authenticated) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 520, in __init__ 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager authenticated=authenticated) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = session.get(url, headers=headers, authenticated=authenticated) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = send(**kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = self.session.request(method, url, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = self.send(prep, **send_kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager r = adapter.send(request, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager timeout=timeout 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager chunked=chunked) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self._validate_conn(conn) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager conn.connect() 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager cert_reqs=resolve_cert_reqs(self.cert_reqs), 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager context.options |= options 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager [Previous line repeated 294 more times] 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager Don't know what's causing it but it somehow related to the eventlet monkey patching done in nova. I filed seperated bug(also filed https://bugs.launchpad.net/nova/+bug/1808951 for nova-api running under wsgi) for nova-compute as issue here seems different(as nova-compute is not running under wsgi). Logs from job: https://logs.rdoproject.org/61/620561/12/openstack-check-rdo/puppet-openstack-integration-5-scenario-py3-tempest-fedora-28/74a9000/logs/nova/nova-compute.txt.gz Expected Result:- nova compute should work correctly under python3 Actual Result:- nova-compute fails [1] https://review.openstack.org/#/c/620561 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1808975 Title: python3 + Fedora + SSL + nova compute RecursionError: maximum recursion depth exceeded while calling a Python object Status in OpenStack Compute (nova): New Bug description: Description:- While Testing python3 Fedora deployment for nova in [1] got below Recursion Error in nova-compute:- 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager [req-f908a9e0-e77a-4d35-9266-fc5e8d79dfde - - - - -] Error updating resources for node rdo-fedora-stable-rdo-cloud-0000358855.: RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager Traceback (most recent call last): 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 7690, in _update_available_resource_for_node 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager rt.update_available_resource(context, nodename, startup=startup) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 738, in update_available_resource 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self._update_available_resource(context, resources, startup=startup) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return f(*args, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 794, in _update_available_resource 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager instance_by_uuid) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/compute/resource_tracker.py", line 1256, in _remove_deleted_instances_allocations 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager context, cn.uuid) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 2165, in get_allocations_for_resource_provider 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = self.get(url, global_request_id=context.global_id) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 297, in get 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self._client.get(url, microversion=version, headers=headers) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.session.request(url, method, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 684, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager auth_headers = self.get_auth_headers(auth) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1071, in get_auth_headers 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return auth.get_headers(self, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager token = self.get_token(session) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.get_access(session).auth_token 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self.auth_ref = self.get_auth_ref(session) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self._plugin = self._do_create_plugin(session) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager authenticated=False) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager authenticated=authenticated) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1436, in get_discovery 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager disc = Discover(session, url, authenticated=authenticated) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 520, in __init__ 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager authenticated=authenticated) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = session.get(url, headers=headers, authenticated=authenticated) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 814, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = send(**kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 903, in _send_request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = self.session.request(method, url, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 524, in request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager resp = self.send(prep, **send_kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 637, in send 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager r = adapter.send(request, **kwargs) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager timeout=timeout 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager chunked=chunked) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager self._validate_conn(conn) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager conn.connect() 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 332, in connect 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager cert_reqs=resolve_cert_reqs(self.cert_reqs), 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 281, in create_urllib3_context 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager context.options |= options 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager File "/usr/lib64/python3.6/ssl.py", line 423, in options 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager super(SSLContext, SSLContext).options.__set__(self, value) 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager [Previous line repeated 294 more times] 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager RecursionError: maximum recursion depth exceeded while calling a Python object 2018-12-18 08:00:05.266 2428 ERROR nova.compute.manager Don't know what's causing it but it somehow related to the eventlet monkey patching done in nova. I filed seperated bug(also filed https://bugs.launchpad.net/nova/+bug/1808951 for nova-api running under wsgi) for nova-compute as issue here seems different(as nova-compute is not running under wsgi). Logs from job: https://logs.rdoproject.org/61/620561/12/openstack-check-rdo/puppet-openstack-integration-5-scenario-py3-tempest-fedora-28/74a9000/logs/nova/nova-compute.txt.gz Expected Result:- nova compute should work correctly under python3 Actual Result:- nova-compute fails [1] https://review.openstack.org/#/c/620561 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1808975/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
