Reviewed: https://review.openstack.org/422780 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4dbc0ddc740881a409085feeca2a1771a9ea95f3 Submitter: Jenkins Branch: master
commit 4dbc0ddc740881a409085feeca2a1771a9ea95f3 Author: Roman Dobosz <[email protected]> Date: Thu Jan 19 18:26:45 2017 +0100 Prevent compute crash on discovery failure Keystoneauth1 may crash during discovery of URL for placement API. This is often happen on situation, when all services are started at once, compute will first be up while keystone still not. This patch fixes this by catching DiscoveyFailure as additional expected failures. Closes-Bug: #1656075 Change-Id: I48aa6802286b408260b9d10f9d13860b44bd7d34 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1656075 Title: DiscoveryFailure when trying to get resource providers from the scheduler report client Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) newton series: New Bug description: I noticed this in a TripleO job: http://logs.openstack.org/04/419604/1/check/gate-puppet-openstack- integration-4-scenario004-tempest-centos-7/5d95a8c/logs/nova/nova- compute.txt.gz#_2017-01-12_18_53_43_459 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager [req-59098025-7c99-41b2-aaa9-0e5714770b3a - - - - -] Error updating resources for node centos-7-osic-cloud1-s3500-6631948. 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager Traceback (most recent call last): 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6537, in update_available_resource_for_node 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager rt.update_available_resource(context, nodename) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 540, in update_available_resource 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self._update_available_resource(context, resources) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return f(*args, **kwargs) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 564, in _update_available_resource 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self._init_compute_node(context, resources) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 451, in _init_compute_node 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self.scheduler_client.update_resource_stats(self.compute_node) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 60, in update_resource_stats 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self.reportclient.update_resource_stats(compute_node) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return getattr(self.instance, __name)(*args, **kwargs) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 476, in update_resource_stats 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager compute_node.hypervisor_hostname) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 296, in _ensure_resource_provider 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager rp = self._get_resource_provider(uuid) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 53, in wrapper 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return f(self, *a, **k) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 209, in _get_resource_provider 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager resp = self.get("/resource_providers/%s" % uuid) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/report.py", line 174, in get 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager endpoint_filter=self.ks_filter, raise_exc=False) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 710, in get 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return self.request(url, 'GET', **kwargs) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return wrapped(*args, **kwargs) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 467, in request 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager auth_headers = self.get_auth_headers(auth) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 770, in get_auth_headers 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return auth.get_headers(self, **kwargs) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/plugin.py", line 90, in get_headers 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager token = self.get_token(session) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 90, in get_token 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager return self.get_access(session).auth_token 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 136, in get_access 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self.auth_ref = self.get_auth_ref(session) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 196, in get_auth_ref 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager self._plugin = self._do_create_plugin(session) 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 191, in _do_create_plugin 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager raise exceptions.DiscoveryFailure('Could not determine a suitable URL ' 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager DiscoveryFailure: Could not determine a suitable URL for the plugin 2017-01-12 18:53:43.459 15495 ERROR nova.compute.manager Looks like another type of keystoneauth1 exception that we're not handling in the safe_connect() decorator. At some point we'll probably just want to handle all KSA failures globally... To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1656075/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
