This is likely related to https://github.com/openstack/nova/commit/6cde77ebbab85bc8ccd2ab7ad977b1d4af4a13fa if it's a regression in Queens.
** Tags added: queens-backport-potential ** Also affects: nova/queens Importance: Undecided Status: New ** Changed in: nova/queens Status: New => Confirmed ** Changed in: nova Importance: Undecided => High ** Changed in: nova/queens Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1752289 Title: ServiceCatalog does not contain "network" service Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) queens series: Confirmed Bug description: On SLE12SP3, openstack-nova 17.0.0.0~xrc2~dev160-1.1, I try to boot an cirros instance and get: 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default] [instance: 0ae671f6-5241-486f-9054-1100b124f704] Instance failed to spawn: EndpointNotFound: ['internal', 'public'] endpoint for network service not found 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] Traceback (most recent call last): 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in _build_resources 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] yield resources 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2019, in _build_and_run_instance 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] block_device_info=block_device_info) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3011, in spawn 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] mdevs=mdevs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5256, in _get_guest_xml 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] network_info_str = str(network_info) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 568, in __str__ 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return self._sync_wrapper(fn, *args, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 551, in _sync_wrapper 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] self.wait() 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 583, in wait 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] self[:] = self._gt.wait() 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return self._exit_event.wait() 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] current.throw(*self._exc) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] result = function(*args, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 906, in context_wrapper 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return func(*args, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1414, in _allocate_network_async 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] six.reraise(*exc_info) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1397, in _allocate_network_async 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] bind_host_id=bind_host_id) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 918, in allocate_for_instance 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] context, instance, neutron, requested_networks, ordered_networks) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 758, in _validate_requested_network_ids 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] auto_allocate=auto_allocate) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 383, in _get_available_networks 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] nets = neutron.list_networks(**search_opts).get('networks', []) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 809, in list_networks 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] **_params) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] for r in self._pagination(collection, path, **params): 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] res = self.get(path, params=params) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] headers=headers, params=params) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] headers=headers, params=params) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] ret = obj(*args, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 282, in do_request 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] headers=headers) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 342, in do_request 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] self._check_uri_length(url) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 335, in _check_uri_length 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] uri_len = len(self.endpoint_url) + len(url) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 349, in endpoint_url 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return self.get_endpoint() 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 223, in get_endpoint 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return self.session.get_endpoint(auth or self.auth, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 942, in get_endpoint 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] return auth.get_endpoint(self, **kwargs) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/nova/context.py", line 78, in get_endpoint 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] region_name=region_name) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 338, in url_for 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] endpoint_id=endpoint_id).url 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] File "/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", line 399, in endpoint_data_for 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] raise exceptions.EndpointNotFound(msg) 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] EndpointNotFound: ['internal', 'public'] endpoint for network service not found 2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 0ae671f6-5241-486f-9054-1100b124f704] 2018-02-28 10:24:07.555 10768 INFO nova.compute.manager [req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default] [instance: 0ae671f6-5241-486f-9054-1100b124f704] Terminating instance 2018-02-28 10:24:07.560 10768 INFO nova.virt.libvirt.driver [req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default] [instance: 0ae671f6-5241-486f-9054-1100b124f704] Instance destroyed successfully. /etc/nova/nova.conf.d/100-nova.conf contains: [keystone_authtoken] auth_type = password username = nova password = openstack user_domain_name = Default project_name = service project_domain_name = Default auth_url = http://192.168.122.80:5000/ [neutron] auth_type = password username = neutron password = openstack user_domain_name = Default user_domain_id = default project_name = service project_domain_name = Default project_domain_id = default auth_url = http://192.168.122.80:5000/ service_metadata_proxy = True metadata_proxy_shared_secret = a3f876fb3286e8bf304c01064e080fae Theses credentials are correct. I can list the full service catalog when using these creds. The problem seems, that the context which is used has only a subset of the complete service_catalog: > /usr/lib/python2.7/site-packages/nova/compute/manager.py(1440)_build_networks_for_instance() -> macs = self.driver.macs_for_instance(instance) (Pdb) pp context.to_dict() {'auth_token': u'gAAAAABaloMiR0qjo1BapDgt93AP5MLLx_VAXY-8GVUHR4iXZkwXmZ-qdnLcba2PKmQETAYqSWEi03ALlvgOCKB1CU6-XWsOXMuqFT9Urnm4D3Zd9Ss_J3QNRAC8furxu7UGtP5XbHymcE8xfjbPDfQNaHfJhTJo511tPd57TkIOcKWbff7JA_M', 'domain': None, 'global_request_id': None, 'instance_lock_checked': False, 'is_admin': True, 'is_admin_project': True, 'project': u'd9685d1130d74a73af6ee213c421d9de', 'project_domain': u'default', 'project_id': u'd9685d1130d74a73af6ee213c421d9de', 'project_name': u'admin', 'quota_class': None, 'read_deleted': u'no', 'read_only': False, 'remote_address': u'192.168.122.80', 'request_id': u'req-53fa6935-f60d-4e07-bc75-14b6a5336330', 'resource_uuid': None, 'roles': [u'admin'], 'service_catalog': [{u'endpoints': [{u'adminURL': u'http://192.168.122.80:8780/', u'internalURL': u'http://192.168.122.80:8780/', u'publicURL': u'http://192.168.122.80:8780/', u'region': u'RegionOne'}], u'name': u'placement', u'type': u'placement'}, {u'endpoints': [{u'adminURL': u'http://192.168.122.80:9292', u'internalURL': u'http://192.168.122.80:9292', u'publicURL': u'http://192.168.122.80:9292', u'region': u'RegionOne'}], u'name': u'glance', u'type': u'image'}, {u'endpoints': [{u'adminURL': u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de', u'internalURL': u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de', u'publicURL': u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de', u'region': u'RegionOne'}], u'name': u'cinderv3', u'type': u'volumev3'}, {u'endpoints': [{u'adminURL': u'http://192.168.122.80:9311', u'internalURL': u'http://192.168.122.80:9311', u'publicURL': u'http://192.168.122.80:9311', u'region': u'RegionOne'}], u'name': u'barbican', u'type': u'key-manager'}], 'show_deleted': False, 'tenant': u'd9685d1130d74a73af6ee213c421d9de', 'timestamp': '2018-02-28T10:23:31.133924', 'user': u'f80483de8573468b869e64262780a903', 'user_domain': u'default', 'user_id': u'f80483de8573468b869e64262780a903', 'user_identity': u'f80483de8573468b869e64262780a903 d9685d1130d74a73af6ee213c421d9de - default default', 'user_name': u'admin'} To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1752289/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp