Reviewed:  https://review.openstack.org/548572
Committed: 
https://git.openstack.org/cgit/openstack/nova/commit/?id=3a3b0f09db318faf1a1ea711a73bb365cab8b233
Submitter: Zuul
Branch:    master

commit 3a3b0f09db318faf1a1ea711a73bb365cab8b233
Author: Thomas Bechtold <tbecht...@suse.com>
Date:   Wed Feb 28 11:45:01 2018 +0100

    Allow 'network' in RequestContext service_catalog
    
    When booting instances, nova might create neutron resources. For that,
    the network service endpoint needs to be available. Otherwise we run
    into:
    
    EndpointNotFound: ['internal', 'public'] endpoint for network service \
    not found
    
    Change-Id: Iaed84826b76ab976ffdd1c93106b7bae700a64a9
    Closes-Bug: #1752289


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1752289

Title:
  ServiceCatalog does not contain "network" service

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) queens series:
  In Progress

Bug description:
  On SLE12SP3, openstack-nova 17.0.0.0~xrc2~dev160-1.1, I try to boot an
  cirros instance and get:

  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager 
[req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 
d9685d1130d74a73af6ee213c421d9de - default default] [instance: 
0ae671f6-5241-486f-9054-1100b124f704] Instance failed to spawn: 
EndpointNotFound: ['internal', 'public'] endpoint for network service not found
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704] Traceback (most recent call last):
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in 
_build_resources
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     yield resources
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2019, in 
_build_and_run_instance
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     block_device_info=block_device_info)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3011, in 
spawn
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     mdevs=mdevs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5256, in 
_get_guest_xml
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     network_info_str = str(network_info)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/model.py", line 568, in __str__
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     return self._sync_wrapper(fn, *args, 
**kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/model.py", line 551, in 
_sync_wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     self.wait()
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/model.py", line 583, in wait
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     self[:] = self._gt.wait()
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     return self._exit_event.wait()
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     current.throw(*self._exc)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     result = function(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/utils.py", line 906, in context_wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     return func(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1414, in 
_allocate_network_async
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     six.reraise(*exc_info)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1397, in 
_allocate_network_async
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     bind_host_id=bind_host_id)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 918, in 
allocate_for_instance
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     context, instance, neutron, 
requested_networks, ordered_networks)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 758, in 
_validate_requested_network_ids
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     auto_allocate=auto_allocate)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 383, in 
_get_available_networks
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     nets = 
neutron.list_networks(**search_opts).get('networks', [])
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in 
wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 809, in 
list_networks
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     **_params)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in 
wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in 
list
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     for r in self._pagination(collection, 
path, **params):
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in 
_pagination
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     res = self.get(path, params=params)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in 
wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in 
get
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     headers=headers, params=params)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in 
wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in 
retry_request
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     headers=headers, params=params)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in 
wrapper
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     ret = obj(*args, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 282, in 
do_request
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     headers=headers)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/neutronclient/client.py", line 342, in 
do_request
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     self._check_uri_length(url)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/neutronclient/client.py", line 335, in 
_check_uri_length
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     uri_len = len(self.endpoint_url) + 
len(url)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/neutronclient/client.py", line 349, in 
endpoint_url
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     return self.get_endpoint()
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 223, in 
get_endpoint
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     return self.session.get_endpoint(auth 
or self.auth, **kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 942, in 
get_endpoint
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     return auth.get_endpoint(self, 
**kwargs)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/nova/context.py", line 78, in get_endpoint
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     region_name=region_name)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", 
line 338, in url_for
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     endpoint_id=endpoint_id).url
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]   File 
"/usr/lib/python2.7/site-packages/keystoneauth1/access/service_catalog.py", 
line 399, in endpoint_data_for
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704]     raise exceptions.EndpointNotFound(msg)
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704] EndpointNotFound: ['internal', 'public'] 
endpoint for network service not found
  2018-02-28 10:24:07.553 10768 ERROR nova.compute.manager [instance: 
0ae671f6-5241-486f-9054-1100b124f704] 
  2018-02-28 10:24:07.555 10768 INFO nova.compute.manager 
[req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 
d9685d1130d74a73af6ee213c421d9de - default default] [instance: 
0ae671f6-5241-486f-9054-1100b124f704] Terminating instance
  2018-02-28 10:24:07.560 10768 INFO nova.virt.libvirt.driver 
[req-53fa6935-f60d-4e07-bc75-14b6a5336330 f80483de8573468b869e64262780a903 
d9685d1130d74a73af6ee213c421d9de - default default] [instance: 
0ae671f6-5241-486f-9054-1100b124f704] Instance destroyed successfully.

  
  /etc/nova/nova.conf.d/100-nova.conf contains:
  [keystone_authtoken]
  auth_type = password
  username = nova
  password = openstack
  user_domain_name = Default
  project_name = service
  project_domain_name = Default
  auth_url = http://192.168.122.80:5000/
  [neutron]
  auth_type = password
  username = neutron
  password = openstack
  user_domain_name = Default
  user_domain_id = default
  project_name = service
  project_domain_name = Default
  project_domain_id = default
  auth_url = http://192.168.122.80:5000/
  service_metadata_proxy = True
  metadata_proxy_shared_secret = a3f876fb3286e8bf304c01064e080fae

  
  Theses credentials are correct. I can list the full service catalog when 
using these creds.
  The problem seems, that the context which is used has only a subset of the 
complete service_catalog:

  > 
/usr/lib/python2.7/site-packages/nova/compute/manager.py(1440)_build_networks_for_instance()
  -> macs = self.driver.macs_for_instance(instance)
  (Pdb) pp context.to_dict()
  {'auth_token': 
u'gAAAAABaloMiR0qjo1BapDgt93AP5MLLx_VAXY-8GVUHR4iXZkwXmZ-qdnLcba2PKmQETAYqSWEi03ALlvgOCKB1CU6-XWsOXMuqFT9Urnm4D3Zd9Ss_J3QNRAC8furxu7UGtP5XbHymcE8xfjbPDfQNaHfJhTJo511tPd57TkIOcKWbff7JA_M',
   'domain': None,
   'global_request_id': None,
   'instance_lock_checked': False,
   'is_admin': True,
   'is_admin_project': True,
   'project': u'd9685d1130d74a73af6ee213c421d9de',
   'project_domain': u'default',
   'project_id': u'd9685d1130d74a73af6ee213c421d9de',
   'project_name': u'admin',
   'quota_class': None,
   'read_deleted': u'no',
   'read_only': False,
   'remote_address': u'192.168.122.80',
   'request_id': u'req-53fa6935-f60d-4e07-bc75-14b6a5336330',
   'resource_uuid': None,
   'roles': [u'admin'],
   'service_catalog': [{u'endpoints': [{u'adminURL': 
u'http://192.168.122.80:8780/',
                                        u'internalURL': 
u'http://192.168.122.80:8780/',
                                        u'publicURL': 
u'http://192.168.122.80:8780/',
                                        u'region': u'RegionOne'}],
                        u'name': u'placement',
                        u'type': u'placement'},
                       {u'endpoints': [{u'adminURL': 
u'http://192.168.122.80:9292',
                                        u'internalURL': 
u'http://192.168.122.80:9292',
                                        u'publicURL': 
u'http://192.168.122.80:9292',
                                        u'region': u'RegionOne'}],
                        u'name': u'glance',
                        u'type': u'image'},
                       {u'endpoints': [{u'adminURL': 
u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de',
                                        u'internalURL': 
u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de',
                                        u'publicURL': 
u'http://192.168.122.80:8776/v3/d9685d1130d74a73af6ee213c421d9de',
                                        u'region': u'RegionOne'}],
                        u'name': u'cinderv3',
                        u'type': u'volumev3'},
                       {u'endpoints': [{u'adminURL': 
u'http://192.168.122.80:9311',
                                        u'internalURL': 
u'http://192.168.122.80:9311',
                                        u'publicURL': 
u'http://192.168.122.80:9311',
                                        u'region': u'RegionOne'}],
                        u'name': u'barbican',
                        u'type': u'key-manager'}],
   'show_deleted': False,
   'tenant': u'd9685d1130d74a73af6ee213c421d9de',
   'timestamp': '2018-02-28T10:23:31.133924',
   'user': u'f80483de8573468b869e64262780a903',
   'user_domain': u'default',
   'user_id': u'f80483de8573468b869e64262780a903',
   'user_identity': u'f80483de8573468b869e64262780a903 
d9685d1130d74a73af6ee213c421d9de - default default',
   'user_name': u'admin'}

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1752289/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

Reply via email to