Public bug reported: This is 3 Node Liberty setup (controller, Network and Compute) configured with neutron Network option 2 : Self servicing network and is with openvswitch. I found everything is working fine except while launching instance I'm getting error as follows , By tracing keystone.log and nova-api.log and based on time stamp I see Keystone fails the authentication followed by nova-api started throwing the exception errors.
Attached the config files and logs for you reference ... Request you to help me ASAP. Controller: 10.182.131.93 Network: 10.182.131.84 Compute: 10.182.131.87 Thank You ****************************************************** 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] Unexpected exception in API method 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions **create_kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auto_disk_config, reservation_id, max_count) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions pci_request_info, requested_networks) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions neutron = get_client(context, admin=True) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auth_token = _ADMIN_AUTH.get_token(_SESSION) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.get_access(session).auth_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions self.auth_ref = self.get_auth_ref(session) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self._plugin.get_auth_ref(session, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions authenticated=False, log=False, **rkwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.request(url, 'POST', **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions raise exceptions.from_response(resp, method, url) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d2ea540d-cdbc-44b9-bbed-ab2fc92ebbe4) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions 2016-02-23 18:46:50.048 17814 INFO nova.api.openstack.wsgi [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneclient.exceptions.Unauthorized'> ****************************************************** ** Affects: nova Importance: Undecided Status: New ** Patch added: "Config_and_logs.txt" https://bugs.launchpad.net/bugs/1549074/+attachment/4579555/+files/Config_and_logs.txt ** Description changed: This is 3 Node Liberty setup (controller, Network and Compute) configured with neutron Network option 2 : Self servicing network and is with openvswitch. I found everything is working fine except while launching instance I'm getting error as follows , By tracing keystone.log and nova-api.log and based on time stamp I see Keystone fails the authentication followed by nova-api started throwing the exception errors. Attached the config files and logs for you reference ... Request you to help me ASAP. + Controller: 10.182.131.93 + Network: 10.182.131.84 + Compute: 10.182.131.87 + Thank You - ****************************************************** 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] Unexpected exception in API method 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions **create_kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auto_disk_config, reservation_id, max_count) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions pci_request_info, requested_networks) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions neutron = get_client(context, admin=True) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auth_token = _ADMIN_AUTH.get_token(_SESSION) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.get_access(session).auth_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions self.auth_ref = self.get_auth_ref(session) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self._plugin.get_auth_ref(session, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions authenticated=False, log=False, **rkwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.request(url, 'POST', **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions raise exceptions.from_response(resp, method, url) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d2ea540d-cdbc-44b9-bbed-ab2fc92ebbe4) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions 2016-02-23 18:46:50.048 17814 INFO nova.api.openstack.wsgi [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneclient.exceptions.Unauthorized'> ****************************************************** -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1549074 Title: Liberty - Unexpected exception in API method Status in OpenStack Compute (nova): New Bug description: This is 3 Node Liberty setup (controller, Network and Compute) configured with neutron Network option 2 : Self servicing network and is with openvswitch. I found everything is working fine except while launching instance I'm getting error as follows , By tracing keystone.log and nova-api.log and based on time stamp I see Keystone fails the authentication followed by nova-api started throwing the exception errors. Attached the config files and logs for you reference ... Request you to help me ASAP. Controller: 10.182.131.93 Network: 10.182.131.84 Compute: 10.182.131.87 Thank You ****************************************************** 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] Unexpected exception in API method 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions **create_kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auto_disk_config, reservation_id, max_count) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions pci_request_info, requested_networks) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions neutron = get_client(context, admin=True) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions auth_token = _ADMIN_AUTH.get_token(_SESSION) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.get_access(session).auth_token 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions self.auth_ref = self.get_auth_ref(session) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self._plugin.get_auth_ref(session, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions authenticated=False, log=False, **rkwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return self.request(url, 'POST', **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions return func(*args, **kwargs) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions raise exceptions.from_response(resp, method, url) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d2ea540d-cdbc-44b9-bbed-ab2fc92ebbe4) 2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions 2016-02-23 18:46:50.048 17814 INFO nova.api.openstack.wsgi [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'keystoneclient.exceptions.Unauthorized'> ****************************************************** To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1549074/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
