Public bug reported: neutronclient in nova uses token_endpoint.Token as auth_plugin, which doesn't have functionality to re-authenticate in case of 401 error. If keystone uses memcached as backend and it was restarted, token that neutronclient use become invalid, and no re-authentication attempts were made. Trace in nova-api: 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1519, in _build_network_info_model 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] data = client.list_ports(**search_opts) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] ret = self.function(instance, args, *kwargs) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 534, in list_ports 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] **_params) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 307, in list 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] for r in self._pagination(collection, path, **params): 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 320, in _pagination 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] res = self.get(path, params=params) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in get 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] headers=headers, params=params) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270, in retry_request 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] headers=headers, params=params) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 211, in do_request 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] self._handle_fault_response(status_code, replybody) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] exception_handler_v20(status_code, des_error_body) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] message=message) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] NeutronClientException: Authentication required 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8]
** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1491905 Title: neutronclient in nova doesn't issue new token Status in OpenStack Compute (nova): New Bug description: neutronclient in nova uses token_endpoint.Token as auth_plugin, which doesn't have functionality to re-authenticate in case of 401 error. If keystone uses memcached as backend and it was restarted, token that neutronclient use become invalid, and no re-authentication attempts were made. Trace in nova-api: 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1519, in _build_network_info_model 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] data = client.list_ports(**search_opts) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] ret = self.function(instance, args, *kwargs) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 534, in list_ports 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] **_params) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 307, in list 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] for r in self._pagination(collection, path, **params): 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 320, in _pagination 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] res = self.get(path, params=params) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 293, in get 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] headers=headers, params=params) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270, in retry_request 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] headers=headers, params=params) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 211, in do_request 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] self._handle_fault_response(status_code, replybody) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] exception_handler_v20(status_code, des_error_body) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 83, in exception_handler_v20 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] message=message) 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] NeutronClientException: Authentication required 2015-08-30 19:02:24.292 8914 TRACE nova.network.base_api [instance: 94ccc989-4531-43a9-a1bd-d7cf848f04a8] To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1491905/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp