It could somehow be related to this change in neutronclient:
https://review.openstack.org/#/q/b8a05333ddc4e248e18080750e3cfa9cbedbca53
** Also affects: python-neutronclient
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1656056
Title:
NeutronAdminCredentialConfigurationInvalid all over n-cpu logs in
successful gate runs during network teardown
Status in OpenStack Compute (nova):
Confirmed
Status in python-neutronclient:
New
Bug description:
I've been seeing this all over the n-cpu logs in gate runs on master
even in successful runs:
http://logs.openstack.org/08/418108/4/gate/gate-tempest-dsvm-neutron-
src-neutron-lib-ubuntu-
xenial/dafcfca/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-01-12_17_05_25_061
2017-01-12 17:05:25.061 1236 ERROR nova.network.neutronv2.api [req-
dfe387da-8f68-40f7-a781-9fec1f7408b1 tempest-ImagesTestJSON-2082975254
tempest-ImagesTestJSON-2082975254] Neutron client was not able to
generate a valid admin token, please verify Neutron admin credential
located in nova.conf
-----
http://logs.openstack.org/08/418108/4/gate/gate-tempest-dsvm-neutron-
src-neutron-lib-ubuntu-
xenial/dafcfca/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-01-12_17_05_25_623
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager
[req-dfe387da-8f68-40f7-a781-9fec1f7408b1 tempest-ImagesTestJSON-2082975254
tempest-ImagesTestJSON-2082975254] [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] Setting instance vm_state to ERROR
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] Traceback (most recent call last):
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2425, in
do_terminate_instance
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] self._delete_instance(context,
instance, bdms, quotas)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/hooks.py", line 154, in inner
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] rv = f(*args, **kwargs)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2388, in _delete_instance
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] quotas.rollback()
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] self.force_reraise()
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] six.reraise(self.type_, self.value,
self.tb)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2352, in _delete_instance
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] self._shutdown_instance(context,
instance, bdms)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2257, in _shutdown_instance
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] self._try_deallocate_network(context,
instance, requested_networks)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2179, in
_try_deallocate_network
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7]
self._set_instance_obj_error_state(context, instance)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
__exit__
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] self.force_reraise()
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
force_reraise
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] six.reraise(self.type_, self.value,
self.tb)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/compute/manager.py", line 2173, in
_try_deallocate_network
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] self._deallocate_network(context,
instance, requested_networks)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/compute/manager.py", line 1667, in _deallocate_network
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] context, instance,
requested_networks=requested_networks)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/network/neutronv2/api.py", line 1156, in
deallocate_for_instance
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] data =
neutron.list_ports(**search_opts)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] ret = obj(*args, **kwargs)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
742, in list_ports
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] **_params)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] ret = obj(*args, **kwargs)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
375, in list
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] for r in self._pagination(collection,
path, **params):
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
390, in _pagination
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] res = self.get(path, params=params)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] ret = obj(*args, **kwargs)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
360, in get
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] headers=headers, params=params)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] ret = obj(*args, **kwargs)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
337, in retry_request
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] headers=headers, params=params)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/network/neutronv2/api.py", line 96, in wrapper
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] ret = obj(*args, **kwargs)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line
300, in do_request
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7]
self._handle_fault_response(status_code, replybody, resp)
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] File
"/opt/stack/new/nova/nova/network/neutronv2/api.py", line 112, in wrapper
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7] raise
exception.NeutronAdminCredentialConfigurationInvalid()
2017-01-12 17:05:25.157 1236 ERROR nova.compute.manager [instance:
4c7f8d12-2142-457c-b725-c95cf931cfa7]
NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing
an unauthorized exception.
This is using an admin client connection to neutron so we should be
regenerating a token for the admin so it's unclear why this is failing
now.
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22NeutronAdminCredentialConfigurationInvalid%3A%20Networking%20client%20is%20experiencing%20an%20unauthorized%20exception.%5C%22%20AND%20message%3A%5C%22deallocate_for_instance%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22%20AND%20voting%3A1&from=7d
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1656056/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
