Hello, thanks for the bug report! I think given how inconsistently reproducible the issue was, we can blame this on an environmental issue as opposed to a Nova bug. I'm not sure there's anything further to be done at this time.
** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1931707 Title: "NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception" error while instantiating instance in Train RDO Status in OpenStack Compute (nova): Invalid Bug description: Hi, Following ticket has been reported in https://bugzilla.redhat.com/show_bug.cgi?id=1964893 Based on their answer, verified the neutron command manually and it works fine. The issue is occurring randomly and failing the instance creation usecase. Recently had failed 2 cases, 1) it failed once out of 72 runs 2021-06-09 10:00:49.391 3194 INFO heat.engine.resource [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] CREATE: Net "vm_network_3" [5776a857-4316-47f9-b3fa-3a1fbff39132] Stack "vapp-Service-060821210256-1" [c7fa51bb-2012-4f5c-aa6f-1720dd47a5ed] 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Traceback (most recent call last): 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 920, in _action_recorder 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource yield 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 1033, in _do_action 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource yield self.action_handler_task(action, args=handler_args) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/scheduler.py", line 346, in wrapper 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource step = next(subtask) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 982, in action_handler_task 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource done = check(handler_data) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/net.py", line 208, in check_create_complete 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource attributes = self._show_resource() 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/neutron/neutron.py", line 139, in _show_resource 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource res_info = client_method(*args) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 822, in show_network 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource return self.get(self.network_path % (network), params=_params) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource headers=headers, params=params) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource headers=headers, params=params) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource self._handle_fault_response(status_code, replybody, resp) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource exception_handler_v20(status_code, error_body) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource request_ids=request_ids) 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}} 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31'] 2021-06-09 10:00:49.391 3194 ERROR heat.engine.resource 2021-06-09 10:00:49.429 3194 INFO heat.engine.stack [req-6d38438b-be89-4619-8517-9edc49bc7a40 - instTermProject-1 - default default] Stack CREATE FAILED (vapp-Service-060821210256-1): Resource CREATE failed: Unauthorized: resources.vm_network_3: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}} Neutron server returns request_ids: ['req-8eb8d67b-1b44-4caf-8c5a-37a442494b31'] 2) it failed once out of 72 runs 2021-06-10 10:01:27.888 13501 INFO nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Took 8.84 seconds to spawn the instance on the hypervisor. 2021-06-10 10:01:28.034 13501 ERROR nova.network.neutronv2.api [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf: Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}} 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Failed to deallocate networks: NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception. 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] Traceback (most recent call last): 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2746, in _cleanup_allocated_networks 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] self._deallocate_network(context, instance, requested_networks) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2002, in _deallocate_network 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] context, instance, requested_networks=requested_networks) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1570, in deallocate_for_instance 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] data = neutron.list_ports(**search_opts) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 795, in list_ports 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] **_params) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] for r in self._pagination(collection, path, **params): 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] res = self.get(path, params=params) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] headers=headers, params=params) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] headers=headers, params=params) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 121, in wrapper 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] ret = obj(*args, **kwargs) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] self._handle_fault_response(status_code, replybody, resp) 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 137, in wrapper 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] raise exception.NeutronAdminCredentialConfigurationInvalid() 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception. 2021-06-10 10:01:28.034 13501 ERROR nova.compute.manager [instance: f0dc327b-1416-400f-9846-86b9ef0dfc71] 2021-06-10 10:01:28.154 13501 INFO nova.scheduler.client.report [req-c531e9c8-49b7-4e16-a06c-813cf32067d6 289cd17405e846c390fbb62a92a8adb9 53e47db492134bf4b58f3d2aadafb639 - default default] Deleted allocation for instance f0dc327b-1416-400f-9846-86b9ef0dfc71 I did tried with neutron auth config from nova.conf manually and it works fine. No similar issue observed with Queens openstack version. Also many errors related to rabbitmq 2021-06-11 03:28:26.028 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-06-11 03:28:26.034 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-06-11 03:28:26.056 26179 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-06-11 03:28:26.057 26158 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer openstack-nova-novncproxy-20.6.0-1.el7.noarch openstack-nova-api-20.6.0-1.el7.noarch openstack-nova-common-20.6.0-1.el7.noarch puppet-nova-15.7.0-1.el7.noarch python2-nova-20.6.0-1.el7.noarch openstack-nova-scheduler-20.6.0-1.el7.noarch openstack-nova-conductor-20.6.0-1.el7.noarch python2-novaclient-15.1.1-1.el7.noarch openstack-neutron-common-15.3.2-1.el7.noarch puppet-neutron-15.5.0-1.el7.noarch python2-neutronclient-6.14.1-1.el7.noarch openstack-neutron-metering-agent-15.3.2-1.el7.noarch openstack-neutron-15.3.2-1.el7.noarch openstack-neutron-openvswitch-15.3.2-1.el7.noarch python2-neutron-lib-1.29.1-1.el7.noarch openstack-neutron-ml2-15.3.2-1.el7.noarch python2-neutron-15.3.2-1.el7.noarch openstack-keystone-16.0.1-1.el7.noarch python2-keystonemiddleware-7.0.1-2.el7.noarch puppet-keystone-15.4.0-1.el7.noarch python2-keystoneclient-3.21.0-2.el7.noarch python2-keystoneauth1-3.17.3-1.el7.noarch python2-keystone-16.0.1-1.el7.noarch Could you please check and let know if any config needs to be changed? Best Regards To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1931707/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp