Public bug reported: Description =========== After upgrading from Antelope to Bobcat, I'm unable to manage instances.
Steps to reproduce ================== Following the upgrade guide https://docs.openstack.org/nova/2023.2/admin/upgrades.html * Pre-existing installation, Antelope 2023.1 * Upgraded the controller node to Bobcat 2023.2 * Ran 'nova-manage api_db sync' and 'nova-manage db sync' * Upgraded other services, 'nova-status upgrade check' reports everything as succesful. * 'nova service-list' does not report orphaned records * Ran 'nova-manage db online_data_migrations' Expected result =============== Upgrade successful, able to use the OpenStack instance Actual result ============= The last step does not complete succesfully. Even after running it multiple times, there still is a pending operation: # nova-manage db online_data_migrations Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. Running batches of 50 until complete 1 rows matched query populate_instance_compute_id, 0 migrated +-------------------------------------+--------------+-----------+ | Migration | Total Needed | Completed | +-------------------------------------+--------------+-----------+ | fill_virtual_interface_list | 0 | 0 | | migrate_empty_ratio | 0 | 0 | | migrate_quota_classes_to_api_db | 0 | 0 | | migrate_quota_limits_to_api_db | 0 | 0 | | migration_migrate_to_uuid | 0 | 0 | | populate_dev_uuids | 0 | 0 | | populate_instance_compute_id | 1 | 0 | | populate_missing_availability_zones | 0 | 0 | | populate_queued_for_delete | 0 | 0 | | populate_user_id | 0 | 0 | | populate_uuids | 0 | 0 | +-------------------------------------+--------------+-----------+ Listing instances on the dashboard now fails with the error: Error: Unable to retrieve instances. Details Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2fa94a10-3168-41a5-8b0f-6d7499465ff1) Environment =========== 1. Exact version of OpenStack you are running? Bobcat 2023.2 2. Which hypervisor did you use? Compute nodes are running qemu-kvm 8.0.0 and libvirt 9.5.0 2. Which storage type did you use? Cinder has two backends, NFS and dcache 3. Which networking type did you use? Neutron with OpenVSwitch Logs & Configs ============== /var/log/nova/nova-api.log Trying to list instances: 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] Unexpected exception in API method: neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request. Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0'] 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times] 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi response = self._view_builder.detail( 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 474, in detail 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi self._add_security_grps(request, list(servers_dict["servers"]), 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 684, in _add_security_grps 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi security_group_api.get_instances_security_groups_bindings( 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 541, in get_instances_security_groups_bindings 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ports = _get_ports_from_server_list(servers, neutron) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 490, in _get_ports_from_server_list 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ports.extend(neutron.list_ports(**search_opts).get('ports')) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 815, in list_ports 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return self.list('ports', self.ports_path, retrieve_all, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 372, in list 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi for r in self._pagination(collection, path, **params): 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 387, in _pagination 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi res = self.get(path, params=params) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 356, in get 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return self.retry_request("GET", action, body=body, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 333, in retry_request 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return self.do_request(method, action, body=body, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 297, in do_request 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi self._handle_fault_response(status_code, replybody, resp) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi exception_handler_v20(status_code, error_body) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi raise client_exc(message=error_message, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request. 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0'] 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi 2024-02-20 08:57:14.750 66460 INFO nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> ** Affects: nova Importance: Undecided Status: New ** Description changed: Description =========== After upgrading from Antelope to Bobcat, I'm unable to manage instances. Steps to reproduce ================== Following the upgrade guide https://docs.openstack.org/nova/2023.2/admin/upgrades.html * Pre-existing installation, Antelope 2023.1 * Upgraded the controller node to Bobcat 2023.2 * Ran 'nova-manage api_db sync' and 'nova-manage db sync' * Upgraded other services, 'nova-status upgrade check' reports everything as succesful. * 'nova service-list' does not report orphaned records * Ran 'nova-manage db online_data_migrations' - - Expected result =============== - After the execution of the steps above, what should have - happened if the issue wasn't present? + Upgrade successful, able to use the OpenStack instance Actual result ============= - The last step does not complete succesfully. Even after running + The last step does not complete succesfully. Even after running it multiple times, there still is a pending operation: # nova-manage db online_data_migrations Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. Running batches of 50 until complete 1 rows matched query populate_instance_compute_id, 0 migrated +-------------------------------------+--------------+-----------+ | Migration | Total Needed | Completed | +-------------------------------------+--------------+-----------+ | fill_virtual_interface_list | 0 | 0 | | migrate_empty_ratio | 0 | 0 | | migrate_quota_classes_to_api_db | 0 | 0 | | migrate_quota_limits_to_api_db | 0 | 0 | | migration_migrate_to_uuid | 0 | 0 | | populate_dev_uuids | 0 | 0 | | populate_instance_compute_id | 1 | 0 | | populate_missing_availability_zones | 0 | 0 | | populate_queued_for_delete | 0 | 0 | | populate_user_id | 0 | 0 | | populate_uuids | 0 | 0 | +-------------------------------------+--------------+-----------+ Listing instances on the dashboard now fails with the error: Error: Unable to retrieve instances. Details - Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ - and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2fa94a10-3168-41a5-8b0f-6d7499465ff1) + Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ + and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2fa94a10-3168-41a5-8b0f-6d7499465ff1) Environment =========== 1. Exact version of OpenStack you are running? Bobcat 2023.2 2. Which hypervisor did you use? Compute nodes are running qemu-kvm 8.0.0 and libvirt 9.5.0 2. Which storage type did you use? Cinder has two backends, NFS and dcache 3. Which networking type did you use? Neutron with OpenVSwitch Logs & Configs ============== /var/log/nova/nova-api.log Trying to list instances: 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] Unexpected exception in API method: neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request. Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0'] 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times] 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi response = self._view_builder.detail( 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 474, in detail 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi self._add_security_grps(request, list(servers_dict["servers"]), 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 684, in _add_security_grps 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi security_group_api.get_instances_security_groups_bindings( 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 541, in get_instances_security_groups_bindings 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ports = _get_ports_from_server_list(servers, neutron) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 490, in _get_ports_from_server_list 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ports.extend(neutron.list_ports(**search_opts).get('ports')) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 815, in list_ports 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return self.list('ports', self.ports_path, retrieve_all, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 372, in list 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi for r in self._pagination(collection, path, **params): 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 387, in _pagination 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi res = self.get(path, params=params) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 356, in get 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return self.retry_request("GET", action, body=body, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 333, in retry_request 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return self.do_request(method, action, body=body, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 297, in do_request 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi self._handle_fault_response(status_code, replybody, resp) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi exception_handler_v20(status_code, error_body) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi raise client_exc(message=error_message, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request. 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0'] - 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi + 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi 2024-02-20 08:57:14.750 66460 INFO nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2054409 Title: <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) after upgrade Status in OpenStack Compute (nova): New Bug description: Description =========== After upgrading from Antelope to Bobcat, I'm unable to manage instances. Steps to reproduce ================== Following the upgrade guide https://docs.openstack.org/nova/2023.2/admin/upgrades.html * Pre-existing installation, Antelope 2023.1 * Upgraded the controller node to Bobcat 2023.2 * Ran 'nova-manage api_db sync' and 'nova-manage db sync' * Upgraded other services, 'nova-status upgrade check' reports everything as succesful. * 'nova service-list' does not report orphaned records * Ran 'nova-manage db online_data_migrations' Expected result =============== Upgrade successful, able to use the OpenStack instance Actual result ============= The last step does not complete succesfully. Even after running it multiple times, there still is a pending operation: # nova-manage db online_data_migrations Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. Running batches of 50 until complete 1 rows matched query populate_instance_compute_id, 0 migrated +-------------------------------------+--------------+-----------+ | Migration | Total Needed | Completed | +-------------------------------------+--------------+-----------+ | fill_virtual_interface_list | 0 | 0 | | migrate_empty_ratio | 0 | 0 | | migrate_quota_classes_to_api_db | 0 | 0 | | migrate_quota_limits_to_api_db | 0 | 0 | | migration_migrate_to_uuid | 0 | 0 | | populate_dev_uuids | 0 | 0 | | populate_instance_compute_id | 1 | 0 | | populate_missing_availability_zones | 0 | 0 | | populate_queued_for_delete | 0 | 0 | | populate_user_id | 0 | 0 | | populate_uuids | 0 | 0 | +-------------------------------------+--------------+-----------+ Listing instances on the dashboard now fails with the error: Error: Unable to retrieve instances. Details Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> (HTTP 500) (Request-ID: req-2fa94a10-3168-41a5-8b0f-6d7499465ff1) Environment =========== 1. Exact version of OpenStack you are running? Bobcat 2023.2 2. Which hypervisor did you use? Compute nodes are running qemu-kvm 8.0.0 and libvirt 9.5.0 2. Which storage type did you use? Cinder has two backends, NFS and dcache 3. Which networking type did you use? Neutron with OpenVSwitch Logs & Configs ============== /var/log/nova/nova-api.log Trying to list instances: 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] Unexpected exception in API method: neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request. Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0'] 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Traceback (most recent call last): 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/wsgi.py", line 658, in wrapped 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return f(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/validation/__init__.py", line 192, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return func(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi [Previous line repeated 2 more times] 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 143, in detail 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi servers = self._get_servers(req, is_detail=True) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/servers.py", line 343, in _get_servers 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi response = self._view_builder.detail( 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 474, in detail 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi self._add_security_grps(request, list(servers_dict["servers"]), 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/api/openstack/compute/views/servers.py", line 684, in _add_security_grps 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi security_group_api.get_instances_security_groups_bindings( 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 541, in get_instances_security_groups_bindings 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ports = _get_ports_from_server_list(servers, neutron) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/security_group_api.py", line 490, in _get_ports_from_server_list 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ports.extend(neutron.list_ports(**search_opts).get('ports')) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 815, in list_ports 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return self.list('ports', self.ports_path, retrieve_all, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 372, in list 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi for r in self._pagination(collection, path, **params): 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 387, in _pagination 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi res = self.get(path, params=params) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 356, in get 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return self.retry_request("GET", action, body=body, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 333, in retry_request 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi return self.do_request(method, action, body=body, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 297, in do_request 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi self._handle_fault_response(status_code, replybody, resp) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/nova/network/neutron.py", line 196, in wrapper 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi exception_handler_v20(status_code, error_body) 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi File "/usr/lib/python3.9/site-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi raise client_exc(message=error_message, 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request. 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi Neutron server returns request_ids: ['req-8894501a-070a-49be-8e46-e9bac7f1afb0'] 2024-02-20 08:57:14.744 66460 ERROR nova.api.openstack.wsgi 2024-02-20 08:57:14.750 66460 INFO nova.api.openstack.wsgi [None req-2fa94a10-3168-41a5-8b0f-6d7499465ff1 458ee6e3adf142048041c8d24fabeb85 c824412ae5904653a037e893827aa693 - - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'neutronclient.common.exceptions.InternalServerError'> To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2054409/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
