Public bug reported:
Intermittently seeing instance with SR-IOV ports attached fail live migration
with KeyError pci_slot
After running the live migration command:
openstack server migrate --live-migration --os-compute-api 2.50 --host
$TARGET_HOST $SERVER
The server went into error state with the below error, and SR-IOV ports
were no longer attached.
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server Traceback (most recent
call last):
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py",
line 309, in dispatch
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py",
line 229, in _do_dispatch
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server result = func(ctxt,
**new_args)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py",
line 65, in wrapped
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py",
line 63, in wrapped
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/utils.py",
line 1453, in decorated_function
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return function(self,
context, *args, **kwargs)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py",
line 213, in decorated_function
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py",
line 203, in decorated_function
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return function(self,
context, *args, **kwargs)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py",
line 9524, in post_live_migration_at_destination
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py",
line 9520, in post_live_migration_at_destination
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
self.driver.post_live_migration_at_destination(
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 11606, in post_live_migration_at_destination
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
self._reattach_instance_vifs(context, instance, network_info)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 11158, in _reattach_instance_vifs
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
self.attach_interface(context, instance,
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 2976, in attach_interface
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server cfg =
self.vif_driver.get_config(instance, vif, image_meta,
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/vif.py",
line 597, in get_config
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return
self.get_config_hw_veb(*args)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/vif.py",
line 364, in get_config_hw_veb
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server conf, net_type,
profile['pci_slot'],
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server KeyError: 'pci_slot'
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
We are running 2024.1 deployed via OpenStack-Ansible
On KVM with linuxbridge.
/openstack/venvs/nova-29.0.1/bin/pip show nova
Name: nova
Version: 29.1.0
Summary: Cloud computing fabric controller
Home-page: https://docs.openstack.org/nova/latest/
Author: OpenStack
Author-email: [email protected]
License: UNKNOWN
Location: /openstack/venvs/nova-29.0.1/lib/python3.10/site-packages
Requires: alembic, castellan, cryptography, cursive, decorator, eventlet,
futurist, greenlet, iso8601, Jinja2, jsonschema, keystoneauth1,
keystonemiddleware, lxml, microversion-parse, netaddr, netifaces, openstacksdk,
os-brick, os-resource-classes, os-service-types, os-traits, os-vif, oslo.cache,
oslo.concurrency, oslo.config, oslo.context, oslo.db, oslo.i18n, oslo.limit,
oslo.log, oslo.messaging, oslo.middleware, oslo.policy, oslo.privsep,
oslo.reports, oslo.rootwrap, oslo.serialization, oslo.service,
oslo.upgradecheck, oslo.utils, oslo.versionedobjects, paramiko, Paste,
PasteDeploy, pbr, PrettyTable, psutil, python-cinderclient, python-dateutil,
python-glanceclient, python-neutronclient, PyYAML, requests, retrying, rfc3986,
Routes, SQLAlchemy, stevedore, tooz, WebOb, websockify
Required-by:
/openstack/venvs/neutron-29.0.1/bin/pip show neutron
Name: neutron
Version: 24.0.1.dev22
Summary: OpenStack Networking
Home-page: https://docs.openstack.org/neutron/latest/
Author: OpenStack
Author-email: [email protected]
License:
Location: /openstack/venvs/neutron-29.0.1/lib/python3.10/site-packages
Requires: alembic, debtcollector, decorator, eventlet, futurist, httplib2,
Jinja2, keystoneauth1, keystonemiddleware, netaddr, netifaces, neutron-lib,
openstacksdk, os-ken, os-resource-classes, os-vif, oslo.cache,
oslo.concurrency, oslo.config, oslo.context, oslo.db, oslo.i18n, oslo.log,
oslo.messaging, oslo.middleware, oslo.policy, oslo.privsep, oslo.reports,
oslo.rootwrap, oslo.serialization, oslo.service, oslo.upgradecheck, oslo.utils,
oslo.versionedobjects, osprofiler, ovs, ovsdbapp, Paste, PasteDeploy, pbr,
pecan, psutil, pyOpenSSL, pyroute2, python-designateclient,
python-neutronclient, python-novaclient, requests, Routes, SQLAlchemy,
stevedore, tenacity, tooz, WebOb
Required-by:
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2082541
Title:
Live migrating instance SR-IOV ports fails KeyError pci_slot
Status in OpenStack Compute (nova):
New
Bug description:
Intermittently seeing instance with SR-IOV ports attached fail live migration
with KeyError pci_slot
After running the live migration command:
openstack server migrate --live-migration --os-compute-api 2.50 --host
$TARGET_HOST $SERVER
The server went into error state with the below error, and SR-IOV
ports were no longer attached.
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server Traceback (most recent
call last):
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/server.py",
line 165, in _process_incoming
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py",
line 309, in dispatch
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return
self._do_dispatch(endpoint, method, ctxt, args)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_messaging/rpc/dispatcher.py",
line 229, in _do_dispatch
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server result = func(ctxt,
**new_args)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py",
line 65, in wrapped
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/exception_wrapper.py",
line 63, in wrapped
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return f(self,
context, *args, **kw)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/utils.py",
line 1453, in decorated_function
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return function(self,
context, *args, **kwargs)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py",
line 213, in decorated_function
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py",
line 203, in decorated_function
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return function(self,
context, *args, **kwargs)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py",
line 9524, in post_live_migration_at_destination
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server with
excutils.save_and_reraise_exception():
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 227, in __exit__
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/oslo_utils/excutils.py",
line 200, in force_reraise
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server raise self.value
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/compute/manager.py",
line 9520, in post_live_migration_at_destination
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
self.driver.post_live_migration_at_destination(
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 11606, in post_live_migration_at_destination
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
self._reattach_instance_vifs(context, instance, network_info)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 11158, in _reattach_instance_vifs
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
self.attach_interface(context, instance,
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/driver.py",
line 2976, in attach_interface
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server cfg =
self.vif_driver.get_config(instance, vif, image_meta,
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/vif.py",
line 597, in get_config
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server return
self.get_config_hw_veb(*args)
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server File
"/openstack/venvs/nova-29.0.1/lib/python3.10/site-packages/nova/virt/libvirt/vif.py",
line 364, in get_config_hw_veb
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server conf, net_type,
profile['pci_slot'],
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server KeyError: 'pci_slot'
2024-09-26
16:51:22.795 1469383 ERROR oslo_messaging.rpc.server
We are running 2024.1 deployed via OpenStack-Ansible
On KVM with linuxbridge.
/openstack/venvs/nova-29.0.1/bin/pip show nova
Name: nova
Version: 29.1.0
Summary: Cloud computing fabric controller
Home-page: https://docs.openstack.org/nova/latest/
Author: OpenStack
Author-email: [email protected]
License: UNKNOWN
Location: /openstack/venvs/nova-29.0.1/lib/python3.10/site-packages
Requires: alembic, castellan, cryptography, cursive, decorator, eventlet,
futurist, greenlet, iso8601, Jinja2, jsonschema, keystoneauth1,
keystonemiddleware, lxml, microversion-parse, netaddr, netifaces, openstacksdk,
os-brick, os-resource-classes, os-service-types, os-traits, os-vif, oslo.cache,
oslo.concurrency, oslo.config, oslo.context, oslo.db, oslo.i18n, oslo.limit,
oslo.log, oslo.messaging, oslo.middleware, oslo.policy, oslo.privsep,
oslo.reports, oslo.rootwrap, oslo.serialization, oslo.service,
oslo.upgradecheck, oslo.utils, oslo.versionedobjects, paramiko, Paste,
PasteDeploy, pbr, PrettyTable, psutil, python-cinderclient, python-dateutil,
python-glanceclient, python-neutronclient, PyYAML, requests, retrying, rfc3986,
Routes, SQLAlchemy, stevedore, tooz, WebOb, websockify
Required-by:
/openstack/venvs/neutron-29.0.1/bin/pip show neutron
Name: neutron
Version: 24.0.1.dev22
Summary: OpenStack Networking
Home-page: https://docs.openstack.org/neutron/latest/
Author: OpenStack
Author-email: [email protected]
License:
Location: /openstack/venvs/neutron-29.0.1/lib/python3.10/site-packages
Requires: alembic, debtcollector, decorator, eventlet, futurist, httplib2,
Jinja2, keystoneauth1, keystonemiddleware, netaddr, netifaces, neutron-lib,
openstacksdk, os-ken, os-resource-classes, os-vif, oslo.cache,
oslo.concurrency, oslo.config, oslo.context, oslo.db, oslo.i18n, oslo.log,
oslo.messaging, oslo.middleware, oslo.policy, oslo.privsep, oslo.reports,
oslo.rootwrap, oslo.serialization, oslo.service, oslo.upgradecheck, oslo.utils,
oslo.versionedobjects, osprofiler, ovs, ovsdbapp, Paste, PasteDeploy, pbr,
pecan, psutil, pyOpenSSL, pyroute2, python-designateclient,
python-neutronclient, python-novaclient, requests, Routes, SQLAlchemy,
stevedore, tenacity, tooz, WebOb
Required-by:
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2082541/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
