Public bug reported:
The name format of mediated devices in libvirt was recently changed from
`mdev_<uuid>` to `mdev_<uuid>_<parent>`, e.g.:
Old: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5`
New: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5_0000_c1_00_0`
This results in the following error:
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
[req-570c7e8f-0540-49fb-b2b0-8c2ac932e4dc - - - - -] Error updating resources
for node: ValueError: badly formed hexadecimal UUID string
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager Traceback (most recent
call last):
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line
9993, in _update_available_resource_for_node
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py",
line 895, in update_available_resource
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
self._update_available_resource(context, resources, startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py",
line 360, in inner
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return f(*args,
**kwargs)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py",
line 975, in _update_available_resource
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager self._update(context,
cn, startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py",
line 1227, in _update
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
self._update_to_placement(context, compute_node, startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 49, in
wrapped_f
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return
Retrying(*dargs, **dkw).call(f, *args, **kw)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 206, in
call
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return
attempt.get(self._wrap_exception)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 247, in get
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
six.reraise(self.value[0], self.value[1], self.value[2])
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/usr/local/lib/python3.6/site-packages/six.py", line 719, in reraise
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager raise value
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 200, in
call
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager attempt =
Attempt(fn(*args, **kwargs), attempt_number, False)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py",
line 1163, in _update_to_placement
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
self.driver.update_provider_tree(prov_tree, nodename)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 8355, in update_provider_tree
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager provider_tree,
nodename, allocations=allocations)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 8757, in _update_provider_tree_for_vgpu
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager inventories_dict =
self._get_gpu_inventories()
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 7597, in _get_gpu_inventories
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager count_per_parent =
self._count_mediated_devices(enabled_mdev_types)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 7538, in _count_mediated_devices
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager mediated_devices =
self._get_mediated_devices(types=enabled_mdev_types)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 7788, in _get_mediated_devices
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager device =
self._get_mediated_device_information(name)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 7769, in _get_mediated_device_information
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager "uuid":
libvirt_utils.mdev_name2uuid(cfgdev.name),
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/utils.py",
line 583, in mdev_name2uuid
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return
str(uuid.UUID(mdev_name[5:].replace('_', '-')))
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/usr/lib64/python3.6/uuid.py", line 140, in __init__
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager raise
ValueError('badly formed hexadecimal UUID string')
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager ValueError: badly formed
hexadecimal UUID string
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
Nova is assuming everything after `mdev_` is the UUID, but this string
now includes the parent PCI address.
Relevant commit in libvirt:
https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d
Environment: qemu/kvm, latest openstack xena (kolla)
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1951656
Title:
Nova fails to parse new libvirt mediated device name format
Status in OpenStack Compute (nova):
New
Bug description:
The name format of mediated devices in libvirt was recently changed from
`mdev_<uuid>` to `mdev_<uuid>_<parent>`, e.g.:
Old: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5`
New: `mdev_a12c7bf8_fcf4_4c3b_a256_604cda8e62d5_0000_c1_00_0`
This results in the following error:
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
[req-570c7e8f-0540-49fb-b2b0-8c2ac932e4dc - - - - -] Error updating resources
for node: ValueError: badly formed hexadecimal UUID string
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager Traceback (most recent
call last):
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line
9993, in _update_available_resource_for_node
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py",
line 895, in update_available_resource
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
self._update_available_resource(context, resources, startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/lockutils.py",
line 360, in inner
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return f(*args,
**kwargs)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py",
line 975, in _update_available_resource
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
self._update(context, cn, startup=startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py",
line 1227, in _update
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
self._update_to_placement(context, compute_node, startup)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 49, in
wrapped_f
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return
Retrying(*dargs, **dkw).call(f, *args, **kw)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 206, in
call
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return
attempt.get(self._wrap_exception)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 247, in get
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
six.reraise(self.value[0], self.value[1], self.value[2])
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/usr/local/lib/python3.6/site-packages/six.py", line 719, in reraise
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager raise value
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/retrying.py", line 200, in
call
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager attempt =
Attempt(fn(*args, **kwargs), attempt_number, False)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/resource_tracker.py",
line 1163, in _update_to_placement
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
self.driver.update_provider_tree(prov_tree, nodename)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 8355, in update_provider_tree
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager provider_tree,
nodename, allocations=allocations)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 8757, in _update_provider_tree_for_vgpu
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager inventories_dict =
self._get_gpu_inventories()
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 7597, in _get_gpu_inventories
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager count_per_parent =
self._count_mediated_devices(enabled_mdev_types)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 7538, in _count_mediated_devices
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager mediated_devices =
self._get_mediated_devices(types=enabled_mdev_types)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 7788, in _get_mediated_devices
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager device =
self._get_mediated_device_information(name)
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py",
line 7769, in _get_mediated_device_information
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager "uuid":
libvirt_utils.mdev_name2uuid(cfgdev.name),
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/utils.py",
line 583, in mdev_name2uuid
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager return
str(uuid.UUID(mdev_name[5:].replace('_', '-')))
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager File
"/usr/lib64/python3.6/uuid.py", line 140, in __init__
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager raise
ValueError('badly formed hexadecimal UUID string')
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager ValueError: badly formed
hexadecimal UUID string
2021-11-19 22:51:45.952 7 ERROR nova.compute.manager
Nova is assuming everything after `mdev_` is the UUID, but this string
now includes the parent PCI address.
Relevant commit in libvirt:
https://github.com/libvirt/libvirt/commit/3bd8181bc5548a0ce81107cbfb480dfdcba5679d
Environment: qemu/kvm, latest openstack xena (kolla)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1951656/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
