https://review.opendev.org/c/openstack/nova/+/765769 proposed to
stable/victoria
** Also affects: nova/queens
Importance: Undecided
Status: New
** Also affects: nova/rocky
Importance: Undecided
Status: New
** Also affects: nova/trunk
Importance: Undecided
Status: New
** Also affects: nova/train
Importance: Undecided
Status: New
** Also affects: nova/stein
Importance: Undecided
Status: New
** Also affects: nova/ussuri
Importance: Undecided
Status: New
** Also affects: nova/victoria
Importance: Undecided
Status: New
** No longer affects: nova/trunk
** Changed in: nova/victoria
Status: New => In Progress
** Changed in: nova/victoria
Assignee: (unassigned) => Lee Yarwood (lyarwood)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1905701
Title:
Do not recreate libvirt secret when one already exists on the host
during a host reboot
Status in OpenStack Compute (nova):
In Progress
Status in OpenStack Compute (nova) queens series:
In Progress
Status in OpenStack Compute (nova) rocky series:
In Progress
Status in OpenStack Compute (nova) stein series:
In Progress
Status in OpenStack Compute (nova) train series:
In Progress
Status in OpenStack Compute (nova) ussuri series:
In Progress
Status in OpenStack Compute (nova) victoria series:
In Progress
Bug description:
Description
===========
When [compute]/resume_guests_state_on_host_boot is enabled the compute
manager will attempt to restart instances on start up.
When using the libvirt driver and instances with attached LUKSv1
encrypted volumes a call is made to _attach_encryptor that currently
assumes that any volume libvirt secrets don't already exist on the
host. As a result this call will currently lead to an attempt to
lookup encryption metadata that fails as the compute service is using
a bare bones local only admin context to drive the restart of the
instances.
The libvirt secrets associated with LUKSv1 encrypted volumes actually
persist a host reboot and thus this call to fetch encryption metadata,
fetch the symmetric key etc are not required. Removal of these calls
in this context should allow the compute service to start instances
with these volumes attached.
Steps to reproduce
==================
* Enable [compute]/resume_guests_state_on_host_boot
* Launch instances with encrypted LUKSv1 volumes attached
* Reboot the underlying host
Expected result
===============
* The instances are restarted successfully by Nova as no external calls are
made and the existing libvirt secret for any encrypted LUKSv1 volumes are
reused.
Actual result
=============
* The instances fail to restart as the initial calls made by the Nova service
use an empty admin context without a service catelog etc.
Environment
===========
1. Exact version of OpenStack you are running. See the following
master
2. Which hypervisor did you use?
(For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
What's the version of that?
libvirt + QEMU/KVM
2. Which storage type did you use?
(For example: Ceph, LVM, GPFS, ...)
What's the version of that?
N/A
3. Which networking type did you use?
(For example: nova-network, Neutron with OpenVSwitch, ...)
N/A
Logs & Configs
==============
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File
"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1641, in
_connect_volume
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] self._attach_encryptor(context,
connection_info, encryption)
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File
"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1760, in
_attach_encryptor
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] key = keymgr.get(context,
encryption['encryption_key_id'])
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File
"/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py",
line 575, in get
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] secret = self._get_secret(context,
managed_object_id)
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File
"/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py",
line 545, in _ge
t_secret
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] barbican_client =
self._get_barbican_client(context)
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File
"/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py",
line 142, in _ge
t_barbican_client
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] self._barbican_endpoint)
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File
"/usr/lib/python3.6/site-packages/castellan/key_manager/barbican_key_manager.py",
line 214, in _cr
eate_base_url
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] service_type='key-manager')
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] File
"/usr/lib/python3.6/site-packages/keystoneauth1/access/service_catalog.py",
line 425, in endpoint_
data_for
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf] raise exceptions.EmptyCatalog('The
service catalog is empty.')
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf]
keystoneauth1.exceptions.catalog.EmptyCatalog: The service catalog is empty.
2020-08-20 11:30:12.273 7 ERROR nova.virt.libvirt.driver [instance:
c5b3e7d4-99ea-409c-aba6-d32751f93ccf]
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1905701/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
