Public bug reported: Description =========== A TypeError occurs when using Nova to boot an instance with ephemeral storage encryption enabled.
When key management was moved from Nova to Castellan in the Newton release, the key retrieval return value was changed from being formatted as a list of unsigned ints (in the case of an octet stream) [1] to not being formatted at all [2]. Nova's dmcrypt still expects the retrieved key [3][4] to be formatted as an array of unsigned bytes [5]. References: [1] https://github.com/openstack/nova/blob/9702e5d2e9433c67067847c02fb100cd808a3596/nova/keymgr/barbican.py#L297-L323 [2] https://github.com/openstack/castellan/blob/8d3f1cf375d28047cfd5f251afbb3f6d3ea7cda5/castellan/key_manager/barbican_key_manager.py#L527-L549 [3] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L714-L715 [4] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L674-L680 [5] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/storage/dmcrypt.py#L48-L70 Steps to reproduce ================== 1. Set up an LVM device: Create a backing file: $ truncate nova-lvm -s 2G Mount the backing file on a loop device: $ sudo losetup /dev/loop1 nova-lvm Prepare the device for LVM: $ sudo pvcreate /dev/loop1 Create the LVM group on the loop device: $ sudo vgcreate nova-lvm /dev/loop1 2. Set up a devstack environment with ephemeral storage encryption enabled by adding the following lines to `lib/nova`: iniset $NOVA_CONF ephemeral_storage_encryption enabled "True" iniset $NOVA_CONF ephemeral_storage_encryption cipher "aes-xts-plain64" iniset $NOVA_CONF ephemeral_storage_encryption key_size "256" iniset $NOVA_CONF libvirt images_type "lvm" iniset $NOVA_CONF libvirt images_volume_group "nova-lvm" 3. Stack: $ ./stack 4. Use Nova to boot an instance: $ nova boot --flavor 1 --image {image_id} Expected result =============== Ephemeral storage encryption succeeds and Nova successfully boots the instance. Actual result ============= Ephemeral storage encryption fails with a TypeError (similar results can be seen from Barbican Tempest gate failures [1]): 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] Traceback (most recent call last): 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/compute/manager.py", line 2122, in _build_resources 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] yield resources 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/compute/manager.py", line 1927, in _build_and_run_instance 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] block_device_info=block_device_info) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2688, in spawn 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] block_device_info=block_device_info) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3095, in _create_image 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] fallback_from_host) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3211, in _create_and_inject_local_root 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] instance, size, fallback_from_host) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6780, in _try_fetch_image_cache 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] size=size) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 227, in cache 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] *args, **kwargs) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 735, in create_image 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] create_lvm_image(base, size) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] return f(*args, **kwargs) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 693, in create_lvm_image 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] encrypt_lvm_image() 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 680, in encrypt_lvm_image 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] key) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in create_volume 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] key = ''.join(map(lambda byte: "%02x" % byte, key)) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in <lambda> 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] key = ''.join(map(lambda byte: "%02x" % byte, key)) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] TypeError: %x format: a number is required, not str Nova fails to boot the instance. [1] http://logs.openstack.org/59/455459/16/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/9c0fa48/logs/screen-n-cpu.txt.gz#_2017-04-24_21_01_37_841 Environment =========== Latest `master` branch for all projects, except for Nova, which has patch [1] applied to get around the error addressed by the patch. However, remnants of the TypeError can still be seen without this patch [2]. [1] https://review.openstack.org/#/c/462348/ [2] http://logs.openstack.org/59/455459/17/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/26e8659/logs/screen-n-cpu.txt.gz#_2017-04-30_16_16_43_532 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1688342 Title: Encryption key is not properly formatted before being passed to dmcrypt Status in OpenStack Compute (nova): New Bug description: Description =========== A TypeError occurs when using Nova to boot an instance with ephemeral storage encryption enabled. When key management was moved from Nova to Castellan in the Newton release, the key retrieval return value was changed from being formatted as a list of unsigned ints (in the case of an octet stream) [1] to not being formatted at all [2]. Nova's dmcrypt still expects the retrieved key [3][4] to be formatted as an array of unsigned bytes [5]. References: [1] https://github.com/openstack/nova/blob/9702e5d2e9433c67067847c02fb100cd808a3596/nova/keymgr/barbican.py#L297-L323 [2] https://github.com/openstack/castellan/blob/8d3f1cf375d28047cfd5f251afbb3f6d3ea7cda5/castellan/key_manager/barbican_key_manager.py#L527-L549 [3] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L714-L715 [4] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/imagebackend.py#L674-L680 [5] https://github.com/openstack/nova/blob/d219a3dcdc9c51bff3ebf2df086ea61a840ea3e9/nova/virt/libvirt/storage/dmcrypt.py#L48-L70 Steps to reproduce ================== 1. Set up an LVM device: Create a backing file: $ truncate nova-lvm -s 2G Mount the backing file on a loop device: $ sudo losetup /dev/loop1 nova-lvm Prepare the device for LVM: $ sudo pvcreate /dev/loop1 Create the LVM group on the loop device: $ sudo vgcreate nova-lvm /dev/loop1 2. Set up a devstack environment with ephemeral storage encryption enabled by adding the following lines to `lib/nova`: iniset $NOVA_CONF ephemeral_storage_encryption enabled "True" iniset $NOVA_CONF ephemeral_storage_encryption cipher "aes-xts-plain64" iniset $NOVA_CONF ephemeral_storage_encryption key_size "256" iniset $NOVA_CONF libvirt images_type "lvm" iniset $NOVA_CONF libvirt images_volume_group "nova-lvm" 3. Stack: $ ./stack 4. Use Nova to boot an instance: $ nova boot --flavor 1 --image {image_id} Expected result =============== Ephemeral storage encryption succeeds and Nova successfully boots the instance. Actual result ============= Ephemeral storage encryption fails with a TypeError (similar results can be seen from Barbican Tempest gate failures [1]): 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] Traceback (most recent call last): 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/compute/manager.py", line 2122, in _build_resources 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] yield resources 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/compute/manager.py", line 1927, in _build_and_run_instance 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] block_device_info=block_device_info) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2688, in spawn 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] block_device_info=block_device_info) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3095, in _create_image 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] fallback_from_host) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3211, in _create_and_inject_local_root 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] instance, size, fallback_from_host) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6780, in _try_fetch_image_cache 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] size=size) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 227, in cache 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] *args, **kwargs) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 735, in create_image 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] create_lvm_image(base, size) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 271, in inner 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] return f(*args, **kwargs) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 693, in create_lvm_image 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] encrypt_lvm_image() 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 680, in encrypt_lvm_image 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] key) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in create_volume 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] key = ''.join(map(lambda byte: "%02x" % byte, key)) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] File "/opt/stack/nova/nova/virt/libvirt/storage/dmcrypt.py", line 64, in <lambda> 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] key = ''.join(map(lambda byte: "%02x" % byte, key)) 2017-05-04 11:51:23.531 TRACE nova.compute.manager [instance: 760dedd5-60d7-470e-b753-70ee65e3ec56] TypeError: %x format: a number is required, not str Nova fails to boot the instance. [1] http://logs.openstack.org/59/455459/16/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/9c0fa48/logs/screen-n-cpu.txt.gz#_2017-04-24_21_01_37_841 Environment =========== Latest `master` branch for all projects, except for Nova, which has patch [1] applied to get around the error addressed by the patch. However, remnants of the TypeError can still be seen without this patch [2]. [1] https://review.openstack.org/#/c/462348/ [2] http://logs.openstack.org/59/455459/17/check/gate-barbican-simple-crypto-dsvm-tempest-ubuntu-xenial-nv/26e8659/logs/screen-n-cpu.txt.gz#_2017-04-30_16_16_43_532 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1688342/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
