** Description changed: [Impact] Live migration of an instance with an ephemeral disk fails when the destination host needs to create a missing ephemeral backing file while using QCOW2. Nova uses a long name such as 'ephemeral_10_0706d66' as the filesystem label. VFAT only supports labels up to 11 characters, so mkfs fails and the live migration does not complete: mkfs -t vfat -n ephemeral_10_0706d66 /var/lib/nova/instances/_base/ephemeral_10_0706d66 mkfs.vfat: Label can be no longer than 11 characters The upstream fix is: https://review.opendev.org/c/openstack/nova/+/947541 The fix uses a short name such as 'ephemeral0', while keeping the backing file name unchanged: mkfs -t vfat -n ephemeral0 /var/lib/nova/instances/_base/ephemeral_10_0706d66 I tested the bug in the following environments: - Jammy/Yoga: Nova 3:25.2.1-0ubuntu2.11 - Noble/Caracal: Nova 3:29.2.0-0ubuntu1.8 - Noble/Epoxy: Nova 3:31.0.0-0ubuntu1.2~cloud3 - Resolute/Gazpacho: Nova 3:33.0.0-0ubuntu3.1 - Stonking/Hibiscus: Nova 3:34.0.0~rc1-0ubuntu1 The bug reproduced with the above error message on Jammy/Yoga, Noble/Caracal and Noble/Epoxy. For Resolute/Gazpacho, I used a single-node setup due to compatibility issues. I deleted the backing file on the same host and hard-rebooted the instance, which triggered the same execution path described in the upstream change. I then confirmed that the bug did not reproduce. For Stonking/Hibiscus, I also confirmed that the bug did not reproduce with the same single-node setup, although I had to manually override the broken unit file to run glance-api using WSGI. The bug has been present in Nova for more than 12 years. The original call to _create_ephemeral() in _create_images_and_backing() was added by this commit in 2014: https://review.opendev.org/c/openstack/nova/+/68658 In upstream, the fix was backported to all maintained and unmaintained branches. The fix is available in Dalmatian 30.3.0 and Epoxy 31.3.1, and in the initial releases of Flamingo and Gazpacho. In Ubuntu, the fix is present in Resolute and Stonking (devel) for UA, and in Flamingo and Gazpacho for UCA. The affected releases are Jammy and Noble for UA, and Yoga, Caracal and Epoxy for UCA. I applied the upstream patch and verified that the same live migration succeeded. The missing ephemeral backing file was created with the short label 'ephemeral0', and mkfs completed successfully. [Test Case] The test requires an environment with two Nova compute hosts. Configure nova-compute on both hosts to use QEMU, qcow2 and VFAT: sudo apt update sudo apt install -y crudini sudo crudini --set /etc/nova/nova.conf libvirt virt_type qemu sudo crudini --set /etc/nova/nova.conf libvirt images_type qcow2 sudo crudini --set /etc/nova/nova.conf libvirt cpu_mode custom sudo crudini --set /etc/nova/nova.conf libvirt cpu_model qemu64 sudo crudini --set /etc/nova/nova.conf DEFAULT default_ephemeral_format vfat sudo crudini --set /etc/nova/nova.conf DEFAULT debug true sudo systemctl restart nova-compute Create a test image: curl -fL -o /tmp/cirros.img \ https://download.cirros-cloud.net/0.6.3/cirros-0.6.3-x86_64-disk.img openstack image create testimg \ --file /tmp/cirros.img \ --disk-format qcow2 \ --container-format bare \ --private Create a test flavour: openstack flavor create testflv \ --ram 1024 \ --disk 5 \ --ephemeral 5 \ --vcpus 1 Launch a Nova instance: openstack server create \ --image testimg \ --flavor testflv \ --network private \ --wait \ testsvr Identify the host names of the source and destination hosts: openstack hypervisor list +--------------------------------------+-----------------------------+-----------------+---------------+-------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | +--------------------------------------+-----------------------------+-----------------+---------------+-------+ | 957ab3ee-e579-454b-a9ae-6c69ea0852c6 | juju-ecb72d-noble-caracal-9 | QEMU | 10.159.67.47 | up | | f326dd68-279d-4897-88f7-074a656a18c2 | juju-ecb72d-noble-caracal-8 | QEMU | 10.159.67.105 | up | +--------------------------------------+-----------------------------+-----------------+---------------+-------+ openstack server show testsvr -f value -c OS-EXT-SRV-ATTR:host juju-ecb72d-noble-caracal-8 SOURCE_HOST=juju-ecb72d-noble-caracal-8 DEST_HOST=juju-ecb72d-noble-caracal-9 # Choose the other host Request a live migration to the destination host: export OS_COMPUTE_API_VERSION=2.30 openstack server migrate \ --live-migration \ --block-migration \ --host "$DEST_HOST" \ testsvr Wait for the live migration to finish: openstack server show testsvr \ -f value \ -c OS-EXT-STS:task_state \ -c OS-EXT-SRV-ATTR:host SSH into the destination host and search for the mkfs error message: sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail ... 2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Command: mkfs -t vfat -n ephemeral_5_0706d66 /var/lib/nova/instances/_base/ephemeral_5_0706d66 2026-09-18 09:05:14.508 52421 ERROR oslo_messaging.rpc.server Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n' Now apply the upstream patch and follow the same steps. Make sure to create a test flavour with a different value for --ephemeral so that the existing backing file is not reused: # Change --ephemeral 5 to 6 openstack flavor create testflv2 \ --ram 1024 \ --disk 5 \ --ephemeral 6 \ --vcpus 1 openstack server create \ --image testimg \ --flavor testflv2 \ --network private \ --wait \ testsvr2 After following the rest of the steps, I verified that mkfs was called with the short label and that the live migration succeeded with the patch: sudo grep -E 'mkfs.*vfat' /var/log/nova/nova-compute.log | tail ... 2026-09-18 09:22:49.407 3788355 DEBUG oslo_concurrency.processutils [None req-d2e72a6c-00c1-4df2-b70b-cd3b167330b0 73d496ce27ea461a8e5f4a1397d90d91 409a8037cb4f414f9d62a5ac852fe243 - - 6adae1c4f64747f595b2980622166731 6adae1c4f64747f595b2980622166731] Running cmd (subprocess): mkfs -t vfat -n ephemeral0 /var/lib/nova/instances/_base/ephemeral_6_0706d66 execute /usr/lib/python3/dist-packages/oslo_concurrency/processutils.py:390 ... [Regression Potential] The change only affects the filesystem label when Nova creates an ephemeral disk or recreates a missing backing file. The backing file name is unchanged. A regression could prevent a new ephemeral disk from being formatted, or - could affect systems that depends on the filesystem label. + could affect guest software that depends on the filesystem label. [Discussion] Although UCA Epoxy was initially marked Fix Released on the bug report, the current UCA Epoxy package does not contain the fix. I have therefore updated the status to In Progress. Also note that a second live migration attempt to the same destination host may appear to succeed because the first attempt leaves the backing file in place. However, this is not a workaround because the backing file is left unformatted: sudo find /var/lib/nova/instances/_base \ -maxdepth 1 -type f \ -name 'ephemeral_*' ... /var/lib/nova/instances/_base/ephemeral_11_0706d66 sudo blkid /var/lib/nova/instances/_base/ephemeral_11_0706d66 && echo formatted || echo unformatted unformatted Original Bug Description Below =========== When testing NBD TLS live-migration the initial attempt failed. Looks like a generated filesystem label is too long. Subsequent attempts succeeded $ nova-compute --version Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. 27.2.1 oslo_messaging.rpc.client.RemoteError: Remote error: ProcessExecutionError Unexpected error while running command. Command: mkfs -t vfat -n ephemeral_1_0706d66 /var/lib/nova/instances/_base/ephemeral_1_0706d66 Exit code: 1 Stdout: 'mkfs.fat 4.2 (2021-01-31)\n' Stderr: 'mkfs.vfat: Label can be no longer than 11 characters\n'
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2061701 Title: [SRU] check_can_live_migrate_source fails with mkfs.vfat error: Label can be no longer than 11 characters To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2061701/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
