Public bug reported: This bug is related to resize with the libvirt driver
If you are performing a resize and it fails the _cleanup_remote_migration() [1] function in the libvirt driver will try to cleanup the /var/lib/nova/instances/<uuid>_resize directory on the remote side [2] - if this fails the <uuid>_resize directory will be left behind and block any future resize attempts. 2021-12-14 14:40:12.535 175177 INFO nova.virt.libvirt.driver [req-9d3477d4-3bb2-456f-9be6-dce9893b0e95 23d6aa8884ab44ef9f214ad195d273c0 050c556faa5944a8953126c867313770 - default default] [instance: 99287438-c37b-44b0-834e-55685b6e83eb] Deletion of /var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb_resize failed Then on next resize attempt a long time later 2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 10429, in migrate_disk_and_power_off 2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server os.rename(inst_base, inst_base_resize) 2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server FileExistsError: [Errno 17] File exists: '/var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb' -> '/var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb_resize' This is happens here [3] because os.rename tries to rename the /var/lib/nova/instances/<uuid> dir to <uuid>_resize that already exists and fails with FileExistsError. We should check if the directory exists before trying to rename and delete it before. [1] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10773 [2] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10965 [3] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10915 ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1960230 Title: resize fails with FileExistsError if earlier resize attempt failed to cleanup Status in OpenStack Compute (nova): New Bug description: This bug is related to resize with the libvirt driver If you are performing a resize and it fails the _cleanup_remote_migration() [1] function in the libvirt driver will try to cleanup the /var/lib/nova/instances/<uuid>_resize directory on the remote side [2] - if this fails the <uuid>_resize directory will be left behind and block any future resize attempts. 2021-12-14 14:40:12.535 175177 INFO nova.virt.libvirt.driver [req-9d3477d4-3bb2-456f-9be6-dce9893b0e95 23d6aa8884ab44ef9f214ad195d273c0 050c556faa5944a8953126c867313770 - default default] [instance: 99287438-c37b-44b0-834e-55685b6e83eb] Deletion of /var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb_resize failed Then on next resize attempt a long time later 2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 10429, in migrate_disk_and_power_off 2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server os.rename(inst_base, inst_base_resize) 2022-02-04 13:07:31.255 175177 ERROR oslo_messaging.rpc.server FileExistsError: [Errno 17] File exists: '/var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb' -> '/var/lib/nova/instances/99287438-c37b-44b0-834e-55685b6e83eb_resize' This is happens here [3] because os.rename tries to rename the /var/lib/nova/instances/<uuid> dir to <uuid>_resize that already exists and fails with FileExistsError. We should check if the directory exists before trying to rename and delete it before. [1] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10773 [2] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10965 [3] https://opendev.org/openstack/nova/src/branch/stable/xena/nova/virt/libvirt/driver.py#L10915 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1960230/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
