[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
This is caused because the libvirt-qemu user is added to the nova group as part of the nova-compute-libvirt package post-install script. Following up on comment #17 above, the user/group of the delta file changes from nova:nova to libvirt-qemu:kvm, whereas in comment #21 above, the user/group of the delta file changes to nova:kvm. Dropping libvirt-qemu from nova in /etc/group fixes this as a work- around. I'm building packages with a fix now and will get this fixed for ussuri and victoria. Marking the upstream bug as invalid. ** Changed in: nova Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: [SRU] Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
As background, adding libvirt-qemu user to the nova group was an attempt
to make /var/lib/nova/* directories more restricted, but that proved to
be difficult with ownership changes between changes nova and
libvirt/qemu.
** Summary changed:
- Creation of image (or live snapshot) from the existing VM fails if
libvirt-image-backend is configured to qcow2 starting from Ussuri
+ [SRU] Creation of image (or live snapshot) from the existing VM fails if
libvirt-image-backend is configured to qcow2 starting from Ussuri
** Also affects: nova (Ubuntu Groovy)
Importance: Critical
Assignee: Corey Bryant (corey.bryant)
Status: Triaged
** Also affects: nova (Ubuntu Focal)
Importance: Undecided
Status: New
** Changed in: nova (Ubuntu Focal)
Status: New => Triaged
** Changed in: nova (Ubuntu Focal)
Importance: Undecided => Critical
** Changed in: nova (Ubuntu Focal)
Assignee: (unassigned) => Corey Bryant (corey.bryant)
** Also affects: cloud-archive
Importance: Undecided
Status: New
** Also affects: cloud-archive/ussuri
Importance: Undecided
Status: New
** Also affects: cloud-archive/victoria
Importance: Undecided
Status: New
** Changed in: cloud-archive/ussuri
Status: New => Triaged
** Changed in: cloud-archive/victoria
Status: New => Triaged
** Changed in: cloud-archive/victoria
Importance: Undecided => Critical
** Changed in: cloud-archive/ussuri
Importance: Undecided => Critical
** Changed in: cloud-archive/victoria
Assignee: (unassigned) => Corey Bryant (corey.bryant)
** Changed in: cloud-archive/ussuri
Assignee: (unassigned) => Corey Bryant (corey.bryant)
** Description changed:
+ [Impact]
+
tl;dr
1) creating the image from the existing VM fails if qcow2 image backend is
used, but everything is fine if using rbd image backend in nova-compute.
2) openstack server image create --name fails with some unrelated error:
$ openstack server image create --wait 842fa12c-19ee-44cb-bb31-36d27ec9d8fc
HTTP 404 Not Found: No image found with ID
f4693860-cd8d-4088-91b9-56b2f173ffc7
== Details ==
Two Tempest tests ([1] and [2]) from the 2018.02 Refstack test lists [0]
are failing with the following exception:
49701867-bedc-4d7d-aa71-7383d877d90c
Traceback (most recent call last):
File
"/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py",
line 369, in create_image_from_server
waiters.wait_for_image_status(client, image_id, wait_until)
File
"/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/common/waiters.py",
line 161, in wait_for_image_status
image = show_image(image_id)
File
"/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/images_client.py",
line 74, in show_image
resp, body = self.get("images/%s" % image_id)
File
"/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py",
line 298, in get
return self.request('GET', url, extra_headers, headers)
File
"/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/services/compute/base_compute_client.py",
line 48, in request
method, url, extra_headers, headers, body, chunked)
File
"/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py",
line 687, in request
self._error_checker(resp, resp_body)
File
"/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/lib/common/rest_client.py",
line 793, in _error_checker
raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {'code': 404, 'message': 'Image not found.'}
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/images/test_images_oneserver.py",
line 69, in test_create_delete_image
wait_until='ACTIVE')
File
"/home/ubuntu/snap/fcbtest/14/.rally/verification/verifier-2d9cbf4d-fcbb-491d-848d-5137a9bde99e/repo/tempest/api/compute/base.py",
line 384, in create_image_from_server
image_id=image_id)
tempest.exceptions.SnapshotNotFoundException: Server snapshot image
d82e95b0-9c62-492d-a08c-5bb118d3bf56 not found.
So far I was able to identify the following:
1)
https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L69
invokes a "create image from server"
2) It fails with the following error message in the
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
It turns out the tempfile.mkdtemp() call in nova/utils.py creates the directory with the restrictive permissions, in our case 0o700. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
I moved this back to New for upstream nova. @Lee or anyone else from upstream nova, do you have an opinion on changing the chmod in nova/virt/libvirt/driver.py from: os.chmod(tmpdir, 0o701) to: st = os.stat(tmpdir) os.chmod(tmpdir, st.st_mode | stat.S_IXOTH) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
I'm still really confused by this but some thoughts on the nova os.chmod() call mentioned in an earlier commit that would fix this. If I chmod the tmp dir that gets created by nova (e.g. /var/lib/nova/instances/snapshots/tmpkajuir8o) to 755 just before the snapshot (after the nova chmod), the snapshot is successful. As mentioned in https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1896617/comments/18, the upstream nova code sets permissions for the tmp dir with: os.chmod(tmpdir, 0o701) That code has been that way since 2015, so it's not new in ussuri, see git blame: 824c3706a3e nova/virt/libvirt/driver.py (Nicolas Simonds 2015-07-23 12:47:24 -0500 2388) # NOTE(xqueralt): libvirt needs o+x in the tempdir 824c3706a3e nova/virt/libvirt/driver.py (Nicolas Simonds 2015-07-23 12:47:24 -0500 2389) os.chmod(tmpdir, 0o701) However, this seems like a heavy handed chmod if the goal, as the comment above it mentions, is to give libvirt o+x in the tempdir. I say this because it overrides any default permissions that were set previously by the operating system. It seems that this should really be a lighter touch such as the following (equivalent to chmod o+x tmpdir): st = os.stat(tmpdir) os.chmod(tmpdir, st.st_mode | stat.S_IXOTH) That would fix this bug for us, but still doesn't explain what changed in Ubuntu to cause this to fail. We did make some permissions changes in the nova package in focal but as compared above (with ussuri-proposed) file/directory permissions above in comment #21 I'm seeing no differences. ** Changed in: nova Status: Invalid => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
@Vlad, I had reported this bug previously on https://bugs.launchpad.net/nova/+bug/1885418. My research with different scenarios showed that the problem is probably related to the latest versions of libvirtd libraries and modules. I did test the previous versions of libvirtd and they worked seamlessly. So this may help you in resolving the issue. Thank you -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
@Vlad, confirmed on my own deployment that it is not fixed in ussuri- proposed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
** Changed in: nova Status: Incomplete => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
@Corey, you're mentioning that it works on instance with id 049f but it was in SHUTOFF state (thus it was working), but if you'd start the instance: $ os server list | 049f76c6-3f6d-4299-b332-bf4c264b8741 | ubuntu-tests2 | SHUTOFF | internal=10.0.0.30 | ubuntu-bionic-cloudimg | m1.medium | | 645f031e-7426-4ad0-8263-5cc35e8be8a8 | ubuntu-test | ACTIVE | internal=10.0.0.161, 172.27.86.120 | ubuntu-bionic-cloudimg | m1.medium | $ os server start 049f76c6-3f6d-4299-b332-bf4c264b8741 $ os server image create --wait 049f76c6-3f6d-4299-b332-bf4c264b8741 HTTP 404 Not Found: No image found with ID 9b5dd242-9d46-41de-955e-5ff97ef50d28 Same as in original issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
I'm fairly certain that 1885269 fixes this. @vlad it's working for node-06 and instance 049f76c6-3f6d-4299-b332-bf4c264b8741 on your deployment. I upgraded all of your nova-compute-kvm's to ussuri-proposed and it didn't work at first. Either it was something else I changed or a restart of libvirtd that was also needed and it is working now. I'm deploying a new ussuri-proposed to try there. Your nova-compute-kvm units have varying degress of changes from me. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
Some directory comparisons (after enabling ussuri-proposed with the fix for 1885269). I'm seeing no differences, except that the snapshot is successful for bionic-train and still fails for focal-ussuri: bionic-train: ubuntu@juju-d9-zaza-4dbb8b0e6cc9-21:~$ ls -al /var/lib/nova/instances/snapshots total 12 drwxr-xr-x 3 nova nova 4096 Sep 23 20:13 . drwxr-xr-x 6 nova nova 4096 Sep 23 19:55 .. drwx-x 2 nova nova 4096 Sep 23 20:13 tmpbd7qzli0 ubuntu@juju-d9-zaza-4dbb8b0e6cc9-21:~$ sudo ls -al /var/lib/nova/instances/snapshots/tmpbd7qzli0/ total 204 drwx-x 2 nova nova 4096 Sep 23 20:13 . drwxr-xr-x 3 nova nova 4096 Sep 23 20:13 .. -rw-r--r-- 1 nova nova 196928 Sep 23 20:13 d1af0f3a804e4109830ef78155b7a4ab.delta ubuntu@juju-d9-zaza-4dbb8b0e6cc9-21:~$ sudo ls -al /var/lib/nova/instances/snapshots/tmpbd7qzli0/ total 1731224 drwx-x 2 nova nova 4096 Sep 23 20:14 . drwxr-xr-x 3 nova nova 4096 Sep 23 20:13 .. -rw-r--r-- 1 nova nova 1149894656 Sep 23 20:14 d1af0f3a804e4109830ef78155b7a4ab -rw-r--r-- 1 nova kvm 622985216 Sep 23 20:14 d1af0f3a804e4109830ef78155b7a4ab.delta ubuntu@juju-d9-zaza-4dbb8b0e6cc9-21:~$ ls -al /var/lib/nova total 40 drwxr-xr-x 10 nova nova 4096 Sep 23 19:07 . drwxr-xr-x 52 root root 4096 Sep 23 19:10 .. drwxr-xr-x 2 nova root 4096 Sep 23 19:14 .ssh drwxr-xr-x 6 nova nova 4096 Sep 23 19:00 CA drwxr-xr-x 2 nova nova 4096 Jun 17 13:47 buckets drwxr-xr-x 2 nova nova 4096 Jun 17 13:47 images drwxr-xr-x 6 nova nova 4096 Sep 23 20:14 instances drwxr-xr-x 2 nova nova 4096 Jun 17 13:47 keys drwxr-xr-x 2 nova nova 4096 Jun 17 13:47 networks drwxr-xr-x 2 nova nova 4096 Jun 17 13:47 tmp focal-ussuri: ubuntu@node06:~$ sudo ls -al /var/lib/nova/instances/snapshots/ total 12 drwxr-xr-x 3 nova nova 4096 Sep 23 20:20 . drwxr-xr-x 6 nova nova 4096 Sep 23 20:02 .. drwx-x 2 nova nova 4096 Sep 23 20:20 tmpt_x2bd57 ubuntu@node06:~$ sudo ls -al /var/lib/nova/instances/snapshots/tmpt_x2bd57 total 204 drwx-x 2 nova nova 4096 Sep 23 20:20 . drwxr-xr-x 3 nova nova 4096 Sep 23 20:20 .. -rw-r--r-- 1 nova nova 197248 Sep 23 20:20 f7d09ac696a04cb5b31925850e4dcfef.delta ubuntu@node06:~$ ls -al /var/lib/nova total 40 drwxr-xr-x 10 nova nova 4096 Sep 23 09:39 . drwxr-xr-x 55 root root 4096 Sep 23 09:46 .. drwxr-xr-x 2 nova root 4096 Sep 23 09:57 .ssh drwxr-xr-x 6 nova nova 4096 Sep 23 09:37 CA drwxr-xr-x 2 nova nova 4096 May 16 00:08 buckets drwxr-xr-x 2 nova nova 4096 May 16 00:08 images drwxr-xr-x 6 nova nova 4096 Sep 23 20:02 instances drwxr-xr-x 2 nova nova 4096 May 16 00:08 keys drwxr-xr-x 2 nova nova 4096 May 16 00:08 networks drwxr-xr-x 2 nova nova 4096 May 16 00:08 tmp -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1896617] Re: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri
I'm fairly certain this is a package bug so I'm going to triage against the package for now. ** Also affects: nova (Ubuntu) Importance: Undecided Status: New ** Changed in: nova (Ubuntu) Status: New => Triaged ** Changed in: nova (Ubuntu) Importance: Undecided => Critical ** Changed in: nova (Ubuntu) Assignee: (unassigned) => Corey Bryant (corey.bryant) ** Changed in: charm-nova-compute Assignee: Corey Bryant (corey.bryant) => (unassigned) ** Changed in: charm-nova-compute Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1896617 Title: Creation of image (or live snapshot) from the existing VM fails if libvirt-image-backend is configured to qcow2 starting from Ussuri To manage notifications about this bug go to: https://bugs.launchpad.net/charm-nova-compute/+bug/1896617/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
