[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
This bug was fixed in the package libvirt - 1.3.1-1ubuntu10.29 --- libvirt (1.3.1-1ubuntu10.29) xenial; urgency=medium * debian/patches/lp1681839-*.patch: Fix block commit timeout races, and ensure that once commit has reached 100%, timeouts no longer apply. (LP: #1681839) -- Matthew Ruffell Thu, 31 Oct 2019 10:52:41 +1300 ** Changed in: libvirt (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
The customer has been unresponsive in testing the package in -proposed in their environment, so we will move on with verification. In my previous comment on 2019-11-27, we showed that libvirt 1.3.1-1ubuntu10.29 can successfully execute a blockcommit on a lvm backed volume with virsh. This still holds today, and this bug has had ample time to soak in -proposed, making me happy to mark this as verified. ** Tags removed: verification-needed verification-needed-xenial ** Tags added: verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
The following is verification performed by Fabio in a lab: - Tested with the original libvirt to make sure I was able to reproduce: root@ubuntu:~# apt-cache policy libvirt-bin libvirt-bin: Installed: 1.3.1-1ubuntu10.27 Candidate: 1.3.1-1ubuntu10.27 Version table: *** 1.3.1-1ubuntu10.27 500 500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages 100 /var/lib/dpkg/status 1.3.1-1ubuntu10 500 500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages root@ubuntu:~# virsh blockcommit testvm vda --active --verbose --pivot --wait Block commit: [100 %]error: failed to pivot job for disk vda error: block copy still active: disk 'vda' not ready for pivot yet - Upgraded to proposed and tested again, and problem is gone: root@ubuntu:~# apt-cache policy libvirt-bin libvirt-bin: Installed: 1.3.1-1ubuntu10.29 Candidate: 1.3.1-1ubuntu10.29 Version table: *** 1.3.1-1ubuntu10.29 500 500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages 100 /var/lib/dpkg/status 1.3.1-1ubuntu10.27 500 500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages 1.3.1-1ubuntu10 500 500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages root@ubuntu:~# virsh blockcommit testvm vda --active --verbose --pivot --wait Block commit: [100 %] Successfully pivoted End of test by Fabio. The package is looking good. We have also asked the customer to install the test package and verify that it works under their workload. We might just wait for their confirmation before marking this as verified, in order to give this a little more time to soak in -proposed. Will update again soon. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
@mruffel / @fabiomartins - would you be so kind doing the SRU verification on this one? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
I thank you for all your work.
I have since moved away from this block architecture and am no longer able
to verify with an existing configuration.
On Fri, Nov 22, 2019 at 6:36 AM Timo Aaltonen
wrote:
> Hello Patrick, or anyone else affected,
>
> Accepted libvirt into xenial-proposed. The package will build now and be
> available at
> https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu10.29 in a few
> hours, and then in the -proposed repository.
>
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
> to enable and use -proposed. Your feedback will aid us getting this
> update out to other Ubuntu users.
>
> If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested and change the tag from
> verification-needed-xenial to verification-done-xenial. If it does not
> fix the bug for you, please add a comment stating that, and change the
> tag to verification-failed-xenial. In either case, without details of
> your testing we will not be able to proceed.
>
> Further information regarding the verification process can be found at
> https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
> advance for helping!
>
> N.B. The updated package will be released to -updates after the bug(s)
> fixed by this package have been verified and the package has been in
> -proposed for a minimum of 7 days.
>
> ** Changed in: libvirt (Ubuntu Xenial)
>Status: In Progress => Fix Committed
>
> ** Tags added: verification-needed verification-needed-xenial
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1681839
>
> Title:
> libvirt: blockcommit fails - disk not ready for pivot yet
>
> Status in libvirt package in Ubuntu:
> Fix Released
> Status in libvirt source package in Xenial:
> Fix Committed
> Status in libvirt source package in Artful:
> Fix Released
> Status in libvirt source package in Bionic:
> Fix Released
>
> Bug description:
> [Impact]
>
> On xenial, if you manually invoke blockcommit through virsh in
> libvirt, the command immediately fails with blockcommit supposedly
> being 100%, and that the disk is not ready for pivot yet:
>
> root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose
> --pivot --wait
> Block commit: [100 %]
> error: failed to pivot job for disk vda
> error: block copy still active: disk 'vda' not ready for pivot yet
>
> However, if you look at the status of the active blockjob, we see that
> the blockcommit is still active in the background:
>
> root@xenial-apparmor:~# virsh blockjob snapvm vda --info
> Active Block Commit: [0 %]
>
> root@xenial-apparmor:~# virsh blockjob snapvm vda --info
> Active Block Commit: [2 %]
>
> root@xenial-apparmor:~# virsh blockjob snapvm vda --info
> Active Block Commit: [6 %]
>
> This happens until it reaches 100%, where it gets stuck. To un-stick
> things, you must then manually --abort the blockjob.
>
> root@xenial-apparmor:~# virsh blockjob snapvm vda --info
> Active Block Commit: [100 %]
>
> This happens in VMs which are experiencing load, and is caused by a
> race condition in libvirt. Users are not able to commit their
> snapshots to disk.
>
> [Test Case]
>
> Credit goes to Fabio Martins, who determined how to reproduce this
> issue.
>
> On a Ubuntu 16.04 host with libvirt 1.3.1-1ubuntu10.27:
>
> 1) Create a VG and define a LVM pool:
>
> root@xenial-apparmor:~# cat lvmpool.xml
>
> LVMpool_vg
>
>
>
>
> /dev/LVMpool_vg
>
>
>
> # virsh pool-define lvmpool.xml
> # virsh pool-start LVMpool_vg
> # virsh pool-autostart LVMpool_vg
>
> 2) Create a config file to use as a cdrom device with the new VM (will
> be created in next steps), just to inject a password with cloud-init:
>
> # cat > config < > #cloud-config
> > password: passw0rd
> > chpasswd: { expire: False }
> > ssh_pwauth: True
> > EOF
>
> # apt install cloud-image-utils
>
> # cloud-localds config.img config
>
> # mv config.img /var/lib/libvirt/images/
> # chown libvirt-qemu:kvm /var/lib/libvirt/images/config.img
> # chmod 664 /var/lib/libvirt/images/config.img
>
> 3) Create one VM using this pool:
>
> # virt-install --connect=qemu:///system --name snapvm --ram 2048
> --vcpus=1 --os-type=linux --disk pool=LVMpool_vg,size=15,bus=virtio
> --disk /var/lib/libvirt/images/config.img,device=cdrom --network
> network=kvm-br0 --graphics none --import --noautoconsole
>
> 4) Stop the VM
>
> # virsh destroy snapvm
>
> 5) Download a Ubuntu cloud image, convert to raw and restore it into
> the LV used as a disk by our VM:
>
> # wget
> https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
> # qemu-img convert ./bionic-server-cloudimg-amd64.img
> ./bionic-server-cloudimg-amd64.raw
> # dd
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
Hello Patrick, or anyone else affected, Accepted libvirt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu10.29 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: libvirt (Ubuntu Xenial) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
FYI: my pre-checks on the PPA build 1.3.1-1ubuntu10.29~ppa1 look good. prep (x86_64): Pass 25 F/S/N 0/0/0 - RC 0 (17 min 84141 lin) migrate (x86_64) : Pass 232 F/S/N 0/12/0 - RC 0 (63 min 104302 lin) cross (x86_64) : Pass 64 F/S/N 0/1/0 - RC 0 (70 min 94458 lin) misc (x86_64): Pass 76 F/S/N 0/0/0 - RC 0 (18 min 29606 lin) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
Hi Mathew, thanks for picking up the torch again on this issue that affected quite some people, but but so far never reached the state to be really fixable. ## Verification ## Most importantly was to get some repro for test and verification. This was formerly a big issue, it affected plenty of people on the bug but we never reached a state to reliably reproduce it to verify its effects. I have read your update to the Description - thanks for adding all that. I was giving the new repro steps outlined there a try as you have seen above I can confirm that they are good \o/ ## Patches ## You included 4 patches which exactly matched what I identified a while ago - thanks. As I said back then in comment #6 they seemed rather backportable, thanks for doing that in the debdiff applied. They miss proper dep3 tagging but I can fix this up ahead sponsoring for you. ## Regression Risk ## I don't fully agree to the regression potential. You didn't say anything wrong, just from lessons-learned int he past blockjobs have turned out to be a source of unexpected and sometime strange regressions. I agree that it should (tm) be safe, but we should be extra cautios as well. Once it is in proposed we'd want more time there and probably should do some extra tests. ## Testing in proposed ## 1) I can provide some regression testing on my own, with a focus but not exclusive on migration. Mine isn't that heavy on snapshots where this certainly has the biggest chance of an impact. 2) @Matthew - if you could provide more tests (mabye SEG has some on top) for regressions in general that would be great. 3) @Matthew - we might consider going to e.g. the Openstack Team to run a test set on it as well just to be on the safe side. Will you ping and ask them or should I do so? ## PPA ## The old PPA I had is long dead. I opened a new one (like yours but with my minimal patch header updates and builds on all architectures) at: => https://launchpad.net/~paelzer/+archive/ubuntu/bug-1681839-blockjob-timeout-xenial/+packages ## Sponsoring ## This LGTM as-is from the patches, but as mentioned we should do tests 1+2+3 The SRU team can already take a look at accepting it, we can either test from the PPA or against xenial-proposed once accepted. The only one that strictly has to be on -proposed is the actual verification of the case then. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
Thanks++ Now that I had (new) steps to recreate I could work on those. I wondered if an LVM is really strictly needed - which would also easen the initialization. So I simplified it to. $ apt install uvtool-libvirt $ uvt-simplestreams-libvirt sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial $ uvt-kvm create xsnaptest arch=amd64 release=xenial label=daily # depending on your apparmor config you might want to add something like this TEMPORARY to /etc/apparmor.d/abstractions/libvirt-qemu '/var/lib/uvtool/libvirt/images/* rwk,' $ virsh snapshot-create-as --domain xsnaptest --diskspec vda,file=/var/lib/libvirt/images/xsnaptest-snapshot.qcow2,snapshot=external --disk-only --atomic I started a loop which on one side dirtied the snapshot and on the other pivoted it. # make dirty: $ while /bin/true; do uvt-kvm ssh --insecure xsnaptest "dd if=/dev/urandom of=file.txt count=4096 bs=1M"; done # snapshot, wait and pivot blockcommit $ while virsh blockcommit xsnaptest vda --active --verbose --pivot --wait; do rm /var/lib/libvirt/images/xsnaptest-snapshot.qcow2; sleep 2s; virsh snapshot-create-as --domain xsnaptest --diskspec vda,file=/var/lib/libvirt/images/xsnaptest-snapshot.qcow2,snapshot=external --disk-only --atomic; sleep $(( RANDOM % 30 ))s; ll -h /var/lib/libvirt/images/xsnaptest-snapshot.qcow2; done The snapshots to commit were about 200M to 6.9G, but none triggered the issue (about 40 tries in the loop). So maybe really it only happens (much more likely) when the original backing to write back is an LVM. Glad you found that for your test to become a reliable reproducer. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
Tagged and sponsored to Xenial-unapproved. Now it is up to the SRU Team. @Matthew - please try to get as much testing in place as possible. As I said all but the final verification can be done either on the PPA in advance or once in -proposed. As it fits your time and setup. I'll setup a test on my own as I mentioned ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
For the sake of seeing it trigger at least once I redeployed a mchine with Xenial to create LVMs there on a free /dev/sdb disk as your example needs it. # create guest $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial $ uvt-kvm create xsnaptest arch=amd64 release=xenial label=daily # create Volume $ sudo pvcreate /dev/sdb $ sudo vgcreate LVMpool_vg /dev/sdb $ cat > lvmpool.xml < LVMpool_vg /dev/LVMpool_vg EOF $ virsh pool-define lvmpool.xml $ sudo vgcreate LVMpool_vg /dev/sdb $ virsh pool-start LVMpool_vg $ virsh vol-create-as LVMpool_vg lvvol1 15G # Use volume in the guest $ cat > lvmdisk.xml < EOF $ virsh attach-device xsnaptest lvmdisk.xml # Prep initial snapshot virsh snapshot-create-as --domain xsnaptest --diskspec vdc,file=/var/lib/libvirt/images/xsnaptest-snapshot.qcow2,snapshot=external --disk-only --atomic # Check snapshot being backed by lvmdisk $ sudo qemu-img info /var/lib/libvirt/images/xsnaptest-snapshot.qcow2 image: /var/lib/libvirt/images/xsnaptest-snapshot.qcow2 file format: qcow2 virtual size: 15G (16106127360 bytes) disk size: 196K cluster_size: 65536 backing file: /dev/LVMpool_vg/lvvol1 backing file format: raw Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false # dump I/O onto that device from inside the guest $ while /bin/true; do uvt-kvm ssh --insecure xsnaptest "sudo dd if=/dev/urandom of=/dev/vdc count=8192 bs=1M"; done # Iterate on it while the disk/snapshot keeps getting dirty $ while virsh blockcommit xsnaptest vdc --active --verbose --pivot --wait; do sudo rm /var/lib/libvirt/images/xsnaptest-snapshot.qcow2; sleep 2s; virsh snapshot-create-as --domain xsnaptest --diskspec vdc,file=/var/lib/libvirt/images/xsnaptest-snapshot.qcow2,snapshot=external --disk-only --atomic; sleep $(( RANDOM % 30 + 20 ))s; sudo ls -laFh /var/lib/libvirt/images/xsnaptest-snapshot.qcow2; done Finally I saw it in action Block commit: [100 %]error: failed to pivot job for disk vdc error: block copy still active: disk 'vdc' not ready for pivot yet I retried and this was reproducible. I upgraded to the PPA (more about that later) and ran my loop. It reached 100% and then got slow (I/O while doing the pivot). I needed to either wait quite a while or slow down the ongoing I/O a bit. I had the loop running a 10 times and with the fix it never failed again (sized between 519M and 7.1G). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
Attached is the debdiff for xenial to fix this issue. I was not sure if the patches in debian/patches should be placed in the debian/patches/ubuntu directory or not, so I left them outside. Feel free to move them if necessary. ** Patch added: "libvirt debdiff for xenial" https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+attachment/5301911/+files/lp1681839_xenial.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1681839 Title: libvirt: blockcommit fails - disk not ready for pivot yet To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1681839] Re: libvirt: blockcommit fails - disk not ready for pivot yet
** Summary changed:
- libvirt - disk not ready for pivot yet
+ libvirt: blockcommit fails - disk not ready for pivot yet
** Description changed:
- root@thewind:/home/bestpa/scripts# virsh blockcommit mail vda --active
--verbose --pivot
- Block commit: [100 %]error: failed to pivot job for disk vda
+ [Impact]
+
+ On xenial, if you manually invoke blockcommit through virsh in libvirt,
+ the command immediately fails with blockcommit supposedly being 100%,
+ and that the disk is not ready for pivot yet:
+
+ root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose
--pivot --wait
+ Block commit: [100 %]
+ error: failed to pivot job for disk vda
error: block copy still active: disk 'vda' not ready for pivot yet
- found related bugfix at redhat... can i get 1.3.2 pushed into ubuntu
- 16.04 release?
-
- bestpa@thewind:~$ cat /etc/os-release
- NAME="Ubuntu"
- VERSION="16.04.2 LTS (Xenial Xerus)"
- ID=ubuntu
- ID_LIKE=debian
- PRETTY_NAME="Ubuntu 16.04.2 LTS"
-
- bestpa@thewind:~$ libvirtd --version
- libvirtd (libvirt) 1.3.1
+ However, if you look at the status of the active blockjob, we see that
+ the blockcommit is still active in the background:
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [0 %]
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [2 %]
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [6 %]
+
+ This happens until it reaches 100%, where it gets stuck. To un-stick
+ things, you must then manually --abort the blockjob.
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [100 %]
+
+ This happens in VMs which are experiencing load, and is caused by a race
+ condition in libvirt. Users are not able to commit their snapshots to
+ disk.
+
+ [Test Case]
+
+ Credit goes to Fabio Martins, who determined how to reproduce this
+ issue.
+
+ On a Ubuntu 16.04 host with libvirt 1.3.1-1ubuntu10.27:
+
+ 1) Create a VG and define a LVM pool:
+
+ root@xenial-apparmor:~# cat lvmpool.xml
+
+ LVMpool_vg
+
+
+
+
+ /dev/LVMpool_vg
+
+
+
+ # virsh pool-define lvmpool.xml
+ # virsh pool-start LVMpool_vg
+ # virsh pool-autostart LVMpool_vg
+
+ 2) Create a config file to use as a cdrom device with the new VM (will
+ be created in next steps), just to inject a password with cloud-init:
+
+ # cat > config < #cloud-config
+ > password: passw0rd
+ > chpasswd: { expire: False }
+ > ssh_pwauth: True
+ > EOF
+
+ # apt install cloud-image-utils
+
+ # cloud-localds config.img config
+
+ # mv config.img /var/lib/libvirt/images/
+ # chown libvirt-qemu:kvm /var/lib/libvirt/images/config.img
+ # chmod 664 /var/lib/libvirt/images/config.img
+
+ 3) Create one VM using this pool:
+
+ # virt-install --connect=qemu:///system --name snapvm --ram 2048
+ --vcpus=1 --os-type=linux --disk pool=LVMpool_vg,size=15,bus=virtio
+ --disk /var/lib/libvirt/images/config.img,device=cdrom --network network
+ =kvm-br0 --graphics none --import --noautoconsole
+
+ 4) Stop the VM
+
+ # virsh destroy snapvm
+
+ 5) Download a Ubuntu cloud image, convert to raw and restore it into the
+ LV used as a disk by our VM:
+
+ # wget
https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
+ # qemu-img convert ./bionic-server-cloudimg-amd64.img
./bionic-server-cloudimg-amd64.raw
+ # dd if=./bionic-server-cloudimg-amd64.raw of=/dev/LVMpool_vg/snapvm bs=8M
conv=sparse
+
+ 6) Start the VM and connect to it in another window
+
+ # virsh start snapvm
+
+ 7) Check that the VM is using the LV as the disk:
+
+ root@xenial-apparmor:~# virsh domblklist snapvm
+ Target Source
+
+ vda /dev/LVMpool_vg/snapvm
+ hda /var/lib/libvirt/images/config.img
+
+ 8) Create a snapshot and check that the new domblklist points to the
+ snapshot file:
+
+ # virsh snapshot-create-as --domain snapvm --diskspec
+ vda,file=/var/lib/libvirt/images/xenial-snapvm.qcow2,snapshot=external
+ --disk-only --atomic
+
+ root@xenial-apparmor:~# virsh domblklist snapvm
+ Target Source
+
+ vda /var/lib/libvirt/images/xenial-snapvm.qcow2
+ hda /var/lib/libvirt/images/config.img
+
+ 9) Connect to your VM and start an I/O intensive job. In this case I'm
+ starting a 'dd' writing zeroes to a file until it gets to 10GBs:
+
+ ubuntu@ubuntu:~$ dd if=/dev/zero of=file.txt count=1024 bs=1024
+
+ 10) Back to the host, monitor the snapshot file and let it grow until at
+ list a bit more than 1GB, as in the example below (where we can see the
+ file with 3.9G):
+
+ root@xenial-apparmor:~# ls -lh /var/lib/libvirt/images/
+ total 5.2G
+ -rw-rw-r-- 1 libvirt-qemu kvm 329M Sep 3 03:18
bionic-server-cloudimg-amd64.img
+ -rw-r--r-- 1 root root 10G Sep 3 03:28 bionic-server-cloudimg-amd64.raw
+ -rw-rw-r-- 1 libvirt-qemu kvm 366K Sep 3 03:19 config.img
+ -rw--- 1 libvirt-qemu kvm 3.9G Sep 3 04:41
