** Summary changed:
- libvirt - disk not ready for pivot yet
+ libvirt: blockcommit fails - disk not ready for pivot yet
** Description changed:
- root@thewind:/home/bestpa/scripts# virsh blockcommit mail vda --active
--verbose --pivot
- Block commit: [100 %]error: failed to pivot job for disk vda
+ [Impact]
+
+ On xenial, if you manually invoke blockcommit through virsh in libvirt,
+ the command immediately fails with blockcommit supposedly being 100%,
+ and that the disk is not ready for pivot yet:
+
+ root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose
--pivot --wait
+ Block commit: [100 %]
+ error: failed to pivot job for disk vda
error: block copy still active: disk 'vda' not ready for pivot yet
- found related bugfix at redhat... can i get 1.3.2 pushed into ubuntu
- 16.04 release?
-
- bestpa@thewind:~$ cat /etc/os-release
- NAME="Ubuntu"
- VERSION="16.04.2 LTS (Xenial Xerus)"
- ID=ubuntu
- ID_LIKE=debian
- PRETTY_NAME="Ubuntu 16.04.2 LTS"
-
- bestpa@thewind:~$ libvirtd --version
- libvirtd (libvirt) 1.3.1
+ However, if you look at the status of the active blockjob, we see that
+ the blockcommit is still active in the background:
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [0 %]
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [2 %]
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [6 %]
+
+ This happens until it reaches 100%, where it gets stuck. To un-stick
+ things, you must then manually --abort the blockjob.
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [100 %]
+
+ This happens in VMs which are experiencing load, and is caused by a race
+ condition in libvirt. Users are not able to commit their snapshots to
+ disk.
+
+ [Test Case]
+
+ Credit goes to Fabio Martins, who determined how to reproduce this
+ issue.
+
+ On a Ubuntu 16.04 host with libvirt 1.3.1-1ubuntu10.27:
+
+ 1) Create a VG and define a LVM pool:
+
+ root@xenial-apparmor:~# cat lvmpool.xml
+ <pool type="logical">
+ <name>LVMpool_vg</name>
+ <source>
+ <device path="/dev/sdb"/>
+ </source>
+ <target>
+ <path>/dev/LVMpool_vg</path>
+ </target>
+ </pool>
+
+ # virsh pool-define lvmpool.xml
+ # virsh pool-start LVMpool_vg
+ # virsh pool-autostart LVMpool_vg
+
+ 2) Create a config file to use as a cdrom device with the new VM (will
+ be created in next steps), just to inject a password with cloud-init:
+
+ # cat > config <<EOF
+ > #cloud-config
+ > password: passw0rd
+ > chpasswd: { expire: False }
+ > ssh_pwauth: True
+ > EOF
+
+ # apt install cloud-image-utils
+
+ # cloud-localds config.img config
+
+ # mv config.img /var/lib/libvirt/images/
+ # chown libvirt-qemu:kvm /var/lib/libvirt/images/config.img
+ # chmod 664 /var/lib/libvirt/images/config.img
+
+ 3) Create one VM using this pool:
+
+ # virt-install --connect=qemu:///system --name snapvm --ram 2048
+ --vcpus=1 --os-type=linux --disk pool=LVMpool_vg,size=15,bus=virtio
+ --disk /var/lib/libvirt/images/config.img,device=cdrom --network network
+ =kvm-br0 --graphics none --import --noautoconsole
+
+ 4) Stop the VM
+
+ # virsh destroy snapvm
+
+ 5) Download a Ubuntu cloud image, convert to raw and restore it into the
+ LV used as a disk by our VM:
+
+ # wget
https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img
+ # qemu-img convert ./bionic-server-cloudimg-amd64.img
./bionic-server-cloudimg-amd64.raw
+ # dd if=./bionic-server-cloudimg-amd64.raw of=/dev/LVMpool_vg/snapvm bs=8M
conv=sparse
+
+ 6) Start the VM and connect to it in another window
+
+ # virsh start snapvm
+
+ 7) Check that the VM is using the LV as the disk:
+
+ root@xenial-apparmor:~# virsh domblklist snapvm
+ Target Source
+ ------------------------------------------------
+ vda /dev/LVMpool_vg/snapvm
+ hda /var/lib/libvirt/images/config.img
+
+ 8) Create a snapshot and check that the new domblklist points to the
+ snapshot file:
+
+ # virsh snapshot-create-as --domain snapvm --diskspec
+ vda,file=/var/lib/libvirt/images/xenial-snapvm.qcow2,snapshot=external
+ --disk-only --atomic
+
+ root@xenial-apparmor:~# virsh domblklist snapvm
+ Target Source
+ ------------------------------------------------
+ vda /var/lib/libvirt/images/xenial-snapvm.qcow2
+ hda /var/lib/libvirt/images/config.img
+
+ 9) Connect to your VM and start an I/O intensive job. In this case I'm
+ starting a 'dd' writing zeroes to a file until it gets to 10GBs:
+
+ ubuntu@ubuntu:~$ dd if=/dev/zero of=file.txt count=1024 bs=10240000
+
+ 10) Back to the host, monitor the snapshot file and let it grow until at
+ list a bit more than 1GB, as in the example below (where we can see the
+ file with 3.9G):
+
+ root@xenial-apparmor:~# ls -lh /var/lib/libvirt/images/
+ total 5.2G
+ -rw-rw-r-- 1 libvirt-qemu kvm 329M Sep 3 03:18
bionic-server-cloudimg-amd64.img
+ -rw-r--r-- 1 root root 10G Sep 3 03:28 bionic-server-cloudimg-amd64.raw
+ -rw-rw-r-- 1 libvirt-qemu kvm 366K Sep 3 03:19 config.img
+ -rw------- 1 libvirt-qemu kvm 3.9G Sep 3 04:41 xenial-snapvm.qcow2
+
+ 11) Start a blockcommit job with --active --verbose --pivot --wait and
+ we'll hit the error when the job gets to 100%:
+
+ root@xenial-apparmor:~# virsh blockcommit snapvm vda --active --verbose
--pivot --wait
+ Block commit: [100 %]
+ error: failed to pivot job for disk vda
+ error: block copy still active: disk 'vda' not ready for pivot yet
+
+ 12) The blkjob will continue in the background, and status increments:
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [0 %]
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [2 %]
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [6 %]
+
+ 13) The blkjob will show it is stuck at 100% until you --abort the
+ blkjob:
+
+ root@xenial-apparmor:~# virsh blockjob snapvm vda --info
+ Active Block Commit: [100 %]
+
+ I have created a test package with the commits needed to solve the
+ problem, and it is available here:
+
+ https://launchpad.net/~mruffell/+archive/ubuntu/sf242822-test
+
+ What should happen:
+
+ If you install the test libvirt-bin and libvirt0 packages from the above
+ ppa, and run through the test case, when blockcommit is invoked, it will
+ not fail immediately, and instead, will continue on until it reaches
+ 100%. Once 100% is reached, the blockjob will complete successfully.
+
+ [Regression Potential]
+
+ While there are four commits which are required to fix this issue, all
+ of them are fairly minor and only modify the way the current status
+ percentage is counted, and how states are being changed, upon reaching
+ 100% blockcommit. All changes are localised to one file.
+
+ Most of the commits are limited to blockcommit, and in event of
+ regression, only blockcommit and by extension, some blockjobs would be
+ impacted.
+
+ The commits have been present in upstream for a long time, have been
+ well tested by the community, and are from a release of libvirt with
+ very small delta to the one in xenial (1.3.2 versus 1.3.1 in xenial), I
+ believe there is little risk of regression.
+
+ [Other Info]
+
+ The following commits were identified in the upstream bug:
+ https://bugzilla.redhat.com/show_bug.cgi?id=1197592
+
+ which are also listed in comment #6.
+
+ commit 86c4df83b913dd73b79caeed2038291374384dc5
+ Author: Michael Chapman <[email protected]>
+ Date: Wed Jan 27 13:24:54 2016 +1100
+ Subject: virsh: improve waiting for block job readiness
+
+ commit 8fa216bbb40df33e7fce5d727aa3dc334480878a
+ Author: Michael Chapman <[email protected]>
+ Date: Wed Jan 27 13:24:53 2016 +1100
+ Subject: virsh: ensure SIGINT action is reset on all errors
+
+ commit 15dee2ef24f2f19f6dcd30d997b81c8a14582361
+ Author: Michael Chapman <[email protected]>
+ Date: Wed Jan 27 13:24:52 2016 +1100
+ Subject: virsh: be consistent with style of loop exit
+
+ commit 704dfd6b0fafe7eafca93a03793389239f8ab869
+ Author: Michael Chapman <[email protected]>
+ Date: Wed Jan 27 13:24:51 2016 +1100
+ Subject: virsh: avoid unnecessary progress updates
+
+ These fix the problem, and were introduced in libvirt 1.3.2 upstream.
+ All commits are clean cherry picks, and the code is still present in B,
+ D, E and F.
** Tags added: sts
** Changed in: libvirt (Ubuntu Xenial)
Status: Won't Fix => In Progress
** Changed in: libvirt (Ubuntu Xenial)
Importance: Undecided => Medium
** Changed in: libvirt (Ubuntu Xenial)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1681839
Title:
libvirt: blockcommit fails - disk not ready for pivot yet
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
