I have done some more lab testing and here is what I found: 1) If one VM is provisioned via Juju and then after the provisioning is fully complete (i.e. a VM is started) other VMs are created successfully.
This means that there is no issue with using the same base image for snapshots of multiple VMs at the same time and that QEMU does not consider that a backing image somehow needs to be write-lock protected when multiple VMs are started. juju add-machine # i juju deploy cs:ubuntu -n 1 --to kvm:<i> # wait until 0/kvm/<i> starts juju add-unit ubuntu --to kvm:<i> # everything is ok 2) If multiple VMs are provisioned at the same time (without any prior VM provisioning) issues start to appear: # juju.container and juju.container.kvm tracing is enabled juju model-config logging-config <root>=WARNING;juju.container.kvm=TRACE;juju.container=TRACE;unit=DEBUG juju add-machine # <i> juju deploy cs:ubuntu -n 6 --to kvm:<i>,kvm:<i>,kvm:<i>,kvm:<i>,kvm:<i>,kvm:<i> # errors: https://paste.ubuntu.com/p/DrsVw6xhzf/ https://private-fileshare.canonical.com/~dima/juju-dumps/kvm-host-var- log-17-03-2019.tar.gz (/var/log from the kvm host) Intermediate Juju status shows that image downloads for multiple VMs on the same machine start concurrently: https://paste.ubuntu.com/p/CXKPCdDZnm/ (multiple consecutive juju status commands showing that there is no synchronization between cloud image downloads) ubuntu@maas:~$ juju status #... Machine State DNS Inst id Series AZ Message 3 started 10.232.24.16 node-2712d152-5daf-11e4-93f3-525400a6ab07 bionic default Deployed 3/kvm/0 pending pending bionic starting 3/kvm/1 pending pending bionic copying http://cloud-images.ubuntu.com/server/releases/bionic/release-20190307/ubuntu-18.04-server-cloudimg-amd64.img 1% (576KB/s) 3/kvm/2 pending pending bionic copying http://cloud-images.ubuntu.com/server/releases/bionic/release-20190307/ubuntu-18.04-server-cloudimg-amd64.img 47% (5.2MB/s) 3/kvm/3 pending pending bionic copying http://cloud-images.ubuntu.com/server/releases/bionic/release-20190307/ubuntu-18.04-server-cloudimg-amd64.img 34% (4.6MB/s) 3/kvm/4 pending pending bionic starting 3/kvm/5 pending pending bionic copying http://cloud-images.ubuntu.com/server/releases/bionic/release-20190307/ubuntu-18.04-server-cloudimg-amd64.img 37% (2.7MB/s) >From the machine log I can see that concurrent image downloads also result in >concurrent executions of `qemu-img convert` to the same (!!!) destination file: machine.log: https://paste.ubuntu.com/p/fkmDv69pR8/ 2019-03-17 13:48:05 DEBUG juju.container.kvm run.go:19 qemu-img [convert -f qcow2 /tmp/juju-kvm-ubuntu-18.04-server-cloudimg-amd64.img-720678751 /var/lib/juju/kvm/guests/bionic-amd64-backing-file.qcow] 2019-03-17 13:48:09 DEBUG juju.container.kvm run.go:19 qemu-img [convert -f qcow2 /tmp/juju-kvm-ubuntu-18.04-server-cloudimg-amd64.img-198086139 /var/lib/juju/kvm/guests/bionic-amd64-backing-file.qcow] 2019-03-17 13:48:10 DEBUG juju.container.kvm run.go:19 qemu-img [convert -f qcow2 /tmp/juju-kvm-ubuntu-18.04-server-cloudimg-amd64.img-727014373 /var/lib/juju/kvm/guests/bionic-amd64-backing-file.qcow] # ... 2019-03-17 13:48:12 DEBUG juju.container.kvm sync.go:241 qemu-image convert output: 2019-03-17 13:48:12 DEBUG juju.container.kvm sync.go:253 close /tmp/juju-kvm-ubuntu-18.04-server-cloudimg-amd64.img-720678751: file already closed 2019-03-17 13:48:12 DEBUG juju.container.kvm sync.go:257 got "remove /tmp/juju-kvm-ubuntu-18.04-server-cloudimg-amd64.img-720678751: no such file or directory" removing "/tmp/juju-kvm-ubuntu-18.04-server-cloudimg-amd64.img-720678751" 2019-03-17 13:48:12 DEBUG juju.container.kvm container.go:84 create the machine juju-487449-3-kvm-3 2019-03-17 13:48:12 DEBUG juju.container.kvm run_linux.go:24 running: genisoimage [-output /var/lib/juju/kvm/guests/juju-487449-3-kvm-3-ds.iso -volid cidata -joliet -rock user-data meta-data network-config] What I noticed as well is that VMs are started concurrently to image operations for other VMs. # start juju-487449-3-kvm-3 2019-03-17 13:48:14 DEBUG juju.container.kvm run.go:19 virsh [start juju-487449-3-kvm-3] # in parallel, another image is created for kvm/5: qemu-img create ... juju-487449-3-kvm-5.qcow 2019-03-17 13:48:16 DEBUG juju.container.kvm run_linux.go:24 running: qemu-img [create -b /var/lib/juju/kvm/guests/bionic-amd64-backing-file.qcow -f qcow2 /var/lib/juju/kvm/guests/juju-487449-3-kvm-5.qcow 8G] 2019-03-17 13:48:16 DEBUG juju.container.kvm run_linux.go:25 running as uid: 64055, gid: 116 2019-03-17 13:48:16 DEBUG juju.container.kvm run.go:21 output: error: Failed to start domain juju-487449-3-kvm-3 error: internal error: qemu unexpectedly closed the monitor: 2019-03-17T13:48:16.305889Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.80000001H:ECX.svm [bit 2] 2019-03-17T13:48:16.320495Z qemu-system-x86_64: -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1: Failed to get shared "write" lock Is another process using the image? 2019-03-17 13:48:16 ERROR juju.provisioner.kvm kvm-broker.go:152 failed to start container: kvm container creation failed: failed to start domain "juju-487449-3-kvm-3": exit status 1 # after that qemu-img create completes for kvm/5 2019-03-17 13:48:17 DEBUG juju.container.kvm run_linux.go:36 output: Formatting '/var/lib/juju/kvm/guests/juju-487449-3-kvm-5.qcow', fmt=qcow2 size=8589934592 backing_file=/var/lib/juju/kvm/guests/bionic-amd64-backing-file.qcow cluster_size=65536 lazy_refcounts=off refcount_bits=16 During provisioning retry attempts a different error occurs for the data store iso which is surfaced in juju status as "exit status 13" (https://paste.ubuntu.com/p/DrsVw6xhzf/): 2019-03-17 13:48:48 TRACE juju.container directory.go:24 create directory: /var/lib/juju/containers/juju-487449-3-kvm-3 2019-03-17 13:48:48 TRACE juju.container.kvm kvm.go:178 write cloud-init 2019-03-17 13:48:48 TRACE juju.container.kvm kvm.go:217 create the container, constraints: 2019-03-17 13:48:48 DEBUG juju.container.kvm container.go:57 synchronise images for amd64 bionic released 2019-03-17 13:48:48 DEBUG juju.container.kvm container.go:84 create the machine juju-487449-3-kvm-3 2019-03-17 13:48:48 DEBUG juju.container.kvm run_linux.go:24 running: genisoimage [-output /var/lib/juju/kvm/guests/juju-487449-3-kvm-3-ds.iso -volid cidata -joliet -rock user-data meta-data network-config] 2019-03-17 13:48:48 DEBUG juju.container.kvm run_linux.go:25 running as uid: 64055, gid: 116 2019-03-17 13:48:48 DEBUG juju.container.kvm run_linux.go:36 output: I: -input-charset not specified, using utf-8 (detected in locale settings) genisoimage: Permission denied. Unable to open disc image file '/var/lib/juju/kvm/guests/juju-487449-3-kvm-3-ds.iso'. 2019-03-17 13:48:48 ERROR juju.provisioner.kvm kvm-broker.go:152 failed to start container: kvm container creation failed: failed to write data source volume for "juju-487449-3-kvm-3": exit status 13 2019-03-17 13:48:48 WARNING juju.provisioner provisioner_task.go:1350 failed to start machine 3/kvm/3 (kvm container creation failed: failed to write data source volume for "juju-487449-3-kvm-3": exit status 13), retrying in 10s (8 more attempts) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1816170 Title: KVM instances fail at creation with: Failed to get shared "write" lock Is another process using the image? To manage notifications about this bug go to: https://bugs.launchpad.net/juju/+bug/1816170/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
