Wido, Could you please let me know when you've done this so I could try it out. Would it be a part of the 4.3 branch or 4.4?
Thanks ----- Original Message ----- From: "Wido den Hollander" <w...@widodh.nl> To: dev@cloudstack.apache.org Sent: Sunday, 23 March, 2014 3:56:44 PM Subject: Re: ACS and KVM uses /tmp for volumes migration and templates On 03/21/2014 02:23 PM, Andrei Mikhailovsky wrote: > > Wido, > > > i would be happy to try the custom ACS build unless 4.3 comes out soon. It > has been overdue for sometime now )). Has this feature been addressed in the > 4.3 release? > No, it hasn't been fixed yet. I have to admit, I forgot about this until you sent this e-mail to the list. I'll fix this in master later this week. > > I can leave with this feature for the time being, but i do see a longer term > issue when my volumes become large as i've only got about 100gb free space on > my host servers. > > I fully agree. While writing this code I was aware of this. See my comments in the code: https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/storage/LibvirtStorageAdaptor.java;h=5de8bd26ae201187f5db5fd16b7e3ca157cab53a;hb=master#l1087 > From what i can tell by looking at the rbd ls -l info all of my volumes are > done in Format 2 > Correct, because I by-pass libvirt and Qemu at some places right now. > > Cheers, > > > Andrei > > > > > ----- Original Message ----- > > From: "Wido den Hollander" <w...@widodh.nl> > To: dev@cloudstack.apache.org > Sent: Thursday, 20 March, 2014 9:40:29 AM > Subject: Re: ACS and KVM uses /tmp for volumes migration and templates > > On 03/20/2014 12:59 AM, Andrei Mikhailovsky wrote: >> Hi guys, >> >> I was wondering if this is a bug? >> > > No, it's a "feature". > >> I've noticed that during volume migration from NFS to RBD primary storage >> the volume image is first copied to /tmp and only then to the RBD storage. >> This seems silly to me as one would expect a typical volume to be larger >> than the host's hard disk. Also, it is a common practice to use tmpfs as >> /tmp for performance reasons. Thus, a typical host server will have far >> smaller /tmp folder than the size of an average volume. As a result, volume >> migration would break after filling the /tmp and could probably cause a >> bunch of issue for the KVM host itself as well as any vms running on the >> server. >> > > Correct. The problem was that RBD images know two formats. Format 1 > (old/legacy) and format 2. > > In order to perform cloning images should be in RBD format 2. > > When running qemu-img convert with a RBD image as a destination qemu-img > will create a RBD image in format 1. > > That's due to this piece of code in block/rbd.c in Qemu: > > ret = rbd_create(io_ctx, name, bytes, &obj_order); > > rbd_create() creates images in format 1. To use format 2 you should use > rbd_create2() or rbd_create3(). > > With RBD format 1 we can't do snapshotting or cloning, which we require > in ACS. > > So I had to do a intermediate step where I first wrote the RAW image > somewhere and afterwards write it to RBD. > > After some discussion a config option has been added to Ceph: > > OPTION(rbd_default_format, OPT_INT, 1) > > This allows me to do this: > > qemu-img convert .. -O raw .. rbd:rbd/myimage:rbd_default_format=2 > > This causes librbd/RBD to create a format 2 image and we can skip the > convert step to /tmp. > > This option is available since Ceph Dumpling 0.67.5 and was not > available when ACS 4.2 was written. > > I'm going to make changes in master which skip the step with /tmp. > > Technically this can be backported to 4.2, but then you would have to > run your own homebrew version of 4.2 > >> It also seems that the /tmp is temporarily used during a template creation . >> > > Same story as above. > >> My setup: >> >> ACS 4.2.1 >> Ubuntu 12.04 with KVM >> RBD + NFS for Primary storage >> NFS for Staging and Secondary storage >> >> >> Thanks >> >> Andrei >> > > >
