Le 11/10/2017 à 16:15, Marcin Kulisz a écrit : > On 2017-10-11 15:18:10, Thomas Lange wrote: >>>>>>> On Fri, 29 Sep 2017 07:22:15 +0100, Steve McIntyre <st...@einval.com> >>>>>>> said: >> >> > Building >> > -------- >> >> > any further. We will need to look into tools for making new VMs. >> I wonder what is meant by "making new VM". >> You do mean creating the disk image for the VM or starting the VM with >> a tool like virsh? > > If I recall correctly this is about creating ephemeral vms (possibly from > template) on demand to used them as build machines for cloud images.
I had a look at various possible tools which could make that possible, here is a short summary. If people have more details, please share, not flame. Background reason: you need root rights for most of the build tools, and the cduser on the build server is an unpriviledged user. So we want to use sandbox VMs for the builds. Might help your local builds too (see https://lists.debian.org/debian-cloud/2017/09/msg00026.html) Overview of tools for Building cloud images in sandbox VMs: 1) Vagrant: First tool I looked why was Vagrant. It is a basically a scriptable wrapper of different virtualization technologies, to build stuff in VMs. It also out of the box syncs the local dir the a /vagrant dir in the VM, so you can work locally and build in the VM. It would be the ideal but the three FLOSS virtualization providers Vagrant proposes has all limitattions. A) * Vagrant with libvirt provider, on top of qemu: this needs the libvirt qemu://system priviledge, which is basically root rights, presumably tocreate tap devices inside the libvirt network bridge and access all kind of blcok devices. Upstream is not interested in supporting non-priviledged vagrant-libvirt, see https://github.com/vagrant-libvirt/vagrant-libvirt/issues/272 B) * Vagrant with VirtualBox provider: works well but you need to bring a couple of extra kernel modules on the host, and Virtualbox itself is in contrib. From what I heard, F-Droid is using that Vagrant + VirtualBox in their build infrastructure. C) * Vagrant with LXC provider: does not provide enough ressource isolation since we need to to do loopback mounts and you can't do that without being priviledged in a way. Same problem for Docker. 2) Autopkgtest: I also had a look at how autopkgtest VMs build work. Autopkgtest start a pre configured VM, connect to the Virtual Serial port of the VM, sends inline python over that channel to mount a shared file 9psystem. Gory details in setup_baseimage() in autopkgtest-virt-qemu.py This inline python is a bit ugly to understand, but it looks like this might be abstracted enough to run any kind of builds. I wrote Martin Pitt, author of autopkg test if he can give his opinion. 3) Fai + libguestfs: There is this nice library called libguestfs, which allows to do anykind of file system and disk work via a programmable interface, all taking place in a qemu sandbox. Basically it abstracts the work of what autopkgtest does manually. Might be interesting to look if fai could use libguestfs. Instead of doing those loop mounts ourselves, we can programmatically tell qemu "hey here is a disk image, mount it and do stuff inside" and we don't need to be root anymore. Libguestfs has a very cooperative upstream, is well documented, and has bindings in many programming languages. See http://libguestfs.org/guestfs-perl.3.html for how to use this in perl. 4) virt-install: At some point if we need to start qemu to run a deboostrap wrapper inside, it might make sense to start the debian installer on a fresh qemu vm, and do the extra configuration either via late-install scripts or after the disk image is create via libguestfs. Works very well and quite easy to start with ( see an example https://anonscm.debian.org/cgit/cloud/debian-vm-templates.git/tree/virt-install-generic-qcow2/build.sh ) I am not sure however if this fits well with the amount of configurability we want, late-install scripts are hard to debug. However post VM creation customization can be easily made with libguestfs. 5) homegrown solution: since basically what we want is a scriptable VM where a share a build dir to be run, and control that over ssh, I had a look how complicated it would be to do that using the libvirt api. well actually this is not complicated at all, I had a 'vmake' proof of concept in a few hours. https://gist.github.com/EmmanuelKasper/2a99f3f67afc3ac1100affbf4630d9d1 instead of calling make -f Makefile ec2-stretch-image.raw you do vmake Makefile ec2-stretch-image.raw and you got your VM build in the build dir. My prefered solutions would be 1)B), then 3) as long as it does not imply even more shell and less high level language, I can help to implement those solutions.