2014/1/22 olivier sallou <[email protected]> > > > > 2014/1/22 Anders Ingemann <[email protected]> > >> On 22 January 2014 11:41, olivier sallou <[email protected]>wrote: >> >>> >>> >>> >>> 2014/1/22 Anders Ingemann <[email protected]> >>> >>>> On 22 January 2014 10:43, olivier sallou <[email protected]>wrote: >>>> >>>>> >>>>> >>>>> >>>>> 2014/1/22 Anders Ingemann <[email protected]> >>>>> >>>>>> On 22 January 2014 08:23, olivier sallou <[email protected]>wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2014/1/22 Jimmy Kaplowitz <[email protected]> >>>>>>> >>>>>>>> Neat. Yeah, a GCE image is simply a raw bootable disk named >>>>>>>> disk.raw, compressed in a gzipped GNU-format tarball with a filename >>>>>>>> ending >>>>>>>> in .tar.gz. We also encourage creating disk.raw as a sparse file and >>>>>>>> using >>>>>>>> GNU tar's S flag to minimize image size and add time. The tarball is >>>>>>>> necessary but hopefully the code is general enough to handle that. >>>>>>>> Certain >>>>>>>> bits of our image snapshotting tool gcimagebundle expect everything >>>>>>>> aside >>>>>>>> from the bootloader to be in a single MSDOS partition number 1. >>>>>>>> >>>>>>> I think this is the case for virtualbox provider. >>>>>>> >>>>>>>> We make various tweaks, such as installing the various integration >>>>>>>> software I've mentioned before, pointing to our Debian mirror, and >>>>>>>> (with >>>>>>>> Tomasz's pull request into the google GitHub fork) the host machine's >>>>>>>> NTP >>>>>>>> server, etc. Nothing that changes the essence. >>>>>>>> >>>>>>> Some of those (setting up a mirror, installing packages) can be done: >>>>>>> 1) with plugins (but as it is a requirement for you this is not the >>>>>>> best choice) >>>>>>> 2) in your plugin task, use the library part that manage package >>>>>>> install etc.. if you look at cloud-init plugin task for example, you >>>>>>> will >>>>>>> see how to add a source (here debian backports) and packages to the >>>>>>> image >>>>>>> (here "sudo"). >>>>>>> >>>>>>> The rest of the plugin task can focus on copying some files in the >>>>>>> image, setting ntp etc... >>>>>>> >>>>>>>> The strategy you suggest sounds worth trying indeed. >>>>>>>> >>>>>>>> - Jimmy >>>>>>>> On Jan 21, 2014 10:32 PM, "olivier sallou" < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> 2014/1/21 Tomasz Rybak <[email protected]> >>>>>>>>> >>>>>>>>>> As Jimmy wrote in his email from 2014-01-14, I started >>>>>>>>>> looking at GCE-related parts of build-debian-cloud. >>>>>>>>>> >>>>>>>>>> I assume that the course for now is changing the scripts >>>>>>>>>> to work in Python, similar to what's being done with EC2 >>>>>>>>>> and VirtualBox parts. Should we branch from andsens/python >>>>>>>>>> and work on it, or do something else? Also, who'll create >>>>>>>>>> the main branch (GCE-python-WIP?), into which we would >>>>>>>>>> pull proposed changes? I think the best solution would be >>>>>>>>>> to create such branch in repository >>>>>>>>>> https://github.com/google/build-debian-cloud >>>>>>>>>> >>>>>>>>>> As for the work to do, I think we'll need to: >>>>>>>>>> 1. change gce file to proper manifest >>>>>>>>>> 2. move tasks from tasks/gce to providers/gce and >>>>>>>>>> rewrite them in Python >>>>>>>>>> 3. integrate cloud-init when appropriate >>>>>>>>>> >>>>>>>>> >>>>>>>>> If the base image requirement is a raw image file and GCE only >>>>>>>>> adds startup/management scripts for boot etc... you may only develop a >>>>>>>>> plugin and use VirtualBox provider which is in fact a quite generic >>>>>>>>> one >>>>>>>>> (not only virtualbox). >>>>>>>>> >>>>>>>>> I personnaly use VirtualBox provider for my KVM machines and use >>>>>>>>> the opennebula plugin for the OpenNebula contextualization (will be >>>>>>>>> modified soon to use cloud-init too). >>>>>>>>> >>>>>>>>> Then, for GCE, it would be, for the user, only a matter of user >>>>>>>>> VirtualBox provider (raw format) and activating the GCE plugin. >>>>>>>>> >>>>>>>>> Olivier >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> This is rough idea, and I have not touched >>>>>>>>>> packaging of >>>>>>>>>> https://github.com/GoogleCloudPlatform/compute-image-packages >>>>>>>>>> >>>>>>>>>> Have I missed something? I assume we need to have >>>>>>>>>> more detailed plan of moving to Python so anyone >>>>>>>>>> can see what is to be done and volunteer to some >>>>>>>>>> tasks ;-) For now I just want to start discussion >>>>>>>>>> to see what I forgot about. >>>>>>>>>> >>>>>>>>>> Best regards. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Tomasz Rybak GPG/PGP key ID: 2AD5 9860 >>>>>>>>>> Fingerprint A481 824E 7DD3 9C0E C40A 488E C654 FB33 2AD5 9860 >>>>>>>>>> http://member.acm.org/~tomaszrybak >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>>>> >>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>>> >>>>>>> Heh, it's great to see others are able to figure out my framework >>>>>> already, without documentation. I should really get on with writing it >>>>>> though... :-) >>>>>> >>>>>> > If the base image requirement is a raw image file and GCE only >>>>>> adds startup/management scripts for boot etc... you may only develop a >>>>>> plugin and use VirtualBox provider which is in fact a quite generic one >>>>>> (not only virtualbox). >>>>>> >>>>>> I would strongly suggest to refrain from doing that. If the >>>>>> VirtualBox provider is indeed very generic, things should be abstracted >>>>>> into task sets. I have not abstracted much of it until now, since I >>>>>> wasn't >>>>>> aware of the commonalities between providers (given that there are only >>>>>> to >>>>>> - or three now with kvm). If You add GCE as a separate provider, I can >>>>>> take >>>>>> a look at it and create some new tasksets that should make the task >>>>>> resolving a bit easier. >>>>>> >>>>> >>>>> So, do you think we should create a KVM provider ? (which would be 99% >>>>> equivalent to VirtualBox for the moment but would preserve from virtualbox >>>>> modifications) >>>>> >>>>> Olivier >>>>> >>>>>> >>>>>> The advantages of having a provider rather than a plugin are >>>>>> manifold. >>>>>> >>>>>> 1. Plugins tasks will be resolved *after* the provider tasks, >>>>>> meaning they will be able to remove some provider tasks if they do >>>>>> something more specific. >>>>>> 2. You can enforce plugin compatibility in the manifest schemas >>>>>> by looking at the provider string, having a plugin look at what other >>>>>> plugins are loaded is just messy. >>>>>> >>>>>> Disadvantages of creating a provider as a plugin: >>>>>> >>>>>> 1. Any changes you make because the virtualbox provider doesn't >>>>>> quite fit will become hard to understand - one provider adding a task >>>>>> and >>>>>> the plugin removing that same task... you might end up with a >>>>>> tasklist that >>>>>> is completely different from virtualbox (once you are done). >>>>>> 2. Changes to the virtualbox provider will now have to be >>>>>> carefully made because other providers suddenly rely on it >>>>>> 3. You will have a hard time adding special things to the >>>>>> manifest since the vbox provider applies its own manifest schema >>>>>> >>>>>> > In your plugin task, use the library part that manage package >>>>>> install etc.. if you look at cloud-init plugin task for example, you will >>>>>> see how to add a source (here debian backports) and packages to the image >>>>>> (here "sudo"). >>>>>> What he said! The package API can save you a lot of trouble. Btw, you >>>>>> can add trusted keyrings to apt through the manifest as well. >>>>>> >>>>>> Anders >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>> >>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>> >>>>> >>>> > So, do you think we should create a KVM provider? >>>> I do. Surely things like guest additions/guest tools are different. It >>>> would also allow use to make a proper vagrant box for it. >>>> The big challenge is making some proper taskset abstractions. It should >>>> be possible to create a minimal resolve_tasks() function if we code the >>>> tasksets properly (seeing how vbox and kvm are very similar), this would be >>>> very useful for future providers as well (e.g. gce or hp-cloud). >>>> >>> For the current code, the only difference between VB and KVM is KVM have >>> no guest-additions and expect raw format instead of vdi. >>> >> Do you want me to create the kvm provider and make a pull request, or do you plan to do it?
Olivier > >>> Olivier >>> >>> >>> >>> -- >>> >>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>> >>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>> >>> >> Alright, but something like the virtio >> drivers<http://www.linux-kvm.org/page/Virtio> would >> be useful, right? >> > > I first planned to create a plugin for this above vb provider, but if we > have a kvm provider it would make sense in this case to create a specific > optional task in kvm provider. > > > > -- > > gpg key id: 4096R/326D8438 (keyring.debian.org) > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > > -- gpg key id: 4096R/326D8438 (keyring.debian.org) Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
