On 22 January 2014 16:02, olivier sallou <[email protected]> wrote:
> > > > 2014/1/22 olivier sallou <[email protected]> > >> >> >> >> 2014/1/22 Anders Ingemann <[email protected]> >> >>> On 22 January 2014 11:41, olivier sallou <[email protected]>wrote: >>> >>>> >>>> >>>> >>>> 2014/1/22 Anders Ingemann <[email protected]> >>>> >>>>> On 22 January 2014 10:43, olivier sallou <[email protected]>wrote: >>>>> >>>>>> >>>>>> >>>>>> >>>>>> 2014/1/22 Anders Ingemann <[email protected]> >>>>>> >>>>>>> On 22 January 2014 08:23, olivier sallou >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> 2014/1/22 Jimmy Kaplowitz <[email protected]> >>>>>>>> >>>>>>>>> Neat. Yeah, a GCE image is simply a raw bootable disk named >>>>>>>>> disk.raw, compressed in a gzipped GNU-format tarball with a filename >>>>>>>>> ending >>>>>>>>> in .tar.gz. We also encourage creating disk.raw as a sparse file and >>>>>>>>> using >>>>>>>>> GNU tar's S flag to minimize image size and add time. The tarball is >>>>>>>>> necessary but hopefully the code is general enough to handle that. >>>>>>>>> Certain >>>>>>>>> bits of our image snapshotting tool gcimagebundle expect everything >>>>>>>>> aside >>>>>>>>> from the bootloader to be in a single MSDOS partition number 1. >>>>>>>>> >>>>>>>> I think this is the case for virtualbox provider. >>>>>>>> >>>>>>>>> We make various tweaks, such as installing the various integration >>>>>>>>> software I've mentioned before, pointing to our Debian mirror, and >>>>>>>>> (with >>>>>>>>> Tomasz's pull request into the google GitHub fork) the host machine's >>>>>>>>> NTP >>>>>>>>> server, etc. Nothing that changes the essence. >>>>>>>>> >>>>>>>> Some of those (setting up a mirror, installing packages) can be >>>>>>>> done: >>>>>>>> 1) with plugins (but as it is a requirement for you this is not the >>>>>>>> best choice) >>>>>>>> 2) in your plugin task, use the library part that manage package >>>>>>>> install etc.. if you look at cloud-init plugin task for example, you >>>>>>>> will >>>>>>>> see how to add a source (here debian backports) and packages to the >>>>>>>> image >>>>>>>> (here "sudo"). >>>>>>>> >>>>>>>> The rest of the plugin task can focus on copying some files in the >>>>>>>> image, setting ntp etc... >>>>>>>> >>>>>>>>> The strategy you suggest sounds worth trying indeed. >>>>>>>>> >>>>>>>>> - Jimmy >>>>>>>>> On Jan 21, 2014 10:32 PM, "olivier sallou" < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2014/1/21 Tomasz Rybak <[email protected]> >>>>>>>>>> >>>>>>>>>>> As Jimmy wrote in his email from 2014-01-14, I started >>>>>>>>>>> looking at GCE-related parts of build-debian-cloud. >>>>>>>>>>> >>>>>>>>>>> I assume that the course for now is changing the scripts >>>>>>>>>>> to work in Python, similar to what's being done with EC2 >>>>>>>>>>> and VirtualBox parts. Should we branch from andsens/python >>>>>>>>>>> and work on it, or do something else? Also, who'll create >>>>>>>>>>> the main branch (GCE-python-WIP?), into which we would >>>>>>>>>>> pull proposed changes? I think the best solution would be >>>>>>>>>>> to create such branch in repository >>>>>>>>>>> https://github.com/google/build-debian-cloud >>>>>>>>>>> >>>>>>>>>>> As for the work to do, I think we'll need to: >>>>>>>>>>> 1. change gce file to proper manifest >>>>>>>>>>> 2. move tasks from tasks/gce to providers/gce and >>>>>>>>>>> rewrite them in Python >>>>>>>>>>> 3. integrate cloud-init when appropriate >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> If the base image requirement is a raw image file and GCE only >>>>>>>>>> adds startup/management scripts for boot etc... you may only develop >>>>>>>>>> a >>>>>>>>>> plugin and use VirtualBox provider which is in fact a quite generic >>>>>>>>>> one >>>>>>>>>> (not only virtualbox). >>>>>>>>>> >>>>>>>>>> I personnaly use VirtualBox provider for my KVM machines and use >>>>>>>>>> the opennebula plugin for the OpenNebula contextualization (will be >>>>>>>>>> modified soon to use cloud-init too). >>>>>>>>>> >>>>>>>>>> Then, for GCE, it would be, for the user, only a matter of user >>>>>>>>>> VirtualBox provider (raw format) and activating the GCE plugin. >>>>>>>>>> >>>>>>>>>> Olivier >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> This is rough idea, and I have not touched >>>>>>>>>>> packaging of >>>>>>>>>>> https://github.com/GoogleCloudPlatform/compute-image-packages >>>>>>>>>>> >>>>>>>>>>> Have I missed something? I assume we need to have >>>>>>>>>>> more detailed plan of moving to Python so anyone >>>>>>>>>>> can see what is to be done and volunteer to some >>>>>>>>>>> tasks ;-) For now I just want to start discussion >>>>>>>>>>> to see what I forgot about. >>>>>>>>>>> >>>>>>>>>>> Best regards. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Tomasz Rybak GPG/PGP key ID: 2AD5 9860 >>>>>>>>>>> Fingerprint A481 824E 7DD3 9C0E C40A 488E C654 FB33 2AD5 9860 >>>>>>>>>>> http://member.acm.org/~tomaszrybak >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>>>>> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>>>> >>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>>>> >>>>>>>> Heh, it's great to see others are able to figure out my framework >>>>>>> already, without documentation. I should really get on with writing it >>>>>>> though... :-) >>>>>>> >>>>>>> > If the base image requirement is a raw image file and GCE only >>>>>>> adds startup/management scripts for boot etc... you may only develop a >>>>>>> plugin and use VirtualBox provider which is in fact a quite generic one >>>>>>> (not only virtualbox). >>>>>>> >>>>>>> I would strongly suggest to refrain from doing that. If the >>>>>>> VirtualBox provider is indeed very generic, things should be abstracted >>>>>>> into task sets. I have not abstracted much of it until now, since I >>>>>>> wasn't >>>>>>> aware of the commonalities between providers (given that there are only >>>>>>> to >>>>>>> - or three now with kvm). If You add GCE as a separate provider, I can >>>>>>> take >>>>>>> a look at it and create some new tasksets that should make the task >>>>>>> resolving a bit easier. >>>>>>> >>>>>> >>>>>> So, do you think we should create a KVM provider ? (which would be >>>>>> 99% equivalent to VirtualBox for the moment but would preserve from >>>>>> virtualbox modifications) >>>>>> >>>>>> Olivier >>>>>> >>>>>>> >>>>>>> The advantages of having a provider rather than a plugin are >>>>>>> manifold. >>>>>>> >>>>>>> 1. Plugins tasks will be resolved *after* the provider tasks, >>>>>>> meaning they will be able to remove some provider tasks if they do >>>>>>> something more specific. >>>>>>> 2. You can enforce plugin compatibility in the manifest schemas >>>>>>> by looking at the provider string, having a plugin look at what other >>>>>>> plugins are loaded is just messy. >>>>>>> >>>>>>> Disadvantages of creating a provider as a plugin: >>>>>>> >>>>>>> 1. Any changes you make because the virtualbox provider doesn't >>>>>>> quite fit will become hard to understand - one provider adding a >>>>>>> task and >>>>>>> the plugin removing that same task... you might end up with a >>>>>>> tasklist that >>>>>>> is completely different from virtualbox (once you are done). >>>>>>> 2. Changes to the virtualbox provider will now have to be >>>>>>> carefully made because other providers suddenly rely on it >>>>>>> 3. You will have a hard time adding special things to the >>>>>>> manifest since the vbox provider applies its own manifest schema >>>>>>> >>>>>>> > In your plugin task, use the library part that manage package >>>>>>> install etc.. if you look at cloud-init plugin task for example, you >>>>>>> will >>>>>>> see how to add a source (here debian backports) and packages to the >>>>>>> image >>>>>>> (here "sudo"). >>>>>>> What he said! The package API can save you a lot of trouble. Btw, >>>>>>> you can add trusted keyrings to apt through the manifest as well. >>>>>>> >>>>>>> Anders >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>>>> >>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>>>> >>>>>> >>>>> > So, do you think we should create a KVM provider? >>>>> I do. Surely things like guest additions/guest tools are different. It >>>>> would also allow use to make a proper vagrant box for it. >>>>> The big challenge is making some proper taskset abstractions. It >>>>> should be possible to create a minimal resolve_tasks() function if we code >>>>> the tasksets properly (seeing how vbox and kvm are very similar), this >>>>> would be very useful for future providers as well (e.g. gce or hp-cloud). >>>>> >>>> For the current code, the only difference between VB and KVM is KVM >>>> have no guest-additions and expect raw format instead of vdi. >>>> >>> Do you want me to create the kvm provider and make a pull request, or do > you plan to do it? > > Olivier > >> >>>> Olivier >>>> >>>> >>>> >>>> -- >>>> >>>> gpg key id: 4096R/326D8438 (keyring.debian.org) >>>> >>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >>>> >>>> >>> Alright, but something like the virtio >>> drivers<http://www.linux-kvm.org/page/Virtio> would >>> be useful, right? >>> >> >> I first planned to create a plugin for this above vb provider, but if we >> have a kvm provider it would make sense in this case to create a specific >> optional task in kvm provider. >> >> >> >> -- >> >> gpg key id: 4096R/326D8438 (keyring.debian.org) >> >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 >> >> > > > -- > > gpg key id: 4096R/326D8438 (keyring.debian.org) > > Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438 > > ATM I am not able to test in kvm so you can go ahead and create the provider.
