Re: virt-builder metadata for Debian cloud images
On Mon, Dec 14, 2015 at 06:30:41PM +0100, Pino Toscano wrote: >On Wednesday 07 October 2015 14:59:15 Steve McIntyre wrote: >> On Fri, Oct 02, 2015 at 09:21:24AM +0200, Pino Toscano wrote: >> >Hi, >> > >> >Friendly ping... anything I can do to move this process forward? >> >> Hi Pino, >> >> Apologies - buried under other stuff at the moment, will come back to >> this shortly. > >Ping^2... Crap, sorry - I've been rubbish about getting back to this. I've accepted and tested your patch, and it looks good in a test build - see the output in http://cdimage.debian.org/cdimage/.live/new/openstack-jessie/ What should we be doing next? -- Steve McIntyre, Cambridge, UK.st...@einval.com "I suspect most samba developers are already technically insane... Of course, since many of them are Australians, you can't tell." -- Linus Torvalds
Re: virt-builder metadata for Debian cloud images
On Wednesday 07 October 2015 14:59:15 Steve McIntyre wrote: > On Fri, Oct 02, 2015 at 09:21:24AM +0200, Pino Toscano wrote: > >Hi, > > > >Friendly ping... anything I can do to move this process forward? > > Hi Pino, > > Apologies - buried under other stuff at the moment, will come back to > this shortly. Ping^2... -- Pino Toscano signature.asc Description: This is a digitally signed message part.
Re: virt-builder metadata for Debian cloud images
On Fri, Oct 02, 2015 at 09:21:24AM +0200, Pino Toscano wrote: >Hi, > >Friendly ping... anything I can do to move this process forward? Hi Pino, Apologies - buried under other stuff at the moment, will come back to this shortly. -- Steve McIntyre, Cambridge, UK.st...@einval.com "Because heaters aren't purple!" -- Catherine Pitt
Re: virt-builder metadata for Debian cloud images
Hi, On Thursday 30 July 2015 13:37:15 Pino Toscano wrote: > On Wednesday 22 July 2015 23:35:02 Steve McIntyre wrote: > > On Wed, Jul 22, 2015 at 11:24:09AM +0200, Pino Toscano wrote: > > >On Wednesday 22 July 2015 00:13:43 Steve McIntyre wrote: > > >> > > >> The build is done insida a VM hosted on our CD / live image production > > >> machine (pettersson.debian.org) - that's where the openstack images > > >> are made. It's not easy to give out access to that VM, but it's easy > > >> enough to add extra steps to the image production code that runs > > >> there. There's a slight snag, though - the signing process is outside > > >> of the VM so we'd probably have to generate the index file then > > >> post-process it to add the signature. > > > > > >Sounds OK for me -- we can create the snippet of index for each image > > >generated during that process, and later when publishing the image > > >aggregate all the index snippets into a single index file. > > >Could you please point me to the build scripts, so I can start taking > > >a look at them? > > > > Sure. My wrapper code is in > > > > http://anonscm.debian.org/cgit/debian-cd/pettersson-live.git/ > > > > There's not much magic there - look at > > available/run-30openstack-build, which calls > > build-openstack-debian-image for the heavy lifting. > > Thanks -- attached there is a first patch to produce a snippet of index > for each OpenStack image generated. > > > >Regarding signing: let's start generating the index for the images as > > >first step, so we have the process running, and later get the signing > > >done. > > After the above patch, there would need to assemble the resulting index > snippets, but I don't see the scripts doing the actual publishing of > the images. Could you please point at me? > > > >> >Also, a different chapter would be having proper non-cloud qcow2 > > >> >images available (always with virt-builder metadata, of course). > > >> >This, other than allowing us to avoid maintaining Debian images, would > > >> >mean virt-builder users can get official images of stable released, > > >> >updated periodically. > > >> >Would that be something useful? If so, how/where/etc could this process > > >> >get started? > > >> > > >> Sure, we can do that too - we have scope for producing all sorts of > > >> images. It's something that there's going to be discussion about at > > >> DebConf next month, in fact! > > > > > >Cool! > > >If it could be helpful, we use d-i based scripts for generate the > > >templates we host on libguestfs.org: you can see at [1] debian.sh and > > >debian.preseed for the actual image building, and compress.sh for the > > >cleaning up and compression. They should be runnable as normal user, > > >so there could be even no need for additional VM for building them. > > > > > >[1] https://github.com/libguestfs/libguestfs/tree/master/builder/website > > > > Cool, that looks useful. We're using the VM explicitly to allow for > > some root access, so it would be nice to play with alternatives. > > Yes, I can understand the concern being that, and this is what > libguestfs can help with: all the image operations are done inside a > small virtual machine (called "appliance") so users can do any sort of > image manipulation with no extra permissions needed, and without any > manual VM handling. Friendly ping... anything I can do to move this process forward? Thanks, -- Pino Toscano signature.asc Description: This is a digitally signed message part.
Re: virt-builder metadata for Debian cloud images
Hi,
(apologies if you get this three times)
On Wednesday 22 July 2015 23:35:02 Steve McIntyre wrote:
On Wed, Jul 22, 2015 at 11:24:09AM +0200, Pino Toscano wrote:
On Wednesday 22 July 2015 00:13:43 Steve McIntyre wrote:
The build is done insida a VM hosted on our CD / live image production
machine (pettersson.debian.org) - that's where the openstack images
are made. It's not easy to give out access to that VM, but it's easy
enough to add extra steps to the image production code that runs
there. There's a slight snag, though - the signing process is outside
of the VM so we'd probably have to generate the index file then
post-process it to add the signature.
Sounds OK for me -- we can create the snippet of index for each image
generated during that process, and later when publishing the image
aggregate all the index snippets into a single index file.
Could you please point me to the build scripts, so I can start taking
a look at them?
Sure. My wrapper code is in
http://anonscm.debian.org/cgit/debian-cd/pettersson-live.git/
There's not much magic there - look at
available/run-30openstack-build, which calls
build-openstack-debian-image for the heavy lifting.
Thanks -- attached there is a first patch to produce a snippet of index
for each OpenStack image generated.
Regarding signing: let's start generating the index for the images as
first step, so we have the process running, and later get the signing
done.
After the above patch, there would need to assemble the resulting index
snippets, but I don't see the scripts doing the actual publishing of
the images. Could you please point at me?
Also, a different chapter would be having proper non-cloud qcow2
images available (always with virt-builder metadata, of course).
This, other than allowing us to avoid maintaining Debian images, would
mean virt-builder users can get official images of stable released,
updated periodically.
Would that be something useful? If so, how/where/etc could this process
get started?
Sure, we can do that too - we have scope for producing all sorts of
images. It's something that there's going to be discussion about at
DebConf next month, in fact!
Cool!
If it could be helpful, we use d-i based scripts for generate the
templates we host on libguestfs.org: you can see at [1] debian.sh and
debian.preseed for the actual image building, and compress.sh for the
cleaning up and compression. They should be runnable as normal user,
so there could be even no need for additional VM for building them.
[1] https://github.com/libguestfs/libguestfs/tree/master/builder/website
Cool, that looks useful. We're using the VM explicitly to allow for
some root access, so it would be nice to play with alternatives.
Yes, I can understand the concern being that, and this is what
libguestfs can help with: all the image operations are done inside a
small virtual machine (called appliance) so users can do any sort of
image manipulation with no extra permissions needed, and without any
manual VM handling.
Thanks,
--
Pino ToscanoFrom 1048c8bd6d436393ba5bb87cc4cb26b78e6b0117 Mon Sep 17 00:00:00 2001
From: Pino Toscano ptosc...@redhat.com
Date: Thu, 30 Jul 2015 11:59:30 +0200
Subject: [PATCH] OpenStack images: generate snippet of virt-builder index
When generating OpenStack images, generate a snippet of virt-builder
metadata/index for each qcow2 image.
These will need to be assembled and signed later.
---
available/run-30openstack-build | 35 +++
1 file changed, 35 insertions(+)
diff --git a/available/run-30openstack-build b/available/run-30openstack-build
index f02bdda..004ce4b 100755
--- a/available/run-30openstack-build
+++ b/available/run-30openstack-build
@@ -74,6 +74,41 @@ if [ $ERROR -eq 0 ] ; then
rename -v $RENAME * ${LOG}/${BUILDNAME}.log
fi
+# snippet of index for virt-builder
+qcow=debian-${VERSION}-openstack-amd64.qcow2
+checksum=$(sha512sum ${qcow} | awk '{print $1;}')
+size=$(stat -c '%s' ${qcow})
+revision=$(date +%Y%m%d)
+case ${CODENAME} in
+testing|unstable) pretty_codename=${CODENAME} ;;
+*) pretty_codename=$(echo ${CODENAME} | sed 's/./\U/') ;;
+esac
+case ${VERSION} in
+testing|unstable)
+version_major_minor=${VERSION}
+osinfo=debian8 # XXX latest stable available
+;;
+*)
+version_major_minor=$(echo ${VERSION} | awk -F. '{print $1.$2;}')
+osinfo=debian$(echo ${VERSION} | awk -F. '{print $1;}')
+;;
+esac
+cat EOF ${qcow}.index
+[debian-${version_major_minor}-cloud]
+name=Debian ${VERSION} (${pretty_codename}) Cloud
+osinfo=${osinfo}
+arch=x86_64
+file=${qcow}
+checksum[sha512]=${checksum}
+format=qcow2
+size=${size}
+revision=${revision}
+notes=Debian ${VERSION} (${pretty_codename}).
+
+ This is a Debian installation, suited for running as OpenStack guest.
+
+EOF
+
cd
Re: virt-builder metadata for Debian cloud images
On Wednesday 22 July 2015 00:13:43 Steve McIntyre wrote: On Tue, Jul 21, 2015 at 07:21:57PM +0200, Pino Toscano wrote: Hi, Hi Pino, I'm one of the upstream of the libguestfs [1] project, which is a library and a set of tools to manipulate disk images. It is also packaged in Debian. One of the tools is virt-builder [2], which allows to create quickly new guests starting from templates. So far, we are providing all the templates ourselves on our website, and some of them are also Debian stable images: $ virt-builder --list centos-6 x86_64 CentOS 6.6 centos-7.0 x86_64 CentOS 7.0 centos-7.1 x86_64 CentOS 7.1 cirros-0.3.1 x86_64 CirrOS 0.3.1 debian-6 x86_64 Debian 6 (Squeeze) debian-7 x86_64 Debian 7 (Wheezy) debian-8 x86_64 Debian 8 (Jessie) fedora-18x86_64 Fedora® 18 fedora-19x86_64 Fedora® 19 fedora-20x86_64 Fedora® 20 fedora-21x86_64 Fedora® 21 Server fedora-21aarch64Fedora® 21 Server (aarch64) fedora-21armv7l Fedora® 21 Server (armv7l) fedora-21ppc64 Fedora® 21 Server (ppc64) fedora-21ppc64leFedora® 21 Server (ppc64le) fedora-22x86_64 Fedora® 22 Server fedora-22aarch64Fedora® 22 Server (aarch64) fedora-22armv7l Fedora® 22 Server (armv7l) scientificlinux-6x86_64 Scientific Linux 6.5 ubuntu-10.04 x86_64 Ubuntu 10.04 (Lucid) ubuntu-12.04 x86_64 Ubuntu 12.04 (Precise) ubuntu-14.04 x86_64 Ubuntu 14.04 (Trusty) Speaking with Thomas Goirand (zigo), I was told about the OpenStack images [3], so it would be nice to have them available in virt-builder for the users of this tool. OK, cool. :-) Basically, it is matter to provide an index file, INI-like style, with the proper metadata for each image (name, description, size, sha512, etc), optionally GPG-signed inline (it is not mandatory, although would be recommeded). You can check the configuration bits in the SOURCES OF TEMPLATES section of [2], and also the current index on libguestfs.org [4] as example. To use this index, there would be the need to install locally (i.e. where virt-builder is installed) a very small .conf file (and the copy of the GPG key, if the index is signed); the files for the libguestfs.org are currently part of the libguestfs-tools package (together with virt-builder), installed under /etc/xdg/virt-builder/repos.d/. I would be happy to help in getting the online metadata (i.e. the index, possibly signed) generated, so how would I get to it? The build is done insida a VM hosted on our CD / live image production machine (pettersson.debian.org) - that's where the openstack images are made. It's not easy to give out access to that VM, but it's easy enough to add extra steps to the image production code that runs there. There's a slight snag, though - the signing process is outside of the VM so we'd probably have to generate the index file then post-process it to add the signature. Sounds OK for me -- we can create the snippet of index for each image generated during that process, and later when publishing the image aggregate all the index snippets into a single index file. Could you please point me to the build scripts, so I can start taking a look at them? Regarding signing: let's start generating the index for the images as first step, so we have the process running, and later get the signing done. Also, a different chapter would be having proper non-cloud qcow2 images available (always with virt-builder metadata, of course). This, other than allowing us to avoid maintaining Debian images, would mean virt-builder users can get official images of stable released, updated periodically. Would that be something useful? If so, how/where/etc could this process get started? Sure, we can do that too - we have scope for producing all sorts of images. It's something that there's going to be discussion about at DebConf next month, in fact! Cool! If it could be helpful, we use d-i based scripts for generate the templates we host on libguestfs.org: you can see at [1] debian.sh and debian.preseed for the actual image building, and compress.sh for the cleaning up and compression. They should be runnable as normal user, so there could be even no need for additional VM for building them. [1] https://github.com/libguestfs/libguestfs/tree/master/builder/website Thanks, -- Pino Toscano signature.asc Description: This is a digitally signed message part.
Re: virt-builder metadata for Debian cloud images
On Wed, Jul 22, 2015 at 11:24:09AM +0200, Pino Toscano wrote: On Wednesday 22 July 2015 00:13:43 Steve McIntyre wrote: The build is done insida a VM hosted on our CD / live image production machine (pettersson.debian.org) - that's where the openstack images are made. It's not easy to give out access to that VM, but it's easy enough to add extra steps to the image production code that runs there. There's a slight snag, though - the signing process is outside of the VM so we'd probably have to generate the index file then post-process it to add the signature. Sounds OK for me -- we can create the snippet of index for each image generated during that process, and later when publishing the image aggregate all the index snippets into a single index file. Could you please point me to the build scripts, so I can start taking a look at them? Sure. My wrapper code is in http://anonscm.debian.org/cgit/debian-cd/pettersson-live.git/ There's not much magic there - look at available/run-30openstack-build, which calls build-openstack-debian-image for the heavy lifting. Regarding signing: let's start generating the index for the images as first step, so we have the process running, and later get the signing done. Sure. Also, a different chapter would be having proper non-cloud qcow2 images available (always with virt-builder metadata, of course). This, other than allowing us to avoid maintaining Debian images, would mean virt-builder users can get official images of stable released, updated periodically. Would that be something useful? If so, how/where/etc could this process get started? Sure, we can do that too - we have scope for producing all sorts of images. It's something that there's going to be discussion about at DebConf next month, in fact! Cool! If it could be helpful, we use d-i based scripts for generate the templates we host on libguestfs.org: you can see at [1] debian.sh and debian.preseed for the actual image building, and compress.sh for the cleaning up and compression. They should be runnable as normal user, so there could be even no need for additional VM for building them. [1] https://github.com/libguestfs/libguestfs/tree/master/builder/website Cool, that looks useful. We're using the VM explicitly to allow for some root access, so it would be nice to play with alternatives. -- Steve McIntyre, Cambridge, UK.st...@einval.com Getting a SCSI chain working is perfectly simple if you remember that there must be exactly three terminations: one on one end of the cable, one on the far end, and the goat, terminated over the SCSI chain with a silver-handled knife whilst burning *black* candles. --- Anthony DeBoer -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/2015073502.gx4...@einval.com
virt-builder metadata for Debian cloud images
Hi, (please keep me CC'ed, as I'm not subscribed) I'm one of the upstream of the libguestfs [1] project, which is a library and a set of tools to manipulate disk images. It is also packaged in Debian. One of the tools is virt-builder [2], which allows to create quickly new guests starting from templates. So far, we are providing all the templates ourselves on our website, and some of them are also Debian stable images: $ virt-builder --list centos-6 x86_64 CentOS 6.6 centos-7.0 x86_64 CentOS 7.0 centos-7.1 x86_64 CentOS 7.1 cirros-0.3.1 x86_64 CirrOS 0.3.1 debian-6 x86_64 Debian 6 (Squeeze) debian-7 x86_64 Debian 7 (Wheezy) debian-8 x86_64 Debian 8 (Jessie) fedora-18x86_64 Fedora® 18 fedora-19x86_64 Fedora® 19 fedora-20x86_64 Fedora® 20 fedora-21x86_64 Fedora® 21 Server fedora-21aarch64Fedora® 21 Server (aarch64) fedora-21armv7l Fedora® 21 Server (armv7l) fedora-21ppc64 Fedora® 21 Server (ppc64) fedora-21ppc64leFedora® 21 Server (ppc64le) fedora-22x86_64 Fedora® 22 Server fedora-22aarch64Fedora® 22 Server (aarch64) fedora-22armv7l Fedora® 22 Server (armv7l) scientificlinux-6x86_64 Scientific Linux 6.5 ubuntu-10.04 x86_64 Ubuntu 10.04 (Lucid) ubuntu-12.04 x86_64 Ubuntu 12.04 (Precise) ubuntu-14.04 x86_64 Ubuntu 14.04 (Trusty) Speaking with Thomas Goirand (zigo), I was told about the OpenStack images [3], so it would be nice to have them available in virt-builder for the users of this tool. Basically, it is matter to provide an index file, INI-like style, with the proper metadata for each image (name, description, size, sha512, etc), optionally GPG-signed inline (it is not mandatory, although would be recommeded). You can check the configuration bits in the SOURCES OF TEMPLATES section of [2], and also the current index on libguestfs.org [4] as example. To use this index, there would be the need to install locally (i.e. where virt-builder is installed) a very small .conf file (and the copy of the GPG key, if the index is signed); the files for the libguestfs.org are currently part of the libguestfs-tools package (together with virt-builder), installed under /etc/xdg/virt-builder/repos.d/. I would be happy to help in getting the online metadata (i.e. the index, possibly signed) generated, so how would I get to it? Also, a different chapter would be having proper non-cloud qcow2 images available (always with virt-builder metadata, of course). This, other than allowing us to avoid maintaining Debian images, would mean virt-builder users can get official images of stable released, updated periodically. Would that be something useful? If so, how/where/etc could this process get started? [1] http://libguestfs.org/ [2] http://libguestfs.org/virt-builder.1.html [3] http://cdimage.debian.org/cdimage/openstack/ [4] http://libguestfs.org/download/builder/index.asc Thanks, -- Pino Toscano signature.asc Description: This is a digitally signed message part.
Re: virt-builder metadata for Debian cloud images
On Tue, Jul 21, 2015 at 07:21:57PM +0200, Pino Toscano wrote: Hi, Hi Pino, I'm one of the upstream of the libguestfs [1] project, which is a library and a set of tools to manipulate disk images. It is also packaged in Debian. One of the tools is virt-builder [2], which allows to create quickly new guests starting from templates. So far, we are providing all the templates ourselves on our website, and some of them are also Debian stable images: $ virt-builder --list centos-6 x86_64 CentOS 6.6 centos-7.0 x86_64 CentOS 7.0 centos-7.1 x86_64 CentOS 7.1 cirros-0.3.1 x86_64 CirrOS 0.3.1 debian-6 x86_64 Debian 6 (Squeeze) debian-7 x86_64 Debian 7 (Wheezy) debian-8 x86_64 Debian 8 (Jessie) fedora-18x86_64 Fedora® 18 fedora-19x86_64 Fedora® 19 fedora-20x86_64 Fedora® 20 fedora-21x86_64 Fedora® 21 Server fedora-21aarch64Fedora® 21 Server (aarch64) fedora-21armv7l Fedora® 21 Server (armv7l) fedora-21ppc64 Fedora® 21 Server (ppc64) fedora-21ppc64leFedora® 21 Server (ppc64le) fedora-22x86_64 Fedora® 22 Server fedora-22aarch64Fedora® 22 Server (aarch64) fedora-22armv7l Fedora® 22 Server (armv7l) scientificlinux-6x86_64 Scientific Linux 6.5 ubuntu-10.04 x86_64 Ubuntu 10.04 (Lucid) ubuntu-12.04 x86_64 Ubuntu 12.04 (Precise) ubuntu-14.04 x86_64 Ubuntu 14.04 (Trusty) Speaking with Thomas Goirand (zigo), I was told about the OpenStack images [3], so it would be nice to have them available in virt-builder for the users of this tool. OK, cool. :-) Basically, it is matter to provide an index file, INI-like style, with the proper metadata for each image (name, description, size, sha512, etc), optionally GPG-signed inline (it is not mandatory, although would be recommeded). You can check the configuration bits in the SOURCES OF TEMPLATES section of [2], and also the current index on libguestfs.org [4] as example. To use this index, there would be the need to install locally (i.e. where virt-builder is installed) a very small .conf file (and the copy of the GPG key, if the index is signed); the files for the libguestfs.org are currently part of the libguestfs-tools package (together with virt-builder), installed under /etc/xdg/virt-builder/repos.d/. I would be happy to help in getting the online metadata (i.e. the index, possibly signed) generated, so how would I get to it? The build is done insida a VM hosted on our CD / live image production machine (pettersson.debian.org) - that's where the openstack images are made. It's not easy to give out access to that VM, but it's easy enough to add extra steps to the image production code that runs there. There's a slight snag, though - the signing process is outside of the VM so we'd probably have to generate the index file then post-process it to add the signature. Also, a different chapter would be having proper non-cloud qcow2 images available (always with virt-builder metadata, of course). This, other than allowing us to avoid maintaining Debian images, would mean virt-builder users can get official images of stable released, updated periodically. Would that be something useful? If so, how/where/etc could this process get started? Sure, we can do that too - we have scope for producing all sorts of images. It's something that there's going to be discussion about at DebConf next month, in fact! -- Steve McIntyre, Cambridge, UK.st...@einval.com Armed with Valor: Centurion represents quality of Discipline, Honor, Integrity and Loyalty. Now you don't have to be a Caesar to concord the digital world while feeling safe and proud. -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: https://lists.debian.org/20150721231343.gs4...@einval.com
