Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-08 Thread Thomas Goirand
On 6/8/20 2:55 PM, Thomas Lange wrote: >> On Sun, 7 Jun 2020 00:26:10 +0200, Thomas Goirand said: > > > It'd be really great if you could implement this. I tried quickly to do > > that on images produced by the team, but also failed quickly and gave up > > quickly too! :) > I

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-08 Thread Thomas Lange
> On Sun, 7 Jun 2020 00:26:10 +0200, Thomas Goirand said: > It'd be really great if you could implement this. I tried quickly to do > that on images produced by the team, but also failed quickly and gave up > quickly too! :) I guess truncating a raw image with GPT fails because

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Andrew Ruthven
Hi Bastian, I was writing a long reply to this, but I've decided not to bother, because it boils down to this: Can you please accept that there are a number, possibly a large, number of clouds out there which can not, and will not, consume qcow2 images natively. We, and I expect all, are not

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Thomas Goirand
On 6/6/20 1:16 PM, Thomas Lange wrote: >> On Sat, 6 Jun 2020 12:28:00 +0200, Bastian Blank said: > > > Also we don't want to specify the size a priori, because it can break to > > easily. So we need to deduct the size during the build process. > I could improve fai-diskimage using

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Thomas Goirand
On 6/6/20 12:11 PM, Bastian Blank wrote: > On Wed, May 27, 2020 at 09:43:01AM +0200, Thomas Goirand wrote: >> On 5/26/20 9:26 PM, Bastian Blank wrote: >>> Please show me this existing 512MB image you are talking about. At >>> least right now it does not exist. The build log currently even

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Thomas Goirand
On 6/6/20 3:13 PM, Bastian Blank wrote: > On Sat, Jun 06, 2020 at 11:16:42PM +1200, Andrew Ruthven wrote: >> Those are examples, and it notes that the formats available are >> configurable and none of them are specified as "must be available". The >> CLI docs also have a similar note. >> "Disk and

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Bastian Blank
On Sat, Jun 06, 2020 at 11:16:42PM +1200, Andrew Ruthven wrote: > Those are examples, and it notes that the formats available are > configurable and none of them are specified as "must be available". The > CLI docs also have a similar note. > "Disk and container formats are configurable on a

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Thomas Lange
> On Sat, 6 Jun 2020 12:28:00 +0200, Bastian Blank said: > Also we don't want to specify the size a priori, because it can break to > easily. So we need to deduct the size during the build process. I could improve fai-diskimage using what zigo is using in his

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Andrew Ruthven
On Sat, 2020-06-06 at 12:06 +0200, Bastian Blank wrote: > On Wed, May 27, 2020 at 10:13:43AM +1200, Andrew Ruthven wrote: > > That may be the case, but not all *backends* can use qcow2 images. > > Can you please show OpenStack documentation detailing this all? I > fail > to find anything. And

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Bastian Blank
On Wed, May 27, 2020 at 12:16:32PM +0100, kuLa wrote: > Are you thinking about doing it within FAI or outside of it using new > tool set for this? Outside. FAI would in any case only create the directory tree, not the filesystem is lives in or the image. So what it does would look the same in

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Bastian Blank
On Thu, Jun 04, 2020 at 01:06:29PM +0200, Thomas Lange wrote: > > On Wed, 27 May 2020 12:01:53 +0200, Bastian Blank > > said: > >> Sadly it is not that easy. A whole bunch of temporary data is deleted > >> in the final stages of the build process. > > > You mean those

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Bastian Blank
Hi Thomas I clearly does not help that you ignore one my questions several times. Maybe you want to stop that. On Wed, May 27, 2020 at 09:43:01AM +0200, Thomas Goirand wrote: > On 5/26/20 9:26 PM, Bastian Blank wrote: > > Please show me this existing 512MB image you are talking about. At > >

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-06 Thread Bastian Blank
On Wed, May 27, 2020 at 10:13:43AM +1200, Andrew Ruthven wrote: > That may be the case, but not all *backends* can use qcow2 images. Can you please show OpenStack documentation detailing this all? I fail to find anything. And if the documentation does not tell clearly, I have to assume that

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-06-04 Thread Thomas Lange
> On Wed, 27 May 2020 12:01:53 +0200, Bastian Blank said: >> Sadly it is not that easy. A whole bunch of temporary data is deleted >> in the final stages of the build process. > You mean those bigger ones? > rm -rf $target/var/cache/apt/* > rm -rf

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-27 Thread kuLa
On 2020-05-27 12:01:53, Bastian Blank wrote: > On Wed, May 27, 2020 at 10:39:01AM +0200, Thomas Lange wrote: > > fai-diskimage has the option -S which specifies the raw disk size. We > > set this in debian_cloud_images/cli/build.py to 2G for genriccloud > > images. It should be easy to set this to

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-27 Thread Thomas Lange
> On Wed, 27 May 2020 12:01:53 +0200, Bastian Blank said: > On Wed, May 27, 2020 at 10:39:01AM +0200, Thomas Lange wrote: >> fai-diskimage has the option -S which specifies the raw disk size. We >> set this in debian_cloud_images/cli/build.py to 2G for genriccloud >> images.

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-27 Thread Bastian Blank
On Wed, May 27, 2020 at 10:39:01AM +0200, Thomas Lange wrote: > fai-diskimage has the option -S which specifies the raw disk size. We > set this in debian_cloud_images/cli/build.py to 2G for genriccloud > images. It should be easy to set this to maybe 550M. Sadly it is not that easy. A whole

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-27 Thread Thomas Lange
> On Tue, 26 May 2020 10:29:17 +0200, Thomas Goirand said: > I already linked to how it could be done. I just asked (mostly Thomas) > if FAI was able to do it by itself, let's wait for his answer. Sorry for answering so late. fai-diskimage has the option -S which specifies the raw

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-27 Thread Bastian Blank
On Tue, May 26, 2020 at 07:50:26PM -0700, Ross Vandegrift wrote: > We'd either need to find the common requirements for all providers (eg AWS > requires 256MiB free), or limit the reduction to the generic images. Free, when and where? Our file-system grow stuff runs pretty early in the boot

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-27 Thread Thomas Goirand
On 5/26/20 9:26 PM, Bastian Blank wrote: > On Tue, May 26, 2020 at 10:29:17AM +0200, Thomas Goirand wrote: >>> The images have 700-800MB of space used, which is still three times the >>> size of the qcow2 file. Why do you refuse to use what's already there? >> >> I already explained: because

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-27 Thread Thomas Goirand
On 5/27/20 4:50 AM, Ross Vandegrift wrote: > On Mon, May 25, 2020 at 07:12:15PM +0200, Thomas Goirand wrote: >> On 5/25/20 5:43 PM, Ross Vandegrift wrote: >>> I don't think it's obvious how to do better. The only ways I know to >>> make a raw image smaller than its fs are: >>> 1) sparse files

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-26 Thread Ross Vandegrift
On Mon, May 25, 2020 at 07:12:15PM +0200, Thomas Goirand wrote: > On 5/25/20 5:43 PM, Ross Vandegrift wrote: > > I don't think it's obvious how to do better. The only ways I know to > > make a raw image smaller than its fs are: > > 1) sparse files > > 2) compression > > > > FAI is using #1,

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-26 Thread Andrew Ruthven
Hey, On Wed, 2020-05-27 at 10:13 +1200, Andrew Ruthven wrote: > That may be the case, but not all *backends* can use qcow2 images. On > our public and private OpenStack clouds we only work with raw images. > We have to convert all qcow2 images. I'd just like to provide some additional feedback

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-26 Thread Andrew Ruthven
On Tue, 2020-05-26 at 21:26 +0200, Bastian Blank wrote: > > Yes, but you point it to the _qcow2_ image. Which, for generic, is > about 250-300MB in size, as it is somewhat compressed, and can be > read > by all the OpenStack tools. That may be the case, but not all *backends* can use qcow2

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-26 Thread Bastian Blank
On Tue, May 26, 2020 at 10:29:17AM +0200, Thomas Goirand wrote: > > The images have 700-800MB of space used, which is still three times the > > size of the qcow2 file. Why do you refuse to use what's already there? > > I already explained: because downloading from ADSL, extracting, then >

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-26 Thread Jeremy Stanley
On 2020-05-26 10:29:17 +0200 (+0200), Thomas Goirand wrote: [...] > Able to extract the .raw file from a tar.xz? I don't think so. If > it can, please provide the command line to do that, but as much as > I know, it's currently not possible. I'd love to be wrong here... [...] It *can* be done

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-26 Thread Thomas Goirand
On 5/25/20 7:36 PM, Bastian Blank wrote: > On Mon, May 25, 2020 at 02:21:48AM +0200, Thomas Goirand wrote: So I was wondering if we could: 1/ Make the resulting extracted disk smaller. That'd be done in FAI, and I have no idea how that would be done. Thomas, can you help, at least

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-25 Thread Andrew Ruthven
On Mon, 2020-05-25 at 19:36 +0200, Bastian Blank wrote: > - the users that would download raw OpenStack images would be > > mainly > > those willing to store them with Ceph as a backend (where sparse > > files > > don't exist anyways, unless I'm mistaking). > > Sorry, but this is bullshit. Ceph

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-25 Thread Bastian Blank
On Mon, May 25, 2020 at 02:21:48AM +0200, Thomas Goirand wrote: > >> So I was wondering if we could: > >> 1/ Make the resulting extracted disk smaller. That'd be done in FAI, and > >> I have no idea how that would be done. Thomas, can you help, at least > >> giving some pointers on how we could

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-25 Thread Thomas Goirand
On 5/25/20 5:43 PM, Ross Vandegrift wrote: > On Mon, May 25, 2020 at 02:21:48AM +0200, Thomas Goirand wrote: >> On 5/24/20 11:39 PM, Bastian Blank wrote: >>> On Sun, May 24, 2020 at 11:26:40PM +0200, Thomas Goirand wrote: So I was wondering if we could: 1/ Make the resulting extracted

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-25 Thread Ross Vandegrift
On Mon, May 25, 2020 at 02:21:48AM +0200, Thomas Goirand wrote: > On 5/24/20 11:39 PM, Bastian Blank wrote: > > On Sun, May 24, 2020 at 11:26:40PM +0200, Thomas Goirand wrote: > >> So I was wondering if we could: > >> 1/ Make the resulting extracted disk smaller. That'd be done in FAI, and > >> I

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-24 Thread Thomas Goirand
On 5/24/20 11:39 PM, Bastian Blank wrote: > On Sun, May 24, 2020 at 11:26:40PM +0200, Thomas Goirand wrote: >> The bigger the image is, the longer it will take to copy, which is an >> operation that OpenStack can do before spawning an instance. > > And you setup instance types with < 2GB disks?

Re: Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-24 Thread Bastian Blank
On Sun, May 24, 2020 at 11:26:40PM +0200, Thomas Goirand wrote: > Currently, our scripts are generating files named for example: > debian-11-genericcloud-amd64-daily-20200524-273.tar.xz No, they generate this one:

Publishing raw generic{,cloud} images without tar, and without compression, plus versionning of point releases

2020-05-24 Thread Thomas Goirand
Hi, Currently, our scripts are generating files named for example: debian-11-genericcloud-amd64-daily-20200524-273.tar.xz The problem with these, is that you can't use then directly with Glance. One must first download them, extract the content, and then upload the result. This is very annoying,