Re: ISO image: to xz or not to xz?

[Vagrant Cascadian]

On 2021-05-03, Tobias Geerinckx-Rice wrote:
> Tangent: I sense some undeserved mysticism surrounding squashfs. 
> It is not designed to be loop-mounted, any more than ext2 was.  It 
> does not enjoy it.  People should stop doing it.

The only mysticism I see here is attributing enjoyment to a
filesystem. :)

Is mounting on a loopback device really any different from any other
block device?


> But they won't, because many distributions still insist that the 
> same installer image must be both a bootable CD/DVD *and* boot 
> when dd'd to a USB drive, on every PC ever made.

> That ‘isohybrid’ dream justifies doing unmentionable things to an 
> iso9660 file system (and only an iso9660 file system), so they 
> must put the real squashfs on top of that and loop-mount it and 
> ignore the screams I guess and--

Never heard the screams; what frequency does squashfs emit screams at?
:)

People have made it work well enough for only slightly less long than I
can remember using free software operating systems...


> Vagrant Cascadian 写道：
>> Well, the suggestion to use squashfs does bear merit;
>
> It's not a *bad* suggestion, just a bit obvious.

Fair enough.


>> it would require having some type of writeable filesystem on top,
>> such as using overlay fs to mount the installer rootfs with squashfs
>> for the readonly bits, and tmpfs for the writeable bits.
>
> We've always done this.

I *thought* so, but...


>> As a bonus, using a tmpfs overlay would solve the issue brought up
>> recently by someone who tried using the same installer image multiple
>> times, and /gnu/store and /var/guix got out of sync due to the
>> cow-store only writing to the newly installed system, so that the
>> second install failed.
>
> ...so no, it definitely wouldn't, but I think it's valuable to 
> understand why you thought so!
>
> Could you elaborate?

Mostly I was referring to:

  https://lists.gnu.org/archive/html/guix-devel/2021-04/msg00546.html

Though I haven't confirmed that behavior myself. Probably deserves a
proper bug report.


live well,
  vagrant


signature.asc
Description: PGP signature

Re: ISO image: to xz or not to xz?

[Tobias Geerinckx-Rice]

Hi Vagrant!

Tangent: I sense some undeserved mysticism surrounding squashfs. 
It is not designed to be loop-mounted, any more than ext2 was.  It 
does not enjoy it.  People should stop doing it.


But they won't, because many distributions still insist that the 
same installer image must be both a bootable CD/DVD *and* boot 
when dd'd to a USB drive, on every PC ever made.


That ‘isohybrid’ dream justifies doing unmentionable things to an 
iso9660 file system (and only an iso9660 file system), so they 
must put the real squashfs on top of that and loop-mount it and 
ignore the screams I guess and--


...sorry; I got carried away.

Vagrant Cascadian 写道：

Well, the suggestion to use squashfs does bear merit;


It's not a *bad* suggestion, just a bit obvious.


it would require
having some type of writeable filesystem on top, such as using 
overlay
fs to mount the installer rootfs with squashfs for the readonly 
bits,

and tmpfs for the writeable bits.


We've always done this.

As a bonus, using a tmpfs overlay would solve the issue brought 
up
recently by someone who tried using the same installer image 
multiple
times, and /gnu/store and /var/guix got out of sync due to the 
cow-store
only writing to the newly installed system, so that the second 
install

failed.


...so no, it definitely wouldn't, but I think it's valuable to 
understand why you thought so!


Could you elaborate?

Another angle might be to use a compressable but writeable 
filesystem

(btrfs?).


A persistently mutable . . . installation medium . . . ?

Very lost, slightly frightened,

T G-R


signature.asc
Description: PGP signature

Re: ISO image: to xz or not to xz?

[Vagrant Cascadian]

On 2021-05-03, Leo Famulari wrote:
> On Mon, May 03, 2021 at 01:47:02PM -0300, Alexandre Oliva wrote:
>> Indeed, install ISOs normally hold already-compressed filesystems or
>> files, so recompressing the .iso doesn't gain much if at all.
>
> To quote the introductory message of this thread:
>
> "The xz-compressed image is 23% smaller, which is not negligible."

Well, the suggestion to use squashfs does bear merit; it would require
having some type of writeable filesystem on top, such as using overlay
fs to mount the installer rootfs with squashfs for the readonly bits,
and tmpfs for the writeable bits.

As a bonus, using a tmpfs overlay would solve the issue brought up
recently by someone who tried using the same installer image multiple
times, and /gnu/store and /var/guix got out of sync due to the cow-store
only writing to the newly installed system, so that the second install
failed.

Another angle might be to use a compressable but writeable filesystem
(btrfs?).

Obviously, it requires someone to do the work to get there!


live well,
  vagrant


signature.asc
Description: PGP signature

Re: ISO image: to xz or not to xz?

[Tobias Geerinckx-Rice]

Alexandre Oliva 写道：
Consider using something like a loopback-mounted squashfs for 
the bulk

of the data in the install media for a future release.


Read .

I did play with the obvious alternative, but got so bored -- 
squashfs is no forgotten 1990s Linux-only extension to iso9660! -- 
that saving a mere 20% more was not worth it.


By this I hope to trick you into thinking it is, and submitting a 
patch, because to do so you'd have to fix one of the Forever Bugs 
that *is* worth fixing: separate /boot support.  :-)


Kind regards,

T G-R


signature.asc
Description: PGP signature

Re: ISO image: to xz or not to xz?

[Leo Famulari]

On Mon, May 03, 2021 at 01:47:02PM -0300, Alexandre Oliva wrote:
> Indeed, install ISOs normally hold already-compressed filesystems or
> files, so recompressing the .iso doesn't gain much if at all.

To quote the introductory message of this thread:

"The xz-compressed image is 23% smaller, which is not negligible."

Re: ISO image: to xz or not to xz?

[Alexandre Oliva]

On Apr 28, 2021, Ludovic Courtès  wrote:

> However, it’s apparently quite unusual to distribute compressed ISOs and
> some services/tools such as libosinfo require plain ISOs (uncompressed).

Indeed, install ISOs normally hold already-compressed filesystems or
files, so recompressing the .iso doesn't gain much if at all.

> Should we distribute the installation ISO without xz compression?

If the iso is so compressible, offering a compressed version for
download makes sense, but it will still be wasteful of install media.

Consider using something like a loopback-mounted squashfs for the bulk
of the data in the install media for a future release.

-- 
Alexandre Oliva, happy hackerhttps://FSFLA.org/blogs/lxo/
   Free Software Activist   GNU Toolchain Engineer
Disinformation flourishes because many people care deeply about injustice
but very few check the facts.  Ask me about 

Re: ISO image: to xz or not to xz?

[Leo Famulari]

On Wed, Apr 28, 2021 at 10:18:01PM +0200, Ludovic Courtès wrote:
> What do people think?

We've received several complaints about the images being compressed. I
guess we might as well offer it uncompressed, and maybe offer both ways.

Re: ISO image: to xz or not to xz?

[Christopher Baines]

Ludovic Courtès  writes:

> Hi!
>
> Here’s the installation ISO image (with built-in zlib compression):
>
> --8<---cut here---start->8---
> $ ./pre-inst-env guix system image -t iso9660 gnu/system/install.scm
> /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso
> $ xz < /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso > /tmp/t.iso.xz
> $ du -h /tmp/t.iso.xz
> 496M  /tmp/t.iso.xz
> $ du -h /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso
> 647M  /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso
> --8<---cut here---end--->8---
>
> The xz-compressed image is 23% smaller, which is not negligible.
> However, it’s apparently quite unusual to distribute compressed ISOs and
> some services/tools such as libosinfo require plain ISOs (uncompressed).
>
> Should we distribute the installation ISO without xz compression?

Personally I think the ISO is more useful if it's published as such,
rather than an as a xz compressed file.

I believe this will make things like getting Guix through Gnome Boxes
easier, and using Guix at various hosting providers easier.


signature.asc
Description: PGP signature

Re: ISO image: to xz or not to xz?

[Development of GNU Guix and the GNU System distribution.]

Ludovic Courtès  writes:

> Should we distribute the installation ISO without xz compression?

Yes.  I've always found that annoying with Guix, because it adds one
more step to the download and PGP verification process.  All of Debian,
Ubuntu, NixOS, etc distribute plain ISO's.

/Simon


signature.asc
Description: PGP signature

Re: ISO image: to xz or not to xz?

[Vincent Legoll]

On Wed, Apr 28, 2021 at 10:18 PM Ludovic Courtès  wrote:
> What do people think?

I'd say distribute both, so the bandwidth-starved can get the
smaller, and the CPU-starved can get the uncompressed one.

-- 
Vincent Legoll

ISO image: to xz or not to xz?

[Ludovic Courtès]

Hi!

Here’s the installation ISO image (with built-in zlib compression):

--8<---cut here---start->8---
$ ./pre-inst-env guix system image -t iso9660 gnu/system/install.scm
/gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso
$ xz < /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso > /tmp/t.iso.xz
$ du -h /tmp/t.iso.xz
496M/tmp/t.iso.xz
$ du -h /gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso
647M/gnu/store/kg63cyg94dmy9x7rpxf5ycn97sqx1ndl-image.iso
--8<---cut here---end--->8---

The xz-compressed image is 23% smaller, which is not negligible.
However, it’s apparently quite unusual to distribute compressed ISOs and
some services/tools such as libosinfo require plain ISOs (uncompressed).

Should we distribute the installation ISO without xz compression?

That could be done in respect of the string freeze (removing text, not
modifying it), though the relevant section would look odd because it’d
have a single step (info "(guix) USB Stick and DVD Installation").

Or maybe it’s safer to postpone that question until the next release.

What do people think?

Ludo’.