Re: Use genimage for disk-image creation.
Hi! Vagrant Cascadian skribis: > On 2020-03-29, Danny Milosavljevic wrote: >> Hi Ludo, >> >> On Sun, 29 Mar 2020 16:44:39 +0200 >> Ludovic Courtès wrote: >> >>> Oh, really? I’m surprised partitioning causes problems (though I’m >>> not familiar with embedded dev!). >> >> Well, maybe not that bad, but it's pretty bad. >> >> It's because the boot sector for ARM, for some unfathomable reason, is not >> the first sector but has basically a vendor-dependent sector number somewhere >> in the middle of the disk :P Oh I see, I actually noticed that in (gnu bootloader u-boot) and in Buildroot, it’s teible. > I haven't really looked at buildroot at all... but I suspect buildroot > is just a collection of all these criteria applied on a board-by-board > basis, but most of these boards don't actually require specific > partition layout, per se, it's just nice to not clobber the raw offsets > of various parts of the boot process... but creating partitions for each > of those can make installation less error-prone in some cases. Yeah, I guess we could keep importing them like Danny did in the past, but of course it would be nice to have an automated process to do that. Ludo’.
Re: Use genimage for disk-image creation.
On 2020-03-29, Danny Milosavljevic wrote: > Hi Ludo, > > On Sun, 29 Mar 2020 16:44:39 +0200 > Ludovic Courtès wrote: > >> Oh, really? I’m surprised partitioning causes problems (though I’m >> not familiar with embedded dev!). > > Well, maybe not that bad, but it's pretty bad. > > It's because the boot sector for ARM, for some unfathomable reason, is not > the first sector but has basically a vendor-dependent sector number somewhere > in the middle of the disk :P > > (with the "boot sector" I mean the sector(s) where usually u-boot resides) > > So there are all kinds of stupid problems like sometimes you have to reduce > the size of the GPT partition table (TABLE!) because it would write right over > the actual boot sector of the platform otherwise. For that vendor. Other > vendors do other silly things. > > Parted doesn't really have support for that stuff either, and I while I > brought it up with them[1], I can't think of a practical way to fix it > (i.e. automatically keep clear of all possible boot ranges of all ARM > vendors) or to even find the extent of the problem. I think it should be > the vendors' responsibility to standardize on one way to boot that doesn't > suck :P > > It turns partitioning basically from straightforward allocation inside > one block into allocation with strange hidden mines to avoid. > > Embedded usually ignores the problem, finds a partitioning that works > (with dummy spacer partition if you are lucky--otherwise just random > gap) and everyone just uses that one. Sigh... Someday... but not today. :/ The biggest offset I'm aware of is rockchip platforms, at what I think is ~16MB: http://opensource.rock-chips.com/wiki_Partitions sunxi/allwinner 64-bit platforms have offsets that conflict with a standard GPT partition table, though there are rumours of it working fine as long as you keep the number of partitions below four: https://bugs.debian.org/928643 32-bit sunxi/allwinner platforms are generally pretty reasonable, but I don't recall the issues off the top of my head. ti/omap/etc platforms tend to fit under 4MB. I haven't really looked at buildroot at all... but I suspect buildroot is just a collection of all these criteria applied on a board-by-board basis, but most of these boards don't actually require specific partition layout, per se, it's just nice to not clobber the raw offsets of various parts of the boot process... but creating partitions for each of those can make installation less error-prone in some cases. I think the default for most partitioning tools these days is to create the first partition starting at 2MB, but If the guix partition started at 16MB and you limited partitions to under four partitions, that should work for all the platforms I'm aware off of the top of my head... at least for now... I seem to also recall an that some disk media (emmc, microSD, maybe some SSDs) have an erase block granularity of 4MB, and so it'd be ideal if the first partition started at a multiple of 4MB: $ lsblk -D NAMEDISC-ALN DISC-GRAN DISC-MAX DISC-ZERO mmcblk004M 3.5G 0 Or alternately, detecting this size and making sure the first partition starts as a multiple of this value. I really need to sort some of these issues out in Debian as well, so happy to carry on the conversation! live well, vagrant signature.asc Description: PGP signature
Re: Use genimage for disk-image creation.
Hey! > I’m completely sold to the idea. :-) I'm glad you like it! > Looking at ‘image-ext2.c’ reveals that genimage actually just shells out > to mke2fs. Indeed, I discovered that ‘mke2fs -d /my/root’ copies > /my/root as the image’s root directory. Likewise, for ISO, it just > shells out to ‘genisoimage’. > > So I think that we could avoid ‘genimage’ altogether and implement > similar functionality for ext4/ISO in (gnu build disk-image). Ok! One useful functionality of genimage seems to be that's its capable of creating complex partitioning layouts. Of course we can re-implement it in (gnu build disk-image), but I don't know how complex will it be. Danny, I found the genimage files for each arm/aarch64 boards inside Buildroot repository. They are a valuable source of information, but regardless of the decision we take for (gnu build disk-image), we can find a way to import them and store them as Guile records. I guess I'll start a small implementation to have a clearer view of this topic. Thank you all for the feedback, Mathieu
Re: Use genimage for disk-image creation.
Hi Ludo, On Sun, 29 Mar 2020 16:44:39 +0200 Ludovic Courtès wrote: > Oh, really? I’m surprised partitioning causes problems (though I’m > not familiar with embedded dev!). Well, maybe not that bad, but it's pretty bad. It's because the boot sector for ARM, for some unfathomable reason, is not the first sector but has basically a vendor-dependent sector number somewhere in the middle of the disk :P (with the "boot sector" I mean the sector(s) where usually u-boot resides) So there are all kinds of stupid problems like sometimes you have to reduce the size of the GPT partition table (TABLE!) because it would write right over the actual boot sector of the platform otherwise. For that vendor. Other vendors do other silly things. Parted doesn't really have support for that stuff either, and I while I brought it up with them[1], I can't think of a practical way to fix it (i.e. automatically keep clear of all possible boot ranges of all ARM vendors) or to even find the extent of the problem. I think it should be the vendors' responsibility to standardize on one way to boot that doesn't suck :P It turns partitioning basically from straightforward allocation inside one block into allocation with strange hidden mines to avoid. Embedded usually ignores the problem, finds a partitioning that works (with dummy spacer partition if you are lucky--otherwise just random gap) and everyone just uses that one. Sigh... (x86 kinda has standardized a gap at the beginning of the disk until the second head (seems to be 2048 sectors with 512 Byte/sector nowadays since the drive doesn't expose valid CHS information anymore), and put the bootloader there--so that's a 1 MiB gap at the beginning of the disk) [1] https://alioth-lists.debian.net/pipermail/parted-devel/2017-December/005152.html pgpE0x1TPzX8z.pgp Description: OpenPGP digital signature
Re: Use genimage for disk-image creation.
Hi, Danny Milosavljevic skribis: > from the standpoint of ARM it would be really good to be able to reuse > genimage config files from buildroot. Well, we can have a parser. :-) >From an engineering viewpoint, it seems weird to add a C parser that basically shells out to other tools when we could just invoke those commands directly and hopefully get more control and better error reporting. > In fact, partition layout is a major pain in the ass to get right on ARM. > If we want to support a large number of ARM platforms, that would > mean the partitioning would be fixed and not user-modifyable anyway. Oh, really? I’m surprised partitioning causes problems (though I’m not familiar with embedded dev!). Ludo’.
Re: Use genimage for disk-image creation.
Hello > > FYI, I'm investigating the test suite failures for the newer > > releases of genimage (v11 (+1 FAIL) & v12 (+2 FAIL))... > > > > I think I'll report / PR them upstream before updating > > guix's version, to avoid unnecessary churn. > > Ok good, let us know how it goes :) I should have mentioned that I had > to disable the tests to be able to build genimage. I've sent an update to v11, this afternoon, since the fix for that one was simple enough. But I've yet to receive the ack from debbugs for it. That's strange, I usually get that ack in just a few minutes... For v12, I'll need to dig further, even if the test suite failure is probably harmless... Details in the guix-patches email for v11, when that surfaces. -- Vincent Legoll
Re: Use genimage for disk-image creation.
Hello Vincent, > FYI, I'm investigating the test suite failures for the newer > releases of genimage (v11 (+1 FAIL) & v12 (+2 FAIL))... > > I think I'll report / PR them upstream before updating > guix's version, to avoid unnecessary churn. Ok good, let us know how it goes :) I should have mentioned that I had to disable the tests to be able to build genimage. Thanks, Mathieu
Re: Use genimage for disk-image creation.
Hi Mathieu, from the standpoint of ARM it would be really good to be able to reuse genimage config files from buildroot. In fact, partition layout is a major pain in the ass to get right on ARM. If we want to support a large number of ARM platforms, that would mean the partitioning would be fixed and not user-modifyable anyway. > Danny, Ludo, WDYT? Could we modify "system-disk-image" to use genimage > as a backend instead of spawning a VM? It depends on whether genimage supports all filesystems we care about. I suspect that it does, though. Genimage images tend to be reproducible btw, so it's kinda nice. pgpDsfBtMiJGj.pgp Description: OpenPGP digital signature
Re: Use genimage for disk-image creation.
> I’m completely sold to the idea. :-) Yep, LGTM too > Apparently ‘genimage’ supports many file systems, including ext[234] and > ISO9660, which are the two formats we support via ‘--file-system-type’. > It does not support Btrfs, but ‘guix system disk-image’ doesn’t support > it either so far. > > Likewise, for ISO, it just shells out to ‘genisoimage’. You probably meant "mkisoimage" ? > So I think that we could avoid ‘genimage’ altogether and implement > similar functionality for ext4/ISO in (gnu build disk-image). Do you anticipate less or similar work in reimplementing rather than reusing ? Genuinely asking, because we may want to support other image formats in the future, so reusing may have its interest too... Or is there another reason that I'm not seing, like maybe avoiding some overhead (probably not the fork/execs going through genisoimage though)... -- Vincent Legoll
Re: Use genimage for disk-image creation.
Hi Mathieu! Mathieu Othacehe skribis: > Let say I want to create a disk-image with one ext4 partition starting > at offset 10M. I can write the following genimage config file: > > image system { > hdimage {} > > partition rootfs { > partition-type = 0x83 > image = "rootfs.ext4" > size = 8G > offset = 10M > } > } > > image rootfs.ext4 { > name = "rootfs" > ext4 { > label = "rootfs" > use-mke2fs = true > } > size = 8G > mountpoint = "/" > } > > > and run the command: > > fakeroot genimage --config ~/tmp/genimage.cfg > --rootpath=/home/mathieu/image-root/ > > where image-root is a directory containing the result of a `guix system > init`. The directory size is about 6GiB. > > It takes 8 minutes to generate this disk-image, versus 2h30 using `guix > system disk-image`. > > I'm aware that this might not be a fair comparison but, I think its > already significant. > > Danny, Ludo, WDYT? Could we modify "system-disk-image" to use genimage > as a backend instead of spawning a VM? I’m completely sold to the idea. :-) Apparently ‘genimage’ supports many file systems, including ext[234] and ISO9660, which are the two formats we support via ‘--file-system-type’. It does not support Btrfs, but ‘guix system disk-image’ doesn’t support it either so far. Looking at ‘image-ext2.c’ reveals that genimage actually just shells out to mke2fs. Indeed, I discovered that ‘mke2fs -d /my/root’ copies /my/root as the image’s root directory. Likewise, for ISO, it just shells out to ‘genisoimage’. So I think that we could avoid ‘genimage’ altogether and implement similar functionality for ext4/ISO in (gnu build disk-image). WDYT? Thanks, Ludo’.
Re: Use genimage for disk-image creation.
On Thu, Mar 26, 2020 at 10:55:04AM +0100, Vincent Legoll wrote: > FYI, I'm investigating the test suite failures for the newer > releases of genimage (v11 (+1 FAIL) & v12 (+2 FAIL))... > > I think I'll report / PR them upstream before updating > guix's version, to avoid unnecessary churn. > I found that updating Guix's version of mtools also spawned errors on genimage on aarch64/armhf (IIRC), so that might also be something worth looking into. -- Efraim Flashner אפרים פלשנר GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted signature.asc Description: PGP signature
Re: Use genimage for disk-image creation.
FYI, I'm investigating the test suite failures for the newer releases of genimage (v11 (+1 FAIL) & v12 (+2 FAIL))... I think I'll report / PR them upstream before updating guix's version, to avoid unnecessary churn. -- Vincent Legoll
Use genimage for disk-image creation.
Hello, I recently spent some time generating a lot of Guix System disk-images (embedded system images, installation tests). The mechanism Guix and Nix are using: spawning a VM to create the disk-image does not scale at all when the disk-image size increase. So, I looked at what the competition does. Buildroot is using genimage[1] to generate disk-images. Yocto is using wic[2], that's based on mic[3]. Both are able to create disk-images as unprivileged users, without resorting to VM. While, Yocto solution is tied to the rest of their framework, genimage is standalone. In fact, Danny has already packaged it and made some research about it. I think we could use genimage to replace our current mechanism to generate disk-images. Here's a small proof of concept. Let say I want to create a disk-image with one ext4 partition starting at offset 10M. I can write the following genimage config file: --8<---cut here---start->8--- image system { hdimage {} partition rootfs { partition-type = 0x83 image = "rootfs.ext4" size = 8G offset = 10M } } image rootfs.ext4 { name = "rootfs" ext4 { label = "rootfs" use-mke2fs = true } size = 8G mountpoint = "/" } --8<---cut here---end--->8--- and run the command: --8<---cut here---start->8--- fakeroot genimage --config ~/tmp/genimage.cfg --rootpath=/home/mathieu/image-root/ --8<---cut here---end--->8--- where image-root is a directory containing the result of a `guix system init`. The directory size is about 6GiB. It takes 8 minutes to generate this disk-image, versus 2h30 using `guix system disk-image`. I'm aware that this might not be a fair comparison but, I think its already significant. Danny, Ludo, WDYT? Could we modify "system-disk-image" to use genimage as a backend instead of spawning a VM? Thanks, Mathieu [1]: https://github.com/pengutronix/genimage [2]: https://www.yoctoproject.org/docs/2.4.2/dev-manual/dev-manual.html#creating-partitioned-images-using-wic [3]: https://source.tizen.org/documentation/reference/mic-image-creator [4]: https://lists.gnu.org/archive/html/guix-devel/2019-09/msg9.html