Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Hi, i wrote: > Date: Fri, 23 Oct 2020 19:29:46 +0200 > Date: Fri, 23 Oct 2020 21:55:56 +0200 My apologies for these repetitions. I had hoped that the queue dropped them either because of their mail header deficiencies or because i was not subscribed when i sent them. Have a nice day :) Thomas ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Hi, Chris Adams wrote: > Although: now that I look at the ISO structure with isoinfo, it does > appear that the files are not actually duplicated ECMA-119 (ISO 9660 for free) allows the equivalent to hardlinking by 6.5.1 Relation to File Sections ... A File Section may be part of more than one file and may occur more than once in the same file. A File Section may be identified by more than one record in the same or a different directory. As you found out, genisoimage makes use of this prescription by putting together hardlink siblings from the input filesystem. libisofs underneath xorriso does the same. --- Rock Ridge 1.12 as add-on to ISO9660 records the equivalent to inode numbers as "File Serial Number" in PX entries. But the Fedora ISOs i have (from genisoimage and xorriso) are equipped with PX entries of version 1.10 which do not bear this field. Whatever, the Linux kernel ignores the recorded numbers of version 1.12 and rather computes the inode numbers from the byte address of the directory record divided by 32. So two different file paths get two different artificial inode numbers. (See isofs_get_ino() in fs/isofs/isofs.h.) The consequence is that the hard-link-like relations do not show up in mounted ISOs and that ISOs made from mounted ISOs don't bear this relation either. Have a nice day :) Thomas ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Hi, Neal Gompa wrote: > Could we just drop syslinux/isolinux and use GRUB there? Other > distributions have done that I understand Ubuntu did it just yesterday. https://releases.ubuntu.com/20.10/ubuntu-20.10-desktop-amd64.iso The lesson from https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148 https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1895131 https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308 is is to append the EFI partition image to the end of the ISO, use xorriso partition offset 16 to enclose it into the overall ISO's filesystem size, mark the partitions by valid GPT (because of recent Lenovo laptops), and add a MBR partition hack which bears the "bootable" flag (for old HP laptops). I.e. mbr=...disk.path.to.grub.mbr.template.with.bios.x86.code... efi=...disk.path.to.FAT.filesystem.image.for.EFI... --grub2-mbr "$mbr" -append_partition 2 0xEF "$efi" -partition_offset 16 -appended_part_as_gpt --mbr-force-bootable ... -b 'boot/grub/i386-pc/eltorito.img' ... -eltorito-alt-boot -e '--interval:appended_partition_2' -no-emul-boot instead of -isohybrid-mbr ...disk.path.to.../isohdpfx.bin ... -b 'isolinux/isolinux.bin' ... -eltorito-alt-boot -e 'images/efiboot.img' -no-emul-boot -isohybrid-gpt-basdat The equipment as used by grub-mkrescue should work. For getting an impression, install the packages for BIOS, 64-bit x86 EFI, 32-bit x86 EFI, and then run grub-mkrescue with some dummy directory as input. The emerging GPT is ugly because grub-mkrescue puts the EFI partition image as file into the ISO and marks it there. But the MBR and /boot/grub/i386-pc/eltorito.img will probably be what you want for legacy BIOS. grub-mkrescue for 64-bit EFI creates a HFS+ filesystem tree and marks it by an Apple Partition Map. One should consider whether there is still an audience of old x86 Macs which doe not do EFI. The new GRUB-only Ubuntu ISO seems to boot on as many systems as did the old layout with ISOLINUX and GRUB, invented by Matthew J. Garrett for Fedora 8 years ago. (Fedora adds a HFS filesystem image and a Apple Partition Map, which Ubuntu did not do in the last years. It could be replaced by a HFS+ tree like for grub-mkrescue.) Further an unpacked copy of the EFI partition filesystem's \BOOT\EFI should be present in the ISO as /BOOT/EFI. MS-Windows users seem to be used to copy the whole ISO file tree into a FAT partition and to expect that it then boots by EFI. (The scandal is that this really does work.) Chris Murphy wrote: > I don't know the history of why we left it as isolinux for BIOS though. Given above woes of Ubuntu, i'd say it was prudent conservatism. The switch needed nerves and some help from bystanders. Have a nice day :) Thomas ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Hi,
Gerd Hoffmann wrote:
> There isn't something like a link
> count field in the directory listing, so this is *not* visible with
> "ls".
There is something like a link count field. But a link count > 1 makes
full sense only if the link siblings have the same inode number. Those
numbers can be recorded in Rock Ridge entry PX if Rock Ridge 1.12 is
enabled:
xorriso -compliance new_rr -as mkisofs ...usual.options...
But the production of Fedora ISOs does not generate them this way and the
interpretation in the Linux kernel would ignore them.
> The only way to figure the link count is scanning the whole file
> system
Not with Linux.
# mount Fedora-Workstation-Live-x86_64-31-1.9.iso
$ ls -l /mnt/iso/images/pxeboot
total 65293
-rw-r--r-- 2 root root 57536044 Oct 24 2019 initrd.img
-rwxr-xr-x 2 root root 9323208 Oct 21 2019 vmlinuz
The PX entry has a field "POSIX File Links" which "shall be used for the
st_nlink field of POSIX:5.6.1.".
This is present even in RRIP 1.10 but makes few sense there, because no
inode numbers can be recorded in 1.10.
Nevertheless Linux interprets this value in fs/isofs/rock.c :
case SIG('P', 'X'):
inode->i_mode = isonum_733(rr->u.PX.mode);
set_nlink(inode, isonum_733(rr->u.PX.n_links));
Simple example:
echo hello >/tmp/x
xorriso -as mkisofs -o test.iso -graft-points /x=/tmp/x /y=/tmp/x
mount test.iso /mnt/iso
ls -li /mnt/iso
yields
total 1
1159 -rw-r--r-- 2 thomas thomas 6 Oct 26 09:58 x
1162 -rw-r--r-- 2 thomas thomas 6 Oct 26 09:58 y
But, as said, Linux does not care if a PX entry has length >= 44 and thus
a field "File Serial Number". The struct RR_PX_s in fs/isofs/rock.h has
no field member for byte offset 36 to 43.
So we have no consistent display of effectively hardlinked files.
The shown inode numbers roughly tell where the directory entries of the
files are:
$ expr 1159 '*' 32
37088
$ dd if=test.iso bs=1 skip=37088 count=110 | od -c
000 \b : ( \0 j \0 ! \0 \0 \0 \0 \0 \0 ! 006 \0
020 \0 \0 \0 \0 \0 006 x \n 032 \b : 026 \0 \0 \0 \0
040 001 \0 \0 001 004 X . ; 1 \0 P X $ 001 244 201
060 \0 \0 \0 \0 201 244 002 \0 \0 \0 \0 \0 \0 002 350 003
100 \0 \0 \0 \0 003 350 350 003 \0 \0 \0 \0 003 350 T F
120 032 001 016 x \n 032 \b : 026 \0 x \a 020 \v 030 /
140 \0 x \n 032 \b : 026 \0 N M 006 001 \0 x
156
The ISO 9660 name start byte "X" marks byte offset 33 in the directory
record. So the "j" = 106 decimal is the size of the record. The record
ends by the Rock Ridge name entry NM, telling the Rock Ridge file name "x".
PX has size 36 and thus contains no File Serial number.
With RRIP 1.12 it looks like:
$ dd if=test.iso bs=1 skip=37088 count=144 | od -c
000 032 001 016 x \n 032 \t 020 * \0 x \n 032 \t 020 *
020 \0 x \n 032 \t 020 * \0 N M 005 001 004 \0 r \0
040 ! \0 \0 \0 \0 \0 \0 ! 006 \0 \0 \0 \0 \0 \0 006
060 x \n 032 \b : 026 \0 \0 \0 \0 001 \0 \0 001 004 X
100 . ; 1 \0 P X , 001 244 201 \0 \0 \0 \0 201 244
120 002 \0 \0 \0 \0 \0 \0 002 350 003 \0 \0 \0 \0 003 350
140 350 003 \0 \0 \0 \0 003 350 002 \0 \0 \0 \0 \0 \0 002
160 T F 032 001 016 x \n 032 \b : 026 \0 x \n 032 \b
200 : ( \0 x \n 032 \b : 026 \0 N M 006 001 \0 x
The directory record starts at the first "r", i.e. length is 114.
Here we see PX of size 44 (",") with last field value
002 \0 \0 \0 \0 \0 \0 002
which is how ISO 9660 expresses the 32 bit value 2 (before "T F").
In the next directory record we see the same File Serial Number 2 (again
before the field name "T F"):
$ dd if=test.iso bs=1 skip=37232 count=114 | od -c
000 r \0 ! \0 \0 \0 \0 \0 \0 ! 006 \0 \0 \0 \0 \0
020 \0 006 x \n 032 \b : 026 \0 \0 \0 \0 001 \0 \0 001
040 004 Y . ; 1 \0 P X , 001 244 201 \0 \0 \0 \0
060 201 244 002 \0 \0 \0 \0 \0 \0 002 350 003 \0 \0 \0 \0
100 003 350 350 003 \0 \0 \0 \0 003 350 002 \0 \0 \0 \0 \0
120 \0 002 T F 032 001 016 x \n 032 \b : 026 \0 x \n
140 032 \b : ( \0 x \n 032 \b : 026 \0 N M 006 001
160 \0 y
162
I am not aware how complicated it would be to put that number 2 into
struct inode.i_ino without confusing the overall isofs driver.
The act itself would be quite trivial in above code snippet after having
extended struct RR_PX_s by the RRIP 1.12 field and interpreting the
member .len of struct rock_ridge to evaluate whether the new field is
valid.
Without reading the inode number of a file from the ISO it makes few sense
to read its link count instead of setting it flatly to 1. So an easy
change
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Hi, > > That's curious, when I mount the server dvd ISO, those files have > > different inode numbers. > > Sorry for the late reply on this, yet, they are hardlinked. Well, iso9660 is pretty simple. File data is basically referenced as "offset,length". So hardlinked files can be trivially be made referencing the same data blocks. There isn't something like a link count field in the directory listing, so this is *not* visible with "ls". The only way to figure the link count is scanning the whole file system and see whenever there are other files with the same offset, and as far I know nobody goes that extra mile just to figure the link count for "ls". take care, Gerd ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Hi, Chris Adams wrote: > Although: now that I look at the ISO structure with isoinfo, it does > appear that the files are not actually duplicated ECMA-119 (ISO 9660 for free) allows the equivalent to hardlinking by 6.5.1 Relation to File Sections ... A File Section may be part of more than one file and may occur more than once in the same file. A File Section may be identified by more than one record in the same or a different directory. As you found out, genisoimage makes use of this prescription by putting together hardlink siblings from the input filesystem. libisofs underneath xorriso does the same. --- Rock Ridge 1.12 as add-on to ISO9660 records the equivalent to inode numbers as "File Serial Number" in PX entries. But the Fedora ISOs i have (from genisoimage and xorriso) are equipped with PX entries of version 1.10 which do not bear this field. Whatever, the Linux kernel ignores the recorded numbers of version 1.12 and rather computes the inode numbers from the byte address of the directory record divided by 32. So two different file paths get two different artificial inode numbers. (See isofs_get_ino() in fs/isofs/isofs.h.) The consequence is that the hard-link-like relations do not show up in mounted ISOs and that ISOs made from mounted ISOs don't bear this relation either. Have a nice day :) Thomas ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Hi, Neal Gompa wrote: > Could we just drop syslinux/isolinux and use GRUB there? Other > distributions have done that I understand Ubuntu did it just yesterday. https://releases.ubuntu.com/20.10/ubuntu-20.10-desktop-amd64.iso The lesson from https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148 https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1895131 https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308 is is to append the EFI partition image to the end of the ISO, use xorriso partition offset 16 to enclose it into the overall ISO's filesystem size, mark the partitions by valid GPT (because of recent Lenovo laptops), and add a MBR partition hack which bears the "bootable" flag (for old HP laptops). I.e. mbr=...disk.path.to.grub.mbr.template.with.bios.x86.code... efi=...disk.path.to.FAT.filesystem.image.for.EFI... --grub2-mbr "$mbr" -append_partition 2 0xEF "$efi" -partition_offset 16 -appended_part_as_gpt --mbr-force-bootable ... -b 'boot/grub/i386-pc/eltorito.img' ... -eltorito-alt-boot -e '--interval:appended_partition_2' -no-emul-boot instead of -isohybrid-mbr ...disk.path.to.../isohdpfx.bin ... -b 'isolinux/isolinux.bin' ... -eltorito-alt-boot -e 'images/efiboot.img' -no-emul-boot -isohybrid-gpt-basdat The equipment as used by grub-mkrescue should work. For getting an impression, install the packages for BIOS, 64-bit x86 EFI, 32-bit x86 EFI, and then run grub-mkrescue with some dummy directory as input. The emerging GPT is ugly because grub-mkrescue puts the EFI partition image as file into the ISO and marks it there. But the MBR and /boot/grub/i386-pc/eltorito.img will probably be what you want for legacy BIOS. grub-mkrescue for 64-bit EFI creates a HFS+ filesystem tree and marks it by an Apple Partition Map. One should consider whether there is still an audience of old x86 Macs which doe not do EFI. The new GRUB-only Ubuntu ISO seems to boot on as many systems as did the old layout with ISOLINUX and GRUB, invented by Matthew J. Garrett for Fedora 8 years ago. (Fedora adds a HFS filesystem image and a Apple Partition Map, which Ubuntu did not do in the last years. It could be replaced by a HFS+ tree like for grub-mkrescue.) Further an unpacked copy of the EFI partition filesystem's \BOOT\EFI should be present in the ISO as /BOOT/EFI. MS-Windows users seem to be used to copy the whole ISO file tree into a FAT partition and to expect that it then boots by EFI. (The scandal is that this really does work.) Chris Murphy wrote: > I don't know the history of why we left it as isolinux for BIOS though. Given above woes of Ubuntu, i'd say it was prudent conservatism. The switch needed nerves and some help from bystanders. Have a nice day :) Thomas ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Fri, Oct 23, 2020 at 09:56:11AM -0600, Chris Murphy wrote: > On Fri, Oct 23, 2020 at 8:18 AM Chris Adams wrote: > > > > Once upon a time, Chris Murphy said: > > > Although: now that I look at the ISO structure with isoinfo, it does > > appear that the files are not actually duplicated - they are pointing to > > the same sectors. Reading the genisoimage man page, it does recognize > > hard links and make only one copy on the resulting ISO image. Should > > have done more research on that part first - d'oh! > > That's curious, when I mount the server dvd ISO, those files have > different inode numbers. Sorry for the late reply on this, yet, they are hardlinked. I don't know if the mounted representation makes that clear, but lorax specifically hardlinks them in x86.tmpl Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On 2020-10-23 17:56, Chris Murphy wrote: That's curious, when I mount the server dvd ISO, those files have different inode numbers. I can't tell from the isoinfo man page what the numbers in brackets mean. Even if inodes are different, data blocks can be reused. The filesystem is readonly so there is no COW complication. I've tried to have a dir with 9 hardinked copies of a 9MB file. A simple mkisofs -o ../test.iso gave me a 10MB iso file (instead of >80MB). Regards. -- Roberto Ragusamail at robertoragusa.it ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Fri, Oct 23, 2020 at 8:33 AM Neal Gompa wrote: > > On Fri, Oct 23, 2020 at 1:39 AM Chris Murphy wrote: > > I'm not sure it's possible (still) for UEFI and BIOS GRUB to co-exist > > on one media, hence using isolinux to boot BIOS. > Could we just drop syslinux/isolinux and use GRUB there? Other > distributions have done that because of how much of a pain syslinux > is. And it would make the ISO setup a lot less special... See above. Not sure. :P I'm guessing it must be possible for the two GRUBs to co-exist if other distros produce a single image that can boot UEFI and BIOS. I don't know the history of why we left it as isolinux for BIOS though. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Fri, Oct 23, 2020 at 8:18 AM Chris Adams wrote: > > Once upon a time, Chris Murphy said: > Although: now that I look at the ISO structure with isoinfo, it does > appear that the files are not actually duplicated - they are pointing to > the same sectors. Reading the genisoimage man page, it does recognize > hard links and make only one copy on the resulting ISO image. Should > have done more research on that part first - d'oh! That's curious, when I mount the server dvd ISO, those files have different inode numbers. I can't tell from the isoinfo man page what the numbers in brackets mean. Directory listing of /images/pxeboot/ d- 0002048 Oct 19 2020 [ 34 02] . d- 0002048 Oct 19 2020 [ 33 02] .. -- 000 441 Oct 19 2020 [ 306435 00] TRANS.TBL -- 00078845696 Oct 19 2020 [ 306436 00] initrd.img -- 00011698160 Oct 15 2020 [ 344935 00] vmlinuz And it looks like FIEMAP doesn't work on ISO 9660. I get nothing useful from filefrag -v. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Fri, Oct 23, 2020 at 1:39 AM Chris Murphy wrote: > > On Thu, Oct 22, 2020 at 5:35 PM Chris Adams wrote: > > > > Once upon a time, Brian C. Lane said: > > > I tried :) It ends up that there are users who boot the install.img with > > > fips=1 which means it needs to be present. It was removed for a bit, and > > > then I reverted the removal: > > > > > > https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c212c89 > > > > Ah. Well, it was worth a try. Since /boot and /usr/lib/modules are on > > the same filesystem (in the particular case of the installer rootfs), > > could they at least be hardlinked? Although theoretically compressing > > it should save that space I guess. > > Pretty sure mksquashfs automatically dedups files by hardinking them > but only if we remove the nested ext4 file system. A so-called "plain > squashfs" image. That's a feature proposal for Fedora 34. > > > And there's still the ISO duplication between images/pxelinux and > > isolinux vmlinuz+initrd.img. I think the initrd.img might have been > > different at one point, but it seems like that was a very long time ago. > > GRUB and U-BOOT can directly read squashfs. If only syslinux/isolinux > could do that, everything could be on the squashfs file and deduped. > I'm not sure it's possible (still) for UEFI and BIOS GRUB to co-exist > on one media, hence using isolinux to boot BIOS. > > ISO 9660 doesn't support hardlinks or symlinks. UDF supports both. But > I think xorriso doesn't support UDF and also I'm not sure isolinux can > boot UDF either. GRUB can. > Could we just drop syslinux/isolinux and use GRUB there? Other distributions have done that because of how much of a pain syslinux is. And it would make the ISO setup a lot less special... -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Fri, Oct 23, 2020 at 09:17:52AM -0500, Chris Adams wrote:
> Well, there is value in having the vmlinuz/initrd.img directly available
> on the ISO (for people setting up PXE boot for example). I don't see
> any reason for there to be an images/pxelinux directory though - you can
> just get the same two files from the isolinux directory.
Note the .treeinfo files point to $ROOT/images/pxeboot/{vmlinuz,initrd.img}.
Tthese paths are also referenced by the libosinfo database, and used
for automating VM installs from virt-install/virt-manager/GNOME Boxes,
and probably more. IOW please be very wary of eliminating the
images/pxeboot directory as it could trigger a cascade of brokeness.
Regards,
Daniel
--
|: https://berrange.com -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Once upon a time, Chris Murphy said: > On Thu, Oct 22, 2020 at 5:35 PM Chris Adams wrote: > > Once upon a time, Brian C. Lane said: > > > I tried :) It ends up that there are users who boot the install.img with > > > fips=1 which means it needs to be present. It was removed for a bit, and > > > then I reverted the removal: > > > > > > https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c212c89 > > > > Ah. Well, it was worth a try. Since /boot and /usr/lib/modules are on > > the same filesystem (in the particular case of the installer rootfs), > > could they at least be hardlinked? Although theoretically compressing > > it should save that space I guess. > > Pretty sure mksquashfs automatically dedups files by hardinking them > but only if we remove the nested ext4 file system. A so-called "plain > squashfs" image. That's a feature proposal for Fedora 34. Nice. > > And there's still the ISO duplication between images/pxelinux and > > isolinux vmlinuz+initrd.img. I think the initrd.img might have been > > different at one point, but it seems like that was a very long time ago. > > GRUB and U-BOOT can directly read squashfs. If only syslinux/isolinux > could do that, everything could be on the squashfs file and deduped. > I'm not sure it's possible (still) for UEFI and BIOS GRUB to co-exist > on one media, hence using isolinux to boot BIOS. > > ISO 9660 doesn't support hardlinks or symlinks. UDF supports both. But > I think xorriso doesn't support UDF and also I'm not sure isolinux can > boot UDF either. GRUB can. Well, there is value in having the vmlinuz/initrd.img directly available on the ISO (for people setting up PXE boot for example). I don't see any reason for there to be an images/pxelinux directory though - you can just get the same two files from the isolinux directory. Although: now that I look at the ISO structure with isoinfo, it does appear that the files are not actually duplicated - they are pointing to the same sectors. Reading the genisoimage man page, it does recognize hard links and make only one copy on the resulting ISO image. Should have done more research on that part first - d'oh! -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Thu, Oct 22, 2020 at 5:35 PM Chris Adams wrote: > > Once upon a time, Brian C. Lane said: > > I tried :) It ends up that there are users who boot the install.img with > > fips=1 which means it needs to be present. It was removed for a bit, and > > then I reverted the removal: > > > > https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c212c89 > > Ah. Well, it was worth a try. Since /boot and /usr/lib/modules are on > the same filesystem (in the particular case of the installer rootfs), > could they at least be hardlinked? Although theoretically compressing > it should save that space I guess. Pretty sure mksquashfs automatically dedups files by hardinking them but only if we remove the nested ext4 file system. A so-called "plain squashfs" image. That's a feature proposal for Fedora 34. > And there's still the ISO duplication between images/pxelinux and > isolinux vmlinuz+initrd.img. I think the initrd.img might have been > different at one point, but it seems like that was a very long time ago. GRUB and U-BOOT can directly read squashfs. If only syslinux/isolinux could do that, everything could be on the squashfs file and deduped. I'm not sure it's possible (still) for UEFI and BIOS GRUB to co-exist on one media, hence using isolinux to boot BIOS. ISO 9660 doesn't support hardlinks or symlinks. UDF supports both. But I think xorriso doesn't support UDF and also I'm not sure isolinux can boot UDF either. GRUB can. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Once upon a time, Brian C. Lane said: > I tried :) It ends up that there are users who boot the install.img with > fips=1 which means it needs to be present. It was removed for a bit, and > then I reverted the removal: > > https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c212c89 Ah. Well, it was worth a try. Since /boot and /usr/lib/modules are on the same filesystem (in the particular case of the installer rootfs), could they at least be hardlinked? Although theoretically compressing it should save that space I guess. I still might play around with some overlay or bind-mount trickeration to remove duplication from the boot ISO and rootfs. Seems like that should still pass muster... but I know zilch about FIPS mode. And there's still the ISO duplication between images/pxelinux and isolinux vmlinuz+initrd.img. I think the initrd.img might have been different at one point, but it seems like that was a very long time ago. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Thu, Oct 22, 2020 at 12:56:26PM -0500, Chris Adams wrote: > Once upon a time, Martin Kolman said: > > I wonder if this can also help make the boot isos and installation images > > smaller ? I'm sure they haver some firmware on > > them. Or alternatively we could reduce the potentially quite meintenance > > intensive firmware culling we do at the moment. > > One thing I hadn't realized until recently is that the kernel, core > modules, and some firmware are duplicated on the install ISOs. > > First, on the ISO, vmlinuz and initrd.img are duplicated in the isolinux > and images/pxeboot directories. Or are these maybe hardlinked? I can't > remember if you can even do that on ISO. > > Beyond that, the kernel is on the ISO in isolinux/vmlinuz, and inside > the rootfs image in both /boot and /usr/lib/modules/. The core > modules and some firmware are in the ISO in initrd.img as well as in the > rootfs image. > > It'd be nice to remove some of that duplication - is there any reason to > have any vmlinuz inside the rootfs image? Could modules and firmware > copied to the initrd.img be removed as well (and then referenced from > the different mount point as needed in the installer, or maybe bind > mounted back where they came from if needed)? While this would make the > installer setup a little more complicated, it would save some space. I tried :) It ends up that there are users who boot the install.img with fips=1 which means it needs to be present. It was removed for a bit, and then I reverted the removal: https://github.com/weldr/lorax/commit/3c745aed8d535cae8430161192e0a1505c212c89 Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
Once upon a time, Martin Kolman said: > I wonder if this can also help make the boot isos and installation images > smaller ? I'm sure they haver some firmware on > them. Or alternatively we could reduce the potentially quite meintenance > intensive firmware culling we do at the moment. One thing I hadn't realized until recently is that the kernel, core modules, and some firmware are duplicated on the install ISOs. First, on the ISO, vmlinuz and initrd.img are duplicated in the isolinux and images/pxeboot directories. Or are these maybe hardlinked? I can't remember if you can even do that on ISO. Beyond that, the kernel is on the ISO in isolinux/vmlinuz, and inside the rootfs image in both /boot and /usr/lib/modules/. The core modules and some firmware are in the ISO in initrd.img as well as in the rootfs image. It'd be nice to remove some of that duplication - is there any reason to have any vmlinuz inside the rootfs image? Could modules and firmware copied to the initrd.img be removed as well (and then referenced from the different mount point as needed in the installer, or maybe bind mounted back where they came from if needed)? While this would make the installer setup a little more complicated, it would save some space. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Thu, Oct 22, 2020 at 02:46:47PM +0200, Martin Kolman wrote: > I wonder if this can also help make the boot isos and installation images > smaller ? I'm sure they haver some firmware on > them. Or alternatively we could reduce the potentially quite meintenance > intensive firmware culling we do at the moment. By itself, probably not since things are already compressed. But when I was doing some experiments a few months back I discovered that using the Bcj armthumb compression reduced the size of the squashfs image by about 400k -- I assumed it was because a bunch of the firmware is actually arm assembly but didn't look any deeper. So if it's possible to apply arch-specific compression to them it may reduce things slightly. Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Thu, 2020-10-22 at 14:57 +0100, Peter Robinson wrote: > On Thu, Oct 22, 2020 at 1:47 PM Martin Kolman wrote: > > On Tue, 2020-09-29 at 16:29 -0400, Ben Cotton wrote: > > > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware > > > > > > == Summary == > > > Compress Kernel Firmwares to reduce on disk size > > > > > > == Owner == > > > * Name: [[User:pbrobinson| Peter Robinson]] > > > * Email: [mailto:pbrobin...@fedoraproject.org| > > > pbrobin...@fedoraproject.org] > > > > > > == Detailed Description == > > > > > > Since the linux 5.3 kernel there has been support for loading firmware > > > from xz compressed firmware. The upstream linux-firmware respository > > > is now over 900Mb, not including other kernel firmware that are in > > > Fedora but come from other sources. By compessing the firmware with > > > "xz -C crc32", the only option currently supported in the kernel, we > > > can reduce the ondisk size of the firmware by almost half. > > I wonder if this can also help make the boot isos and installation images > > smaller ? I'm sure they haver some > > firmware on > > them. Or alternatively we could reduce the potentially quite meintenance > > intensive firmware culling we do at the > > moment. > > I wouldn't expect it to have too much of an image on the size of rpms > in the installer as the rpms are already compressed, this is more > about on disk install size. Hmm, right - the installation environment squashfs is already compressed so indeed, no gains likely expected. > > > > == Benefit to Fedora == > > > > > > Reduced on disk size of the firmware used by the kernel. > > > > > > == Scope == > > > * Proposal owners: > > > ** Add support upstream to the linux-firmware copy-firmware script to > > > compess the firmware and create the symlinks to the compressed > > > firmware > > > ** Enable the upstream support in the Fedora linux-firmware package to > > > compress the firmware at build time > > > ** Enable compressing firmware in packages that contain firmware used > > > by the kernel: eg alsa-sof-firmware, atmel-firmware, zd1211-firmware > > > ** Enable the CONFIG_FW_LOADER_COMPRESS kernel option (long complete) > > > * Other developers: > > > ** No impact > > > * Policies and guidelines: N/A (not a System Wide Change) > > > * Trademark approval: N/A (not needed for this Change) > > > > > > == Upgrade/compatibility impact == > > > > > > There should be no upgrade impact, the support was added to the > > > generic firmware loader interface in the kernel. > > > > > > == How To Test == > > > > > > Check devices that need firmware still work. Some devices that need > > > firmware include: > > > * GPU drivers such as nvidia, AMD and i915 > > > * Wireless drivers such as those from Intel, Broadcom/Cyprus, TI, > > > Qualcomm and Marvel > > > * Wired network interfaces > > > * Storage drivers > > > * USB controllers and drivers > > > > > > == User Experience == > > > > > > Generally users should notice little to no difference. > > > > > > == Dependencies == > > > N/A (not a System Wide Change) > > > > > > == Contingency Plan == > > > > > > * Contingency mechanism: Don't compress kernel firmware > > > * Contingency deadline: GA > > > * Blocks release? No. > > > * Blocks product? No. > > > > > > == Documentation == > > > N/A (not a System Wide Change) > > > > > > == Release Notes == > > > N/A > > > > > > > > > -- > > > Ben Cotton > > > He / Him / His > > > Senior Program Manager, Fedora & CentOS Stream > > > Red Hat > > > TZ=America/Indiana/Indianapolis > > > ___ > > > devel mailing list -- devel@lists.fedoraproject.org > > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > > Fedora Code of Conduct: > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > > List Archives: > > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > > ___ > > devel mailing list -- devel@lists.fedoraproject.org > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list --
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Thu, Oct 22, 2020 at 1:47 PM Martin Kolman wrote: > > On Tue, 2020-09-29 at 16:29 -0400, Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware > > > > == Summary == > > Compress Kernel Firmwares to reduce on disk size > > > > == Owner == > > * Name: [[User:pbrobinson| Peter Robinson]] > > * Email: [mailto:pbrobin...@fedoraproject.org| pbrobin...@fedoraproject.org] > > > > == Detailed Description == > > > > Since the linux 5.3 kernel there has been support for loading firmware > > from xz compressed firmware. The upstream linux-firmware respository > > is now over 900Mb, not including other kernel firmware that are in > > Fedora but come from other sources. By compessing the firmware with > > "xz -C crc32", the only option currently supported in the kernel, we > > can reduce the ondisk size of the firmware by almost half. > I wonder if this can also help make the boot isos and installation images > smaller ? I'm sure they haver some firmware on > them. Or alternatively we could reduce the potentially quite meintenance > intensive firmware culling we do at the moment. I wouldn't expect it to have too much of an image on the size of rpms in the installer as the rpms are already compressed, this is more about on disk install size. > > > > == Benefit to Fedora == > > > > Reduced on disk size of the firmware used by the kernel. > > > > == Scope == > > * Proposal owners: > > ** Add support upstream to the linux-firmware copy-firmware script to > > compess the firmware and create the symlinks to the compressed > > firmware > > ** Enable the upstream support in the Fedora linux-firmware package to > > compress the firmware at build time > > ** Enable compressing firmware in packages that contain firmware used > > by the kernel: eg alsa-sof-firmware, atmel-firmware, zd1211-firmware > > ** Enable the CONFIG_FW_LOADER_COMPRESS kernel option (long complete) > > * Other developers: > > ** No impact > > * Policies and guidelines: N/A (not a System Wide Change) > > * Trademark approval: N/A (not needed for this Change) > > > > == Upgrade/compatibility impact == > > > > There should be no upgrade impact, the support was added to the > > generic firmware loader interface in the kernel. > > > > == How To Test == > > > > Check devices that need firmware still work. Some devices that need > > firmware include: > > * GPU drivers such as nvidia, AMD and i915 > > * Wireless drivers such as those from Intel, Broadcom/Cyprus, TI, > > Qualcomm and Marvel > > * Wired network interfaces > > * Storage drivers > > * USB controllers and drivers > > > > == User Experience == > > > > Generally users should notice little to no difference. > > > > == Dependencies == > > N/A (not a System Wide Change) > > > > == Contingency Plan == > > > > * Contingency mechanism: Don't compress kernel firmware > > * Contingency deadline: GA > > * Blocks release? No. > > * Blocks product? No. > > > > == Documentation == > > N/A (not a System Wide Change) > > > > == Release Notes == > > N/A > > > > > > -- > > Ben Cotton > > He / Him / His > > Senior Program Manager, Fedora & CentOS Stream > > Red Hat > > TZ=America/Indiana/Indianapolis > > ___ > > devel mailing list -- devel@lists.fedoraproject.org > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Tue, 2020-09-29 at 16:29 -0400, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware > > == Summary == > Compress Kernel Firmwares to reduce on disk size > > == Owner == > * Name: [[User:pbrobinson| Peter Robinson]] > * Email: [mailto:pbrobin...@fedoraproject.org| pbrobin...@fedoraproject.org] > > == Detailed Description == > > Since the linux 5.3 kernel there has been support for loading firmware > from xz compressed firmware. The upstream linux-firmware respository > is now over 900Mb, not including other kernel firmware that are in > Fedora but come from other sources. By compessing the firmware with > "xz -C crc32", the only option currently supported in the kernel, we > can reduce the ondisk size of the firmware by almost half. I wonder if this can also help make the boot isos and installation images smaller ? I'm sure they haver some firmware on them. Or alternatively we could reduce the potentially quite meintenance intensive firmware culling we do at the moment. > > == Benefit to Fedora == > > Reduced on disk size of the firmware used by the kernel. > > == Scope == > * Proposal owners: > ** Add support upstream to the linux-firmware copy-firmware script to > compess the firmware and create the symlinks to the compressed > firmware > ** Enable the upstream support in the Fedora linux-firmware package to > compress the firmware at build time > ** Enable compressing firmware in packages that contain firmware used > by the kernel: eg alsa-sof-firmware, atmel-firmware, zd1211-firmware > ** Enable the CONFIG_FW_LOADER_COMPRESS kernel option (long complete) > * Other developers: > ** No impact > * Policies and guidelines: N/A (not a System Wide Change) > * Trademark approval: N/A (not needed for this Change) > > == Upgrade/compatibility impact == > > There should be no upgrade impact, the support was added to the > generic firmware loader interface in the kernel. > > == How To Test == > > Check devices that need firmware still work. Some devices that need > firmware include: > * GPU drivers such as nvidia, AMD and i915 > * Wireless drivers such as those from Intel, Broadcom/Cyprus, TI, > Qualcomm and Marvel > * Wired network interfaces > * Storage drivers > * USB controllers and drivers > > == User Experience == > > Generally users should notice little to no difference. > > == Dependencies == > N/A (not a System Wide Change) > > == Contingency Plan == > > * Contingency mechanism: Don't compress kernel firmware > * Contingency deadline: GA > * Blocks release? No. > * Blocks product? No. > > == Documentation == > N/A (not a System Wide Change) > > == Release Notes == > N/A > > > -- > Ben Cotton > He / Him / His > Senior Program Manager, Fedora & CentOS Stream > Red Hat > TZ=America/Indiana/Indianapolis > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Thu, Oct 22, 2020 at 7:53 AM Zbigniew Jędrzejewski-Szmek wrote: > > On Tue, Sep 29, 2020 at 09:46:14PM +0100, Peter Robinson wrote: > > On Tue, Sep 29, 2020 at 9:41 PM Neal Gompa wrote: > > > > > > On Tue, Sep 29, 2020 at 4:30 PM Ben Cotton wrote: > > > > > > > > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware > > > > > > > > == Summary == > > > > Compress Kernel Firmwares to reduce on disk size > > > > > > > > == Owner == > > > > * Name: [[User:pbrobinson| Peter Robinson]] > > > > * Email: [mailto:pbrobin...@fedoraproject.org| > > > > pbrobin...@fedoraproject.org] > > > > > > > > == Detailed Description == > > > > > > > > Since the linux 5.3 kernel there has been support for loading firmware > > > > from xz compressed firmware. The upstream linux-firmware respository > > > > is now over 900Mb, not including other kernel firmware that are in > > > > Fedora but come from other sources. By compessing the firmware with > > > > "xz -C crc32", the only option currently supported in the kernel, we > > > > can reduce the ondisk size of the firmware by almost half. > > > > > > > > > > I vaguely recall that there was some effort to add zstd support to > > > compress kernel stuff. Could we consider using that for this instead > > > of xz? Or is that still only for kernel modules? > > > > ATM the only thing that's upstream is xz hence why that's the only > > thing mentioned. > > zstd support has been merged for 5.9: > https://kernelnewbies.org/Linux_5.9#Support_for_ZSTD_compressed_kernel.2C_ramdisk_and_initramfs. > Does this support also extend to firmware? Nope, it's unrelated. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Tue, Sep 29, 2020 at 09:46:14PM +0100, Peter Robinson wrote: > On Tue, Sep 29, 2020 at 9:41 PM Neal Gompa wrote: > > > > On Tue, Sep 29, 2020 at 4:30 PM Ben Cotton wrote: > > > > > > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware > > > > > > == Summary == > > > Compress Kernel Firmwares to reduce on disk size > > > > > > == Owner == > > > * Name: [[User:pbrobinson| Peter Robinson]] > > > * Email: [mailto:pbrobin...@fedoraproject.org| > > > pbrobin...@fedoraproject.org] > > > > > > == Detailed Description == > > > > > > Since the linux 5.3 kernel there has been support for loading firmware > > > from xz compressed firmware. The upstream linux-firmware respository > > > is now over 900Mb, not including other kernel firmware that are in > > > Fedora but come from other sources. By compessing the firmware with > > > "xz -C crc32", the only option currently supported in the kernel, we > > > can reduce the ondisk size of the firmware by almost half. > > > > > > > I vaguely recall that there was some effort to add zstd support to > > compress kernel stuff. Could we consider using that for this instead > > of xz? Or is that still only for kernel modules? > > ATM the only thing that's upstream is xz hence why that's the only > thing mentioned. zstd support has been merged for 5.9: https://kernelnewbies.org/Linux_5.9#Support_for_ZSTD_compressed_kernel.2C_ramdisk_and_initramfs. Does this support also extend to firmware? Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Tue, Sep 29, 2020 at 9:41 PM Neal Gompa wrote: > > On Tue, Sep 29, 2020 at 4:30 PM Ben Cotton wrote: > > > > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware > > > > == Summary == > > Compress Kernel Firmwares to reduce on disk size > > > > == Owner == > > * Name: [[User:pbrobinson| Peter Robinson]] > > * Email: [mailto:pbrobin...@fedoraproject.org| pbrobin...@fedoraproject.org] > > > > == Detailed Description == > > > > Since the linux 5.3 kernel there has been support for loading firmware > > from xz compressed firmware. The upstream linux-firmware respository > > is now over 900Mb, not including other kernel firmware that are in > > Fedora but come from other sources. By compessing the firmware with > > "xz -C crc32", the only option currently supported in the kernel, we > > can reduce the ondisk size of the firmware by almost half. > > > > I vaguely recall that there was some effort to add zstd support to > compress kernel stuff. Could we consider using that for this instead > of xz? Or is that still only for kernel modules? ATM the only thing that's upstream is xz hence why that's the only thing mentioned. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
On Tue, Sep 29, 2020 at 4:30 PM Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/CompressKernelFirmware > > == Summary == > Compress Kernel Firmwares to reduce on disk size > > == Owner == > * Name: [[User:pbrobinson| Peter Robinson]] > * Email: [mailto:pbrobin...@fedoraproject.org| pbrobin...@fedoraproject.org] > > == Detailed Description == > > Since the linux 5.3 kernel there has been support for loading firmware > from xz compressed firmware. The upstream linux-firmware respository > is now over 900Mb, not including other kernel firmware that are in > Fedora but come from other sources. By compessing the firmware with > "xz -C crc32", the only option currently supported in the kernel, we > can reduce the ondisk size of the firmware by almost half. > I vaguely recall that there was some effort to add zstd support to compress kernel stuff. Could we consider using that for this instead of xz? Or is that still only for kernel modules? -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
https://fedoraproject.org/wiki/Changes/CompressKernelFirmware == Summary == Compress Kernel Firmwares to reduce on disk size == Owner == * Name: [[User:pbrobinson| Peter Robinson]] * Email: [mailto:pbrobin...@fedoraproject.org| pbrobin...@fedoraproject.org] == Detailed Description == Since the linux 5.3 kernel there has been support for loading firmware from xz compressed firmware. The upstream linux-firmware respository is now over 900Mb, not including other kernel firmware that are in Fedora but come from other sources. By compessing the firmware with "xz -C crc32", the only option currently supported in the kernel, we can reduce the ondisk size of the firmware by almost half. == Benefit to Fedora == Reduced on disk size of the firmware used by the kernel. == Scope == * Proposal owners: ** Add support upstream to the linux-firmware copy-firmware script to compess the firmware and create the symlinks to the compressed firmware ** Enable the upstream support in the Fedora linux-firmware package to compress the firmware at build time ** Enable compressing firmware in packages that contain firmware used by the kernel: eg alsa-sof-firmware, atmel-firmware, zd1211-firmware ** Enable the CONFIG_FW_LOADER_COMPRESS kernel option (long complete) * Other developers: ** No impact * Policies and guidelines: N/A (not a System Wide Change) * Trademark approval: N/A (not needed for this Change) == Upgrade/compatibility impact == There should be no upgrade impact, the support was added to the generic firmware loader interface in the kernel. == How To Test == Check devices that need firmware still work. Some devices that need firmware include: * GPU drivers such as nvidia, AMD and i915 * Wireless drivers such as those from Intel, Broadcom/Cyprus, TI, Qualcomm and Marvel * Wired network interfaces * Storage drivers * USB controllers and drivers == User Experience == Generally users should notice little to no difference. == Dependencies == N/A (not a System Wide Change) == Contingency Plan == * Contingency mechanism: Don't compress kernel firmware * Contingency deadline: GA * Blocks release? No. * Blocks product? No. == Documentation == N/A (not a System Wide Change) == Release Notes == N/A -- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis ___ devel-announce mailing list -- devel-announce@lists.fedoraproject.org To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org
F34 Change proposal: Compress Kernel Firmware (Self-Contained Change)
https://fedoraproject.org/wiki/Changes/CompressKernelFirmware == Summary == Compress Kernel Firmwares to reduce on disk size == Owner == * Name: [[User:pbrobinson| Peter Robinson]] * Email: [mailto:pbrobin...@fedoraproject.org| pbrobin...@fedoraproject.org] == Detailed Description == Since the linux 5.3 kernel there has been support for loading firmware from xz compressed firmware. The upstream linux-firmware respository is now over 900Mb, not including other kernel firmware that are in Fedora but come from other sources. By compessing the firmware with "xz -C crc32", the only option currently supported in the kernel, we can reduce the ondisk size of the firmware by almost half. == Benefit to Fedora == Reduced on disk size of the firmware used by the kernel. == Scope == * Proposal owners: ** Add support upstream to the linux-firmware copy-firmware script to compess the firmware and create the symlinks to the compressed firmware ** Enable the upstream support in the Fedora linux-firmware package to compress the firmware at build time ** Enable compressing firmware in packages that contain firmware used by the kernel: eg alsa-sof-firmware, atmel-firmware, zd1211-firmware ** Enable the CONFIG_FW_LOADER_COMPRESS kernel option (long complete) * Other developers: ** No impact * Policies and guidelines: N/A (not a System Wide Change) * Trademark approval: N/A (not needed for this Change) == Upgrade/compatibility impact == There should be no upgrade impact, the support was added to the generic firmware loader interface in the kernel. == How To Test == Check devices that need firmware still work. Some devices that need firmware include: * GPU drivers such as nvidia, AMD and i915 * Wireless drivers such as those from Intel, Broadcom/Cyprus, TI, Qualcomm and Marvel * Wired network interfaces * Storage drivers * USB controllers and drivers == User Experience == Generally users should notice little to no difference. == Dependencies == N/A (not a System Wide Change) == Contingency Plan == * Contingency mechanism: Don't compress kernel firmware * Contingency deadline: GA * Blocks release? No. * Blocks product? No. == Documentation == N/A (not a System Wide Change) == Release Notes == N/A -- Ben Cotton He / Him / His Senior Program Manager, Fedora & CentOS Stream Red Hat TZ=America/Indiana/Indianapolis ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
