Re: Small rant: installer environment size
On Tue, Feb 14, 2023 at 06:40:05PM +0100, Florian Weimer wrote: > > I'm still curious about the impact of glibc-all-langpacks on > download/image size. FWIW here is the discussion that happened when that was added: https://bugzilla.redhat.com/show_bug.cgi?id=1312607 Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
* Adam Williamson: > On Tue, 2023-02-14 at 09:40 +0100, Florian Weimer wrote: >> * Adam Williamson: >> >> > On Thu, 2022-12-08 at 07:57 +0100, Florian Weimer wrote: >> > > * Adam Williamson: >> > > >> > > > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is >> > > > 224M uncompressed. A quick test just compressing the file with xz on my >> > > > system shows it compresses to around 11M, though, so that's probably >> > > > all it adds up to after compression (the image is an xz-compressed >> > > > squashfs) >> > > >> > > Isn't the compression block-based? I think it would be interesting to >> > > measure the image size with the file removed. >> > >> > I'll try it tomorrow, it's not too hard. >> >> Have you posted the outcome of the experiment somewhere? > > Sorry, after we established I was wrong about the image being loaded > into memory, this became less important so I don't think I went ahead > with it. If you're still interested I can try and find a minute to do > it this week, once we have Rawhide and F38 calmed down after > branching... I'm still curious about the impact of glibc-all-langpacks on download/image size. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Tue, 2023-02-14 at 09:40 +0100, Florian Weimer wrote: > * Adam Williamson: > > > On Thu, 2022-12-08 at 07:57 +0100, Florian Weimer wrote: > > > * Adam Williamson: > > > > > > > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is > > > > 224M uncompressed. A quick test just compressing the file with xz on my > > > > system shows it compresses to around 11M, though, so that's probably > > > > all it adds up to after compression (the image is an xz-compressed > > > > squashfs) > > > > > > Isn't the compression block-based? I think it would be interesting to > > > measure the image size with the file removed. > > > > I'll try it tomorrow, it's not too hard. > > Have you posted the outcome of the experiment somewhere? Sorry, after we established I was wrong about the image being loaded into memory, this became less important so I don't think I went ahead with it. If you're still interested I can try and find a minute to do it this week, once we have Rawhide and F38 calmed down after branching... -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
* Adam Williamson: > On Thu, 2022-12-08 at 07:57 +0100, Florian Weimer wrote: >> * Adam Williamson: >> >> > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is >> > 224M uncompressed. A quick test just compressing the file with xz on my >> > system shows it compresses to around 11M, though, so that's probably >> > all it adds up to after compression (the image is an xz-compressed >> > squashfs) >> >> Isn't the compression block-based? I think it would be interesting to >> measure the image size with the file removed. > > I'll try it tomorrow, it's not too hard. Have you posted the outcome of the experiment somewhere? Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Mon, 2022-12-12 at 13:05 +0100, Vitaly Zaitsev via devel wrote: > On 12/12/2022 11:48, Florian Weimer wrote: > > The way I read it, Adam is removing firmware files that current Fedora > > kernels won't load, ever. > > Are you sure that that firmware is not needed at all? Yes. I've double-checked that with Intel's firmware maintainers. > > If Intel ships all these blobs in linux-firmware, then they have a good > reason for this, don't they? Not really. Just nobody ever suggested removing them before, I don't think, and they weren't causing Intel any problems. Unused stuff left lying around for years is hardly unusual in software projects... -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
* Vitaly Zaitsev via devel: > On 12/12/2022 13:35, Frantisek Zatloukal wrote: >> So newer linux-firmware can support older kernels, which isn't >> relevant for Fedora (to a degree that might be relevant upstream). > > Thanks for the info. I thought they were used by older hardware revisions. > > Sorry for the noise. No, the kernel decrements the version number (usually before the .ucode suffix) until there's something it can load. When I looked at this, I didn't see any logic like “start at this older version for these devices”. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On 12/12/2022 13:35, Frantisek Zatloukal wrote: So newer linux-firmware can support older kernels, which isn't relevant for Fedora (to a degree that might be relevant upstream). Thanks for the info. I thought they were used by older hardware revisions. Sorry for the noise. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Mon, Dec 12, 2022 at 1:25 PM Vitaly Zaitsev via devel < devel@lists.fedoraproject.org> wrote: > If Intel ships all these blobs in linux-firmware, then they have a good > reason for this, don't they? > So newer linux-firmware can support older kernels, which isn't relevant for Fedora (to a degree that might be relevant upstream). -- Best regards / S pozdravem, František Zatloukal Senior Quality Engineer Red Hat ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On 12/12/2022 11:48, Florian Weimer wrote: The way I read it, Adam is removing firmware files that current Fedora kernels won't load, ever. Are you sure that that firmware is not needed at all? If Intel ships all these blobs in linux-firmware, then they have a good reason for this, don't they? -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
* Vitaly Zaitsev via devel: > On 11/12/2022 20:53, Adam Williamson wrote: >> There's a PR at >> https://src.fedoraproject.org/rpms/linux-firmware/pull-request/9 , and >> Intel folks seem receptive to the idea of removing at least some of the >> older ones upstream too. > > So, you want the still-working Wi-Fi chipsets to stop working? > Terrible change. Strongly -1. > > 72XX, 82XX and 9XXX are still used even in new budget-class laptops. The way I read it, Adam is removing firmware files that current Fedora kernels won't load, ever. For example, we currently have these files in the package: /usr/lib/firmware/iwlwifi-7265D-22.ucode.xz /usr/lib/firmware/iwlwifi-7265D-27.ucode.xz /usr/lib/firmware/iwlwifi-7265D-29.ucode.xz And Adam is only deleting the two older ones. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Monday, December 12, 2022, Vitaly Zaitsev via devel < devel@lists.fedoraproject.org> wrote: > On 11/12/2022 20:53, Adam Williamson wrote: > >> There's a PR at >> https://src.fedoraproject.org/rpms/linux-firmware/pull-request/9 , and >> Intel folks seem receptive to the idea of removing at least some of the >> older ones upstream too. >> > > So, you want the still-working Wi-Fi chipsets to stop working? Terrible > change. Strongly -1. > > 72XX, 82XX and 9XXX are still used even in new budget-class laptops. > > There is a misunderstanding, it's not about older chips but about older firmware versions, required for older kernels. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On 11/12/2022 20:53, Adam Williamson wrote: There's a PR at https://src.fedoraproject.org/rpms/linux-firmware/pull-request/9 , and Intel folks seem receptive to the idea of removing at least some of the older ones upstream too. So, you want the still-working Wi-Fi chipsets to stop working? Terrible change. Strongly -1. 72XX, 82XX and 9XXX are still used even in new budget-class laptops. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Sun, 2022-12-11 at 20:23 +0100, Florian Weimer wrote: > * Kevin Kofler via devel: > > > Adam Williamson wrote: > > > Looking at sizes, iwlwifi firmware alone is 75M(!) ath10k is 6.8M, > > > ath11k is 12M, ath6k is 812K, so that's nearly another 20M. brcm/ is > > > another 6.4M and I *think* that's all wifi. There's a few other minor > > > ones, but that's a little over 100M of just wifi, with Intel by a huge > > > margin the worst offender. > > > > > > Does anyone know anyone we can talk to at Intel about this? It's pretty > > > obnoxious. > > > > It's the same situation as for some of the GPU firmwares: They have > > submitted as vendor driver to upstream, which does very little, together > > with a huge proprietary firmware, which does most of the real work. > > Hardware > > manufacturers love playing that trick. > > Uhm, part of the problem is that firmware versions which cannot be > loaded by the kernel are not removed upstream, or in the linux-firmware > package. The wifi firmware file I use is just 450 KiB compressed, but > that turns into 6.8 MiB compressed due to this kind of near-duplication. > > Unfortunately, I don't see an easy way to discover which firmware files > the kernel can ever load. I'm working on this with Peter and the Intel firmware folks ATM. There's a PR at https://src.fedoraproject.org/rpms/linux-firmware/pull-request/9 , and Intel folks seem receptive to the idea of removing at least some of the older ones upstream too. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
* Kevin Kofler via devel: > Adam Williamson wrote: >> Looking at sizes, iwlwifi firmware alone is 75M(!) ath10k is 6.8M, >> ath11k is 12M, ath6k is 812K, so that's nearly another 20M. brcm/ is >> another 6.4M and I *think* that's all wifi. There's a few other minor >> ones, but that's a little over 100M of just wifi, with Intel by a huge >> margin the worst offender. >> >> Does anyone know anyone we can talk to at Intel about this? It's pretty >> obnoxious. > > It's the same situation as for some of the GPU firmwares: They have > submitted as vendor driver to upstream, which does very little, together > with a huge proprietary firmware, which does most of the real work. Hardware > manufacturers love playing that trick. Uhm, part of the problem is that firmware versions which cannot be loaded by the kernel are not removed upstream, or in the linux-firmware package. The wifi firmware file I use is just 450 KiB compressed, but that turns into 6.8 MiB compressed due to this kind of near-duplication. Unfortunately, I don't see an easy way to discover which firmware files the kernel can ever load. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Sun, 2022-12-11 at 13:31 +, Zbigniew Jędrzejewski-Szmek wrote: > > > > > > We had some discussions on this a few years ago, but this never went > > > anywhere. > > > (I did a quick search, but can't find the ticket now.) But the idea was > > > that > > > dnf would load "main" metadata by default, and then load "filepath" > > > metadata > > > if needed and restart the transaction. Our Packaging Guidelines forbid > > > filepath > > > dependencies (except for a small set of directories which are in the > > > "main" part), > > > so normal rpm transactions don't need the "filepath" metadata. It is only > > > needed for non-confirming rpms and when the user does something like > > > 'dnf install /some/strange/path'. > > > > AIUI this is already a part of dnf5. > > I would love to hear more about this. > > A quick test suggests that on F37 dnf5 at least downloads the same metadata > that dnf4: > > $ sudo dnf upgrade > Fedora 37 - x86_64 - Updates 46 kB/s | 12 kB 00:00 > Fedora 37 - x86_64 - Updates 1.3 MB/s | 2.4 MB 00:01 > ... > > $ sudo dnf5 upgrade > Fedora 37 - x86_64 - Updates100% | 44.4 KiB/s | 12.4 KiB | 00m00s > Fedora 37 - x86_64 - Updates100% | 724.6 KiB/s | 2.0 MiB | 00m03s > ... It's just my recollection from an earlier discussion on this list about dnf5. Possibly I misremembered, or it's a plan rather than already implemented? -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
> > > We *could* do something about repo metadata: only install the "main" > > > metadata, > > > and not the "filepath" metadata. This would reduce the metadata size by > > > ~80%. > > > It'd also have huge benefits for speed: on small dnf operations a > > > significant > > > portion of time is spent just loading metadata. > > > > > > We had some discussions on this a few years ago, but this never went > > > anywhere. > > > (I did a quick search, but can't find the ticket now.) But the idea was > > > that > > > dnf would load "main" metadata by default, and then load "filepath" > > > metadata > > > if needed and restart the transaction. Our Packaging Guidelines forbid > > > filepath > > > dependencies (except for a small set of directories which are in the > > > "main" part), > > > so normal rpm transactions don't need the "filepath" metadata. It is only > > > needed for non-confirming rpms and when the user does something like > > > 'dnf install /some/strange/path'. > > > > AIUI this is already a part of dnf5. > > I would love to hear more about this. The original yum used to do this ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Sat, Dec 10, 2022 at 08:48:56AM -0800, Adam Williamson wrote: > On Sat, 2022-12-10 at 11:59 +, Zbigniew Jędrzejewski-Szmek wrote: > > On Fri, Dec 09, 2022 at 03:08:19PM -0700, Chris Murphy wrote: > > > > > > > > > On Fri, Dec 9, 2022, at 7:30 AM, Ray Strode wrote: > > > > Hi, > > > > > > > > On Thu, Dec 8, 2022 at 2:55 PM Adam Williamson > > > > wrote: > > > > > This is the direction Daniel was thinking down. I'm waiting for > > > > > someone > > > > > with more expertise to reply, but I suspect the reply is going to be > > > > > along the lines of "yes, we *can* do that, but it's somewhat tricky > > > > > work that involves thinking about lots of paths that aren't obvious, > > > > > and somebody would need to dedicate their time to working on that". > > > > Presumably we could package the firmware separately and just unpack it > > > > into place from a udev rule when the hardware is detected? > > > > > > > > But first, do we actually know this is a problem? > > > > I think you're saying squashfs loads the whole decompressed image into > > > > memory, but my expectation prior to your mail was that it performs I/O > > > > on the usb stick (with a cache in between). If my intuition was right > > > > and files only hit ram when accessed, then it seems like this is > > > > pretty much not an issue, right? > > > > > > From a certain point of view there's a potential inefficiency with > > > squashfs reads in that there's a minimum block size that it needs to read > > > in order decompress its 128 KiB block. It's possible quite a lot of > > > what's decompressed isn't (immediately) needed. But it's still a random > > > access file system. It's not necessary to read the whole image into RAM. > > > > > > Repo metadata is the big hit for netinstall because it's downloaded into > > > /tmp which is tmpfs. And DVD already has repomd on it, and only downloads > > > more if you enable some other repo. Live doesn't need repomd. > > > > > > So initially netinstaller uses less memory up until the the Anaconda > > > language selection screen appears, at which point it starts background > > > downloading repomd. It quickly catches up to, and surpasses, Live media > > > memory consumption. > > > > We *could* do something about repo metadata: only install the "main" > > metadata, > > and not the "filepath" metadata. This would reduce the metadata size by > > ~80%. > > It'd also have huge benefits for speed: on small dnf operations a > > significant > > portion of time is spent just loading metadata. > > > > We had some discussions on this a few years ago, but this never went > > anywhere. > > (I did a quick search, but can't find the ticket now.) But the idea was > > that > > dnf would load "main" metadata by default, and then load "filepath" metadata > > if needed and restart the transaction. Our Packaging Guidelines forbid > > filepath > > dependencies (except for a small set of directories which are in the "main" > > part), > > so normal rpm transactions don't need the "filepath" metadata. It is only > > needed for non-confirming rpms and when the user does something like > > 'dnf install /some/strange/path'. > > AIUI this is already a part of dnf5. I would love to hear more about this. A quick test suggests that on F37 dnf5 at least downloads the same metadata that dnf4: $ sudo dnf upgrade Fedora 37 - x86_64 - Updates 46 kB/s | 12 kB 00:00 Fedora 37 - x86_64 - Updates 1.3 MB/s | 2.4 MB 00:01 ... $ sudo dnf5 upgrade Fedora 37 - x86_64 - Updates100% | 44.4 KiB/s | 12.4 KiB | 00m00s Fedora 37 - x86_64 - Updates100% | 724.6 KiB/s | 2.0 MiB | 00m03s ... Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Adam Williamson wrote: > Looking at sizes, iwlwifi firmware alone is 75M(!) ath10k is 6.8M, > ath11k is 12M, ath6k is 812K, so that's nearly another 20M. brcm/ is > another 6.4M and I *think* that's all wifi. There's a few other minor > ones, but that's a little over 100M of just wifi, with Intel by a huge > margin the worst offender. > > Does anyone know anyone we can talk to at Intel about this? It's pretty > obnoxious. It's the same situation as for some of the GPU firmwares: They have submitted as vendor driver to upstream, which does very little, together with a huge proprietary firmware, which does most of the real work. Hardware manufacturers love playing that trick. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Chris Murphy wrote: > Net costs: Fedora releng takes one compression hit per image created, but > consumers of those images which also includes a ton of Fedora QA bot time > as well as human users are in the dozens to thousands of hits per image > created. But for those human users, a smaller image (thanks to stronger compression) means less time and energy, and for users on metered connections also less money, spent downloading the image. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Sat, 2022-12-10 at 11:59 +, Zbigniew Jędrzejewski-Szmek wrote: > On Fri, Dec 09, 2022 at 03:08:19PM -0700, Chris Murphy wrote: > > > > > > On Fri, Dec 9, 2022, at 7:30 AM, Ray Strode wrote: > > > Hi, > > > > > > On Thu, Dec 8, 2022 at 2:55 PM Adam Williamson > > > wrote: > > > > This is the direction Daniel was thinking down. I'm waiting for someone > > > > with more expertise to reply, but I suspect the reply is going to be > > > > along the lines of "yes, we *can* do that, but it's somewhat tricky > > > > work that involves thinking about lots of paths that aren't obvious, > > > > and somebody would need to dedicate their time to working on that". > > > Presumably we could package the firmware separately and just unpack it > > > into place from a udev rule when the hardware is detected? > > > > > > But first, do we actually know this is a problem? > > > I think you're saying squashfs loads the whole decompressed image into > > > memory, but my expectation prior to your mail was that it performs I/O > > > on the usb stick (with a cache in between). If my intuition was right > > > and files only hit ram when accessed, then it seems like this is > > > pretty much not an issue, right? > > > > From a certain point of view there's a potential inefficiency with squashfs > > reads in that there's a minimum block size that it needs to read in order > > decompress its 128 KiB block. It's possible quite a lot of what's > > decompressed isn't (immediately) needed. But it's still a random access > > file system. It's not necessary to read the whole image into RAM. > > > > Repo metadata is the big hit for netinstall because it's downloaded into > > /tmp which is tmpfs. And DVD already has repomd on it, and only downloads > > more if you enable some other repo. Live doesn't need repomd. > > > > So initially netinstaller uses less memory up until the the Anaconda > > language selection screen appears, at which point it starts background > > downloading repomd. It quickly catches up to, and surpasses, Live media > > memory consumption. > > We *could* do something about repo metadata: only install the "main" metadata, > and not the "filepath" metadata. This would reduce the metadata size by ~80%. > It'd also have huge benefits for speed: on small dnf operations a significant > portion of time is spent just loading metadata. > > We had some discussions on this a few years ago, but this never went anywhere. > (I did a quick search, but can't find the ticket now.) But the idea was that > dnf would load "main" metadata by default, and then load "filepath" metadata > if needed and restart the transaction. Our Packaging Guidelines forbid > filepath > dependencies (except for a small set of directories which are in the "main" > part), > so normal rpm transactions don't need the "filepath" metadata. It is only > needed for non-confirming rpms and when the user does something like > 'dnf install /some/strange/path'. AIUI this is already a part of dnf5. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Once upon a time, Zbigniew Jędrzejewski-Szmek said: > We *could* do something about repo metadata: only install the "main" metadata, > and not the "filepath" metadata. This would reduce the metadata size by ~80%. > It'd also have huge benefits for speed: on small dnf operations a significant > portion of time is spent just loading metadata. [snip] > If we did the split, we'd gain nice speedup on typical dnf invocations, and > smaller memory usage, and things would be better in general. A hero who > groks dnf and the underlying libraries and can tackle this would be needed. Now doesn't seem like the time to dig into a significant change to the existing dnf, but maybe it could be a requested feature in dnf5? -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 09, 2022 at 03:08:19PM -0700, Chris Murphy wrote: > > > On Fri, Dec 9, 2022, at 7:30 AM, Ray Strode wrote: > > Hi, > > > > On Thu, Dec 8, 2022 at 2:55 PM Adam Williamson > > wrote: > >> This is the direction Daniel was thinking down. I'm waiting for someone > >> with more expertise to reply, but I suspect the reply is going to be > >> along the lines of "yes, we *can* do that, but it's somewhat tricky > >> work that involves thinking about lots of paths that aren't obvious, > >> and somebody would need to dedicate their time to working on that". > > Presumably we could package the firmware separately and just unpack it > > into place from a udev rule when the hardware is detected? > > > > But first, do we actually know this is a problem? > > I think you're saying squashfs loads the whole decompressed image into > > memory, but my expectation prior to your mail was that it performs I/O > > on the usb stick (with a cache in between). If my intuition was right > > and files only hit ram when accessed, then it seems like this is > > pretty much not an issue, right? > > From a certain point of view there's a potential inefficiency with squashfs > reads in that there's a minimum block size that it needs to read in order > decompress its 128 KiB block. It's possible quite a lot of what's > decompressed isn't (immediately) needed. But it's still a random access file > system. It's not necessary to read the whole image into RAM. > > Repo metadata is the big hit for netinstall because it's downloaded into /tmp > which is tmpfs. And DVD already has repomd on it, and only downloads more if > you enable some other repo. Live doesn't need repomd. > > So initially netinstaller uses less memory up until the the Anaconda language > selection screen appears, at which point it starts background downloading > repomd. It quickly catches up to, and surpasses, Live media memory > consumption. We *could* do something about repo metadata: only install the "main" metadata, and not the "filepath" metadata. This would reduce the metadata size by ~80%. It'd also have huge benefits for speed: on small dnf operations a significant portion of time is spent just loading metadata. We had some discussions on this a few years ago, but this never went anywhere. (I did a quick search, but can't find the ticket now.) But the idea was that dnf would load "main" metadata by default, and then load "filepath" metadata if needed and restart the transaction. Our Packaging Guidelines forbid filepath dependencies (except for a small set of directories which are in the "main" part), so normal rpm transactions don't need the "filepath" metadata. It is only needed for non-confirming rpms and when the user does something like 'dnf install /some/strange/path'. If we did the split, we'd gain nice speedup on typical dnf invocations, and smaller memory usage, and things would be better in general. A hero who groks dnf and the underlying libraries and can tackle this would be needed. Zbyszek ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Sat, 2022-12-10 at 00:20 +0100, Fabio Valentini wrote: > On Fri, Dec 9, 2022 at 11:43 PM Adam Williamson > wrote: > > > > So since this turns out to be less important than I thought (thanks bcl > > for the correction) I won't poke it much further than I have today, but > > following up on the above, I've done a couple of PRs, one to strip more > > stuff in lorax: > > https://github.com/weldr/lorax/pull/1291 > > and one to dump a chunk of older iwlwifi firmwares: > > https://src.fedoraproject.org/rpms/linux-firmware/pull-request/9 > > those combined would get us some breathing room for a while... > > I wonder, hasn't this discussion been side-tracked a little bit? > Looking at the bug report linked in your first post, a big part of the > recent size increase were *media codecs*, which I'm pretty sure are > definitely not needed during install (decoders for JPEG-XL and AVIF > images, or AV1 video, etc.). > Unless I'm reading things wrong, dropping these from whatever they are > pulled in by (GStreamer / WebKitGTK?) would make a much bigger > difference than dropping a few firmware files? It's all part of the same discussion to me. I've filed https://bugzilla.redhat.com/show_bug.cgi?id=2152229 now for the original problem. Dropping the new deps of webkitgtk in lorax runtime- cleanup is possible, but would be rather ugly and fragile (any time the deps *changed*, we wouldn't necessarily notice and update the stripping; we already don't update it enough and it's probably missing stuff). It's much more robust if the unnecessary deps can be avoided at source. Still, that change in webkitgtk caused the images to grow by ~15M. The lorax change I filed today saves ~11M and the iwlwifi change should save ~30M I think. So no, actually, fixing the new webkitgtk deps wouldn't make a "much bigger difference", though I'd still like for it to happen! -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On 12/8/22 07:31, Kevin Kofler via devel wrote: > Tomáš Popela wrote: >> firefox, because that's what the web based installer should/will use in >> the end > > 臘 A full-blown, 71 MB compressed (!) web browser just to show the UI for the > installer! > > IMHO, the web-based UI is a major mistake and should never be shipped in > Fedora. It is good as a prototype, but way too resource-heavy to be shipped > in production. We need a rewrite of the UI design in a desktop toolkit. (If > GTK is not suitable, how about Qt? ☺) I wholeheartedly agree. Not only because of resource consumption, but also because of the total disaster that is the NPM ecosystem. Do they really plan on building each and every Node module from the ACTUAL source code? What ‘npm install’ puts in node_modules is often NOT actual source code, but rather minified or transpiled in some way. Using a desktop toolkit would be far FAR better. -- Sincerely, Demi Marie Obenour (she/her/hers) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 9, 2022 at 11:43 PM Adam Williamson wrote: > > So since this turns out to be less important than I thought (thanks bcl > for the correction) I won't poke it much further than I have today, but > following up on the above, I've done a couple of PRs, one to strip more > stuff in lorax: > https://github.com/weldr/lorax/pull/1291 > and one to dump a chunk of older iwlwifi firmwares: > https://src.fedoraproject.org/rpms/linux-firmware/pull-request/9 > those combined would get us some breathing room for a while... I wonder, hasn't this discussion been side-tracked a little bit? Looking at the bug report linked in your first post, a big part of the recent size increase were *media codecs*, which I'm pretty sure are definitely not needed during install (decoders for JPEG-XL and AVIF images, or AV1 video, etc.). Unless I'm reading things wrong, dropping these from whatever they are pulled in by (GStreamer / WebKitGTK?) would make a much bigger difference than dropping a few firmware files? Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 09:48 -0800, Adam Williamson wrote: > On Fri, 2022-12-09 at 12:04 +0100, Florian Weimer wrote: > > * Richard W. M. Jones: > > > > > You only need network / wifi firmware blobs (although I'm sure they > > > are in themselves large) and then you can fetch anything else needed > > > for the hardware including graphics, right? > > > > I think you need graphics to set up wifi. > > Yeah, this is an awkward chicken-and-egg problem. Even if we assume > you're on a wired network, kernel modules generally - AIUI - try to > load the firmware once, on initial module load, and if they can't find > it, just give up, right? So we still have an ordering problem: how can > we delay the loading of modules that need firmware until the network is > up for us to be able to access the firmware files? > > Maybe I'm missing something that would help there, but it seems > tricky... > > Looking at sizes, iwlwifi firmware alone is 75M(!) ath10k is 6.8M, > ath11k is 12M, ath6k is 812K, so that's nearly another 20M. brcm/ is > another 6.4M and I *think* that's all wifi. There's a few other minor > ones, but that's a little over 100M of just wifi, with Intel by a huge > margin the worst offender. > > Does anyone know anyone we can talk to at Intel about this? It's pretty > obnoxious. > > In terms of what the other big space takers are in general: > > * amdgpu/ (AMD video cards) is ~20M > * intel/ (mainly Intel bluetooth) is ~15M [0] > * qed/ (some very high-end QLogic network cards) is ~10M [0] > * i915/ (Intel video firmware) is 8.4M > * mediatek/ is 7.7M [1] > * qcom/ is 7.3M > > Then it trails off from there. Just the wifi plus those 6 things are > around 170M, so the large majority of all the space taken. > > [0] No, we can't lose this - people install with Bluetooth > mice/keyboards > [1] For a quick win right now possibly we could assume nobody's going > to use one of those as the interface for a Fedora install and drop > that, not sure if it's a safe assumption > [2] We could possibly lose a bunch of this stuff, I'll look into it So since this turns out to be less important than I thought (thanks bcl for the correction) I won't poke it much further than I have today, but following up on the above, I've done a couple of PRs, one to strip more stuff in lorax: https://github.com/weldr/lorax/pull/1291 and one to dump a chunk of older iwlwifi firmwares: https://src.fedoraproject.org/rpms/linux-firmware/pull-request/9 those combined would get us some breathing room for a while... -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 14:45 -0700, Chris Murphy wrote: > > On Thu, Dec 8, 2022, at 12:08 PM, Adam Williamson wrote: > > > I mean, the modern systems that *need* GPU firmware generally seem to > > do pretty well with using native resolution and don't perform too > > badly, especially in the simple installer UI. When I test the fallback > > path on my bare metal test box on UEFI it uses the monitor's native > > resolution and performs fine (even, honestly, in GNOME), and that > > motherboard is nearly a decade old even. Don't know if this is the same > > for everyone, of course. > > > For what it's worth, this is back on the change list for Fedora 37. > > https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization I hope they've co-ordinated with Peter this time, it seems kinda rude to keep pushing this without involving the package maintainer who's already done quite a lot of work on splitting things up and so on. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 09, 2022 at 02:38:48PM -0600, Chris Adams wrote: > Once upon a time, Brian C. Lane said: > > On Thu, Dec 08, 2022 at 02:17:22PM -0600, Chris Adams wrote: > > > One other thing that I noticed a while back that takes up a chunk of > > > space is the kernel... it's included inside install.img (in two places > > > even, although I assume it's hardlinked?), even though it has to have > > > already been loaded before install.img can be read. > > > > What two places? On rawhide we have one on the iso under /images/pxeboot/ > > and another inside the install.img under /usr/lib/modules... > > There used to be two on the ISO with syslinux (but they were effectively > hardlinked, so that didn't matter), didn't realize that'd been reduced. > > There's also two inside install.img, /boot/vmlinux- and > /usr/lib/modules//vmlinuz. This is what I'm not sure if it's > linked, or if the squashfs compression makes it an effective wash, or > what; extracting the squashfs does not result in hardlinked files. I checked, it's not hardlinked. I'd hope that squashfs does the right thing and takes advantage of them being the same. Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 9, 2022, at 7:30 AM, Ray Strode wrote: > Hi, > > On Thu, Dec 8, 2022 at 2:55 PM Adam Williamson > wrote: >> This is the direction Daniel was thinking down. I'm waiting for someone >> with more expertise to reply, but I suspect the reply is going to be >> along the lines of "yes, we *can* do that, but it's somewhat tricky >> work that involves thinking about lots of paths that aren't obvious, >> and somebody would need to dedicate their time to working on that". > Presumably we could package the firmware separately and just unpack it > into place from a udev rule when the hardware is detected? > > But first, do we actually know this is a problem? > I think you're saying squashfs loads the whole decompressed image into > memory, but my expectation prior to your mail was that it performs I/O > on the usb stick (with a cache in between). If my intuition was right > and files only hit ram when accessed, then it seems like this is > pretty much not an issue, right? From a certain point of view there's a potential inefficiency with squashfs reads in that there's a minimum block size that it needs to read in order decompress its 128 KiB block. It's possible quite a lot of what's decompressed isn't (immediately) needed. But it's still a random access file system. It's not necessary to read the whole image into RAM. Repo metadata is the big hit for netinstall because it's downloaded into /tmp which is tmpfs. And DVD already has repomd on it, and only downloads more if you enable some other repo. Live doesn't need repomd. So initially netinstaller uses less memory up until the the Anaconda language selection screen appears, at which point it starts background downloading repomd. It quickly catches up to, and surpasses, Live media memory consumption. Off hand, I'm not sure what's producing all the anonymous pages during Live installation but it's a fairly linear increase as the installation progresses. Since it's an rsync based installation, I'm currently frownie facing pondering the cause of the anon page explosion. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022, at 12:08 PM, Adam Williamson wrote: > I mean, the modern systems that *need* GPU firmware generally seem to > do pretty well with using native resolution and don't perform too > badly, especially in the simple installer UI. When I test the fallback > path on my bare metal test box on UEFI it uses the monitor's native > resolution and performs fine (even, honestly, in GNOME), and that > motherboard is nearly a decade old even. Don't know if this is the same > for everyone, of course. For what it's worth, this is back on the change list for Fedora 37. https://fedoraproject.org/wiki/Changes/Linux_Firmware_Minimization -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022, at 10:28 AM, Adam Williamson wrote: > On Thu, 2022-12-08 at 17:10 +, Gary Buhrmaster wrote: >> On Thu, Dec 8, 2022 at 5:06 PM Adam Williamson >> wrote: >> >> > It already *is* compressed, which is why it doesn't get any smaller in >> > the compressed filesystem image, unlike the other things I mentioned. >> > Check for yourself - look under /lib/firmware and you'll see only >> > things ending in .xz. >> >> Right, but zstd *may* have a better compression >> ratio than .xz (that was at least one of the reasons >> given for the changes to support it). > > I actually spent half an hour looking into that yesterday as I got > diverted into the details of how the filesystem compression stuff > works, but all the references I found say xz consistently compresses > better than zstd. zstd's advantages over xz are in *performance* (time > taken to compress and decompress). So switching from xz to zstd seems > like it would make things bigger, not smaller. xz will use a larger computation cost upfront (compression) to achieve a higher compression ratio than zstd, i.e. you can always ask xz to spend more time to get a higher compression ratio and thus beat zstd on resulting file size. But zstd will always beat xz if you favor time to compress, and significantly beats xz on decompression time. Net costs: Fedora releng takes one compression hit per image created, but consumers of those images which also includes a ton of Fedora QA bot time as well as human users are in the dozens to thousands of hits per image created. The net energy cost is quite a lot higher using xz compared to zstd. Only focusing on image size is misleading. There's a very real energy hit of all this compression and decompression. I don't know how to weigh the costs and figure out a compromise, but totally ignoring one of the costs is probably incorrect. For all I know a balanced approach means using xz but at a lower compression ratio, but someone with round tuits and interest would need to look at it. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022, at 10:06 AM, Daniel P. Berrangé wrote: > Is there something that can be done to optimize the RAM usage, > in spite of the large installer env size ? What's wrong with the RAM usage now? We do semi-regularly run into issues with openQA VM's running out of memory. So far they're all been considered bugs when we hit them, and the change is reverted including a system change to bump tmpfs on /tmp quite a lot. But keeping the VMs at 2G has helped discover changes that in retrospect shouldn't have been changed. Ergo, even though it's a pain to regularly hit these bugs, I don't think there is a per se problem with the 2G RAM selection. When it's all working as expected, swap on zram is used rather significantly and works as expected. > If we're installing off DVD media, it shouldn't be required to > pull all of the content into RAM, since it can be fetched on > demand from the media. AFAIK this doesn't happen. Files are read in only on demand, not a monolithic read of everything and kept in cache somehow. > IOW, 99% of the firmware never need > leave the ISO, so shouldn't matter if firmware is GBs in size [1] > if we never load it off the media. Same for languages, only > the one we actually want to use should ever get into RAM. At first glance it might seem you can get memory savings by not having swap on zram enabled. But what happens is anonymous pages can't be compressed. And also they can't be dropped since they have no files backing them. The result is file pages get dropped in memory tight situations and we end up getting (file) page faults, and it's just super expensive. Yes you can read these pages back from files but it's extra costly because even if it's only a 4K read that's needed, it'll translate into potentially upwards of 1M reads: to find the 4K extent requires reading multiple 4K ext4 metadata blocks, which aren't necessarily colocated in a 128 KiB squashfs block, so we end up reading 1 to 10 of those, taking the ram and memory hit to decompress them to reveal their 4K content we need, dropping the rest. And then doing that every time there's a page fault. So I'd say it's probably asking for a performance hit that isn't really going to save much memory. On high latency devices like USB sticks, it's not a good UX. > If we're installing off a network source, we need to pull content > into RAM, but that doesn't mean we should pull everything in at > once upfront. Pretty sure netinstaller's big RAM hit is repo metadata. All of it is downloaded before we have partitioning done, thus the repo metadata isn't stored on disk, rather in memory and it's tmpfs so it may not be compressed either (at least it's not subject to swap on zram out of the gate). I'm pretty sure partitioning happens before the packages are downloaded though, which means they get stored on disk not in memory. But the repo metadata is pretty big now and that's a big memory hit for netinstallers. -- Chris Murphy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Once upon a time, Brian C. Lane said: > On Thu, Dec 08, 2022 at 02:17:22PM -0600, Chris Adams wrote: > > One other thing that I noticed a while back that takes up a chunk of > > space is the kernel... it's included inside install.img (in two places > > even, although I assume it's hardlinked?), even though it has to have > > already been loaded before install.img can be read. > > What two places? On rawhide we have one on the iso under /images/pxeboot/ > and another inside the install.img under /usr/lib/modules... There used to be two on the ISO with syslinux (but they were effectively hardlinked, so that didn't matter), didn't realize that'd been reduced. There's also two inside install.img, /boot/vmlinux- and /usr/lib/modules//vmlinuz. This is what I'm not sure if it's linked, or if the squashfs compression makes it an effective wash, or what; extracting the squashfs does not result in hardlinked files. I know that /boot on an installed system is typically separate, so hard links don't work when installing the kernel, but separate-/boot is not always the case. It'd be nice if the kernel install scripts tried to hard link where possible. > I think I tried removing the kernel from the install.img at one point, > but it ended up being required for FIPS (see > https://github.com/weldr/lorax/issues/1021). Ahh. I think I brought it up (on fedora-devel or maybe anaconda-devel) too, but forgot (and forgot the reason why). I agree that it seems silly that looking at a kernel file that was not used to boot is considered acceptable. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 12:18 -0800, Brian C. Lane wrote: > On Fri, Dec 09, 2022 at 09:30:29AM -0500, Ray Strode wrote: > > Hi, > > > > On Thu, Dec 8, 2022 at 2:55 PM Adam Williamson > > wrote: > > > This is the direction Daniel was thinking down. I'm waiting for someone > > > with more expertise to reply, but I suspect the reply is going to be > > > along the lines of "yes, we *can* do that, but it's somewhat tricky > > > work that involves thinking about lots of paths that aren't obvious, > > > and somebody would need to dedicate their time to working on that". > > Presumably we could package the firmware separately and just unpack it > > into place from a udev rule when the hardware is detected? > > > > But first, do we actually know this is a problem? > > I think you're saying squashfs loads the whole decompressed image into > > memory, but my expectation prior to your mail was that it performs I/O > > on the usb stick (with a cache in between). If my intuition was right > > and files only hit ram when accessed, then it seems like this is > > pretty much not an issue, right? > > > > Do you have stats on memory usage when running in a live environment? > > Your intuition is correct, if you boot from an ISO, USB, or NFS the > squashfs image is not read into memory. If you are PXE booting (without > using NFS for stage2) then it all goes into RAM. > > Loading firmware off the iso later isn't going to help things :) Thanks for the correction. So, image size and memory usage are only correlated in the specific case of a PXE install with no NFS? In that case, this is overall probably less important than I thought it was initially. Sorry for the error. In that case I guess we could be rather more relaxed about the size. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 09, 2022 at 09:30:29AM -0500, Ray Strode wrote: > Hi, > > On Thu, Dec 8, 2022 at 2:55 PM Adam Williamson > wrote: > > This is the direction Daniel was thinking down. I'm waiting for someone > > with more expertise to reply, but I suspect the reply is going to be > > along the lines of "yes, we *can* do that, but it's somewhat tricky > > work that involves thinking about lots of paths that aren't obvious, > > and somebody would need to dedicate their time to working on that". > Presumably we could package the firmware separately and just unpack it > into place from a udev rule when the hardware is detected? > > But first, do we actually know this is a problem? > I think you're saying squashfs loads the whole decompressed image into > memory, but my expectation prior to your mail was that it performs I/O > on the usb stick (with a cache in between). If my intuition was right > and files only hit ram when accessed, then it seems like this is > pretty much not an issue, right? > > Do you have stats on memory usage when running in a live environment? Your intuition is correct, if you boot from an ISO, USB, or NFS the squashfs image is not read into memory. If you are PXE booting (without using NFS for stage2) then it all goes into RAM. Loading firmware off the iso later isn't going to help things :) Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 08, 2022 at 02:17:22PM -0600, Chris Adams wrote: > Once upon a time, Adam Williamson said: > > This is the direction Daniel was thinking down. I'm waiting for someone > > with more expertise to reply, but I suspect the reply is going to be > > along the lines of "yes, we *can* do that, but it's somewhat tricky > > work that involves thinking about lots of paths that aren't obvious, > > and somebody would need to dedicate their time to working on that". > > One other thing that I noticed a while back that takes up a chunk of > space is the kernel... it's included inside install.img (in two places > even, although I assume it's hardlinked?), even though it has to have > already been loaded before install.img can be read. What two places? On rawhide we have one on the iso under /images/pxeboot/ and another inside the install.img under /usr/lib/modules... I think I tried removing the kernel from the install.img at one point, but it ended up being required for FIPS (see https://github.com/weldr/lorax/issues/1021). And when there were 2 kernels on the iso they were hardlinked (one under pxeboot and the other under isolinux). Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 20:33 +0100, drago01 wrote: > On Friday, December 9, 2022, Adam Williamson > wrote: > > > On Fri, 2022-12-09 at 12:04 +0100, Florian Weimer wrote: > > > * Richard W. M. Jones: > > > > > > > You only need network / wifi firmware blobs (although I'm sure they > > > > are in themselves large) and then you can fetch anything else needed > > > > for the hardware including graphics, right? > > > > > > I think you need graphics to set up wifi. > > > > Yeah, this is an awkward chicken-and-egg problem. Even if we assume > > you're on a wired network, kernel modules generally - AIUI - try to > > load the firmware once, on initial module load, and if they can't find > > it, just give up, right? So we still have an ordering problem: how can > > we delay the loading of modules that need firmware until the network is > > up for us to be able to access the firmware files? > > > > Maybe I'm missing something that would help there, but it seems > > tricky... > > > > Looking at sizes, iwlwifi firmware alone is 75M(!) ath10k is 6.8M, > > ath11k is 12M, ath6k is 812K, so that's nearly another 20M. brcm/ is > > another 6.4M and I *think* that's all wifi. There's a few other minor > > ones, but that's a little over 100M of just wifi, with Intel by a huge > > margin the worst offender. > > > > Does anyone know anyone we can talk to at Intel about this? It's pretty > > obnoxious. > > > > In terms of what the other big space takers are in general: > > > > * amdgpu/ (AMD video cards) is ~20M > > * intel/ (mainly Intel bluetooth) is ~15M [0] > > * qed/ (some very high-end QLogic network cards) is ~10M [0] > > * i915/ (Intel video firmware) is 8.4M > > * mediatek/ is 7.7M [1] > > * qcom/ is 7.3M > > > > Then it trails off from there. Just the wifi plus those 6 things are > > around 170M, so the large majority of all the space taken. > > > > [0] No, we can't lose this - people install with Bluetooth > > mice/keyboards > > [1] For a quick win right now possibly we could assume nobody's going > > to use one of those as the interface for a Fedora install and drop > > that, not sure if it's a safe assumption > > > > It's not given that AMD wifi is rebranded mediatek, meaning it will drop > wifi for lots of newer AMD laptops. Sorry, I messed up my numbering there. That note was meant for the qed/ directory, not mediatek/ . I've been working on this this morning. I'm pretty sure we can just drop every file but one in qed/ - it contains a lot of old versions that we don't need to care about any more. We can lose some stuff from mediatek/ - not any of the wifi stuff, but there's some firmware in there for ARM SoCs we do not even build the drivers for. I found a few other little cleanups, too. I *think* we can fairly safely drop about 31M of iwlwifi firmwares from linux-firmware, I'm testing a PR for that right now. We could potentially drop even more in lorax (since we don't really need to support booting the current installer with an older kernel - that's a constraint on dropping things from the linux-firmware package too soon, as it would be a bit mean to break things for people booting older kernels on installed systems for some reason). -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 09, 2022 at 03:49:08PM +0100, Miroslav Suchý wrote: > Dne 08. 12. 22 v 19:33 Adam Williamson napsal(a): > >> On that note, /usr/share/doc, /usr/share/man, and /usr/share/info could be > >> removed from the installer image if they are present. That likely won't > >> free > >> a whole lot of space, but it's not nothing. > > All of those are already stripped: > > https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-cleanup.tmpl#L69 > > Am I blind, or there is no removal of /var/cache/* ? /var/cache doesn't need to be cleaned up, it only has directories created by installing rpm packages. Remember, this is not the rootfs of a running system, it is built by installing a pile of rpms and then selectively removing bits and pieces of those rpms to try and trim the size. Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Friday, December 9, 2022, Adam Williamson wrote: > On Fri, 2022-12-09 at 12:04 +0100, Florian Weimer wrote: > > * Richard W. M. Jones: > > > > > You only need network / wifi firmware blobs (although I'm sure they > > > are in themselves large) and then you can fetch anything else needed > > > for the hardware including graphics, right? > > > > I think you need graphics to set up wifi. > > Yeah, this is an awkward chicken-and-egg problem. Even if we assume > you're on a wired network, kernel modules generally - AIUI - try to > load the firmware once, on initial module load, and if they can't find > it, just give up, right? So we still have an ordering problem: how can > we delay the loading of modules that need firmware until the network is > up for us to be able to access the firmware files? > > Maybe I'm missing something that would help there, but it seems > tricky... > > Looking at sizes, iwlwifi firmware alone is 75M(!) ath10k is 6.8M, > ath11k is 12M, ath6k is 812K, so that's nearly another 20M. brcm/ is > another 6.4M and I *think* that's all wifi. There's a few other minor > ones, but that's a little over 100M of just wifi, with Intel by a huge > margin the worst offender. > > Does anyone know anyone we can talk to at Intel about this? It's pretty > obnoxious. > > In terms of what the other big space takers are in general: > > * amdgpu/ (AMD video cards) is ~20M > * intel/ (mainly Intel bluetooth) is ~15M [0] > * qed/ (some very high-end QLogic network cards) is ~10M [0] > * i915/ (Intel video firmware) is 8.4M > * mediatek/ is 7.7M [1] > * qcom/ is 7.3M > > Then it trails off from there. Just the wifi plus those 6 things are > around 170M, so the large majority of all the space taken. > > [0] No, we can't lose this - people install with Bluetooth > mice/keyboards > [1] For a quick win right now possibly we could assume nobody's going > to use one of those as the interface for a Fedora install and drop > that, not sure if it's a safe assumption > It's not given that AMD wifi is rebranded mediatek, meaning it will drop wifi for lots of newer AMD laptops. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 12:04 +0100, Florian Weimer wrote: > * Richard W. M. Jones: > > > You only need network / wifi firmware blobs (although I'm sure they > > are in themselves large) and then you can fetch anything else needed > > for the hardware including graphics, right? > > I think you need graphics to set up wifi. Yeah, this is an awkward chicken-and-egg problem. Even if we assume you're on a wired network, kernel modules generally - AIUI - try to load the firmware once, on initial module load, and if they can't find it, just give up, right? So we still have an ordering problem: how can we delay the loading of modules that need firmware until the network is up for us to be able to access the firmware files? Maybe I'm missing something that would help there, but it seems tricky... Looking at sizes, iwlwifi firmware alone is 75M(!) ath10k is 6.8M, ath11k is 12M, ath6k is 812K, so that's nearly another 20M. brcm/ is another 6.4M and I *think* that's all wifi. There's a few other minor ones, but that's a little over 100M of just wifi, with Intel by a huge margin the worst offender. Does anyone know anyone we can talk to at Intel about this? It's pretty obnoxious. In terms of what the other big space takers are in general: * amdgpu/ (AMD video cards) is ~20M * intel/ (mainly Intel bluetooth) is ~15M [0] * qed/ (some very high-end QLogic network cards) is ~10M [0] * i915/ (Intel video firmware) is 8.4M * mediatek/ is 7.7M [1] * qcom/ is 7.3M Then it trails off from there. Just the wifi plus those 6 things are around 170M, so the large majority of all the space taken. [0] No, we can't lose this - people install with Bluetooth mice/keyboards [1] For a quick win right now possibly we could assume nobody's going to use one of those as the interface for a Fedora install and drop that, not sure if it's a safe assumption [2] We could possibly lose a bunch of this stuff, I'll look into it -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 15:44 +0100, mkol...@redhat.com wrote: > Or perhaps drop even the help system ? The help content has actually > not been updated in a while and does not really have an active > "upstream" - the Fedora docs moved on to a very different format, that > can no longer be used to produce the per-screen content the current > help system needs. > > So maybe dropping the help support-yelp-webkit & getting 40 MB back for > now could be worth it ? Ehhh, I dunno, it seems like such a dead-end thing to do when clearly all the effort is going into the webUI. It'd be nicer to find something that will last. But maybe if we really need the space for an upcoming release and can't find anything else to cut we could consider it. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 14:42 +0100, Miroslav Suchý wrote: > Dne 08. 12. 22 v 17:52 Adam Williamson napsal(a): > > > No, and we're looking at splitting those out, but the fact is they are > > > a tiny amount of the overall firmware collection. You could even argue > > > either way for something like bluetooth due to it sometimes being used > > > by keyboards. > > We actually already strip a lot of those in lorax. > ... > > then, in cleanup, we wipe a bunch of files that have not yet been split > > out from linux-firmware but which we know the installer env doesn't > > need: > > > > https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-cleanup.tmpl#L201 > > Thanks for the links. Putting aside the installer image... the firmware files > mentioned in last link are not so small: > > [msuchy@dri//tmp/linux-firmware-20221109-144.fc38.noarch]$ > du-hcusr/lib/firmware/dvb*usr/lib/firmware/*_12mhz*usr/lib/firmware/v4l*usr/lib/firmware/brcm/BCM-*usr/lib/firmware/ttusb- > budget/dspbootcode.bin*usr/lib/firmware/emi26/*usr/lib/firmware/emi62/*usr/lib/firmware/cpia2/*usr/lib/firmware/dabusb/*usr/lib/firmware/vicam/*usr/lib/firmware/dsp56k/*usr/lib/firmw > are/sun/*usr/lib/firmware/av7110/*usr/lib/firmware/usbdux*usr/lib/firmware/f2255usb.bin*usr/lib/firmware/lgs8g75.fw*usr/lib/firmware/tlg2300_firmware.bin*usr/lib/firmware/s5p-mfc*us > r/lib/firmware/go7007/*usr/lib/firmware/intel/IntcSST2.bin*usr/lib/firmware/intel/fw_sst*usr/lib/firmware/intel/dsp*usr/lib/firmware/as102*usr/lib/firmware/qcom/apq8096/*usr/lib/firmw > are/qcom/sdm845/*usr/lib/firmware/qcom/sm8250/*usr/lib/firmware/qcom/venus*/*usr/lib/firmware/qcom/vpu*/*usr/lib/firmware/meson/vdec/*usr/lib/firmware/phanfw.bin* > > > 28M total > > This is about 20 % of size of linux-firmware package. Can this be moved to to > separate subpackage? Peter is the domain expert on the process of carving off chunks of linux-firmware. It's a bit of a balancing act, because you kinda need a large enough, clearly thematically-related chunk to make it worthwhile. A lot of the things in that list aren't really related to each other and aren't big enough on their own to be worth splitting out, it's a whole compendium... -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 11:12 +, Peter Robinson wrote: > On Thu, Dec 8, 2022 at 4:56 PM Adam Williamson > wrote: > > > > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > > > I've done a few passes, dropping a bunch of older firmware upstream > > > that are no longer supported in any stable kernel release, also a > > > bunch of de-dupe and linking of files rather than shipping of multiple > > > copies of the same firmware. It's improved things a bit, unfortunately > > > a lot of the dead firmware was tiny compared to say average modern > > > devices like GPUs or WiFI. > > > > > > The problem with a lot of the firmware, and with the new nvidia "open > > > driver" which shoves a lot of stuff into firmware in order to have an > > > upstreamable driver apparently the firmwares there are going to be > > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > > even just install so I don't know how we can get around this and still > > > have a device work enough to be able to install the needed firmware > > > across the network. > > > > > > Ideas on how to solve that problem welcome. > > > > Sorry if this is way off, but - do we need the GPU firmwares to run a > > graphical install on the fallback path, just using the framebuffer set > > up by the firmware? How crazy would it be to just do that - ship the > > installer env with no GPU firmware? > > That has crossed my mind, and with simpledrm that may be more straight > forward now, but TBH it's not something I am skilled enough to deal > with, nor have the resources to test, or actually care enough about, > but the big GPU firmwares are now all split out so that should be much > more straightforward for someone with the resources to investigate. Heck, if people want to try it out, we can. I can re-run the openQA test (the old one's assets will have been garbage collected by now) and pull the ISO out and upload it somewhere, and everyone can see how it behaves on their system. maybe I'll do that later... -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 10:21 -0500, Neal Gompa wrote: > On Fri, Dec 9, 2022 at 10:17 AM wrote: > > > > On Thu, 2022-12-08 at 15:42 +, Gary Buhrmaster wrote: > > > On Thu, Dec 8, 2022 at 12:31 PM Kevin Kofler via devel > > > wrote: > > > > > > > IMHO, the web-based UI is a major mistake and should never be > > > > shipped in > > > > Fedora. > > > > > > As I am sure you are aware, it seems a number of distros > > > are experimenting with their next gen installer. OpenSUSE > > > has their new web based D-installer, > > The OpenSUSE effort is very close on the library/tooling level - > > they > > also use PatternFly/React/Cockkpit to build their Web UI, pretty > > much > > like we do for the Anaconda Web UI. > > > > > and Canonical is > > > writing an installer in flutter. > > > > > > While I have not personally tried them out, so don't have > > > any real opinion in whether they are good approaches, > > > maybe eventually some of the good ideas from all the > > > distros next generations of installer will eventually cross > > > pollinate even though we likely all agree that there will > > > not be one installer to rule them all. > > Agreed! I would hope it could help with say extending PatternFly > > widgets or some Cockpit tooling to better cover some installation > > specific use cases, more people looking into library/tooling bugs > > affecting installers, etc. > > > > What I'd like to see out of this effort is some API documentation of > how the frontend (web UI) and backend (privileged services) interact > so that other frontends for Anaconda could be developed in the > future. > Eliminating desktop frontends entirely is a painful regression. :( Actually the current GTK3 GUI and even the TUI do talk to the backend over DBus. They do talk to the backend directly in a few remaining places (IIRC related to the payload handling code) but this should be also all moved to DBus in the near future. As for reference documentation of the DBus API - that's indeed on our TODO list, and definitely necessary, not just for any alternative UI efforts but also for Anaconda addon authors. > > And this new model would let us have the installer UI operate > unprivileged and use the appropriate mechanisms for privilege > escalation like any other desktop app. We are I would say already half way there, with most of the priviledged actions already taking place in the backend providing a DBus API, which the GUI still running under root mostly due to the few remaining bits of direct interaction and inertia. And some specifics of keyboard layout configuration that Jiri Konecny knows more about than me. Definitely with the Web UI the Web view app does not need to run as root. > > > > > -- > 真実はいつも一つ！/ Always, there's only one truth! > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, 2022-12-09 at 11:15 +, Richard W.M. Jones wrote: > On Fri, Dec 09, 2022 at 12:04:24PM +0100, Florian Weimer wrote: > > * Richard W. M. Jones: > > > > > You only need network / wifi firmware blobs (although I'm sure > > > they > > > are in themselves large) and then you can fetch anything else > > > needed > > > for the hardware including graphics, right? > > > > I think you need graphics to set up wifi. > > I long for old school text mode installers ... At least you knew > that the tab key would always work. Well, if you pass int.text on the boot command line, Anaconda will show you the TUI - which supports a sizeable sub-set of the GUI functionality. :) > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat > http://people.redhat.com/~rjones > Read my programming and virtualization blog: > http://rwmj.wordpress.com > virt-builder quickly builds VMs from scratch > http://libguestfs.org/virt-builder.1.html > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 9, 2022 at 10:17 AM wrote: > > On Thu, 2022-12-08 at 15:42 +, Gary Buhrmaster wrote: > > On Thu, Dec 8, 2022 at 12:31 PM Kevin Kofler via devel > > wrote: > > > > > IMHO, the web-based UI is a major mistake and should never be > > > shipped in > > > Fedora. > > > > As I am sure you are aware, it seems a number of distros > > are experimenting with their next gen installer. OpenSUSE > > has their new web based D-installer, > The OpenSUSE effort is very close on the library/tooling level - they > also use PatternFly/React/Cockkpit to build their Web UI, pretty much > like we do for the Anaconda Web UI. > > > and Canonical is > > writing an installer in flutter. > > > > While I have not personally tried them out, so don't have > > any real opinion in whether they are good approaches, > > maybe eventually some of the good ideas from all the > > distros next generations of installer will eventually cross > > pollinate even though we likely all agree that there will > > not be one installer to rule them all. > Agreed! I would hope it could help with say extending PatternFly > widgets or some Cockpit tooling to better cover some installation > specific use cases, more people looking into library/tooling bugs > affecting installers, etc. > What I'd like to see out of this effort is some API documentation of how the frontend (web UI) and backend (privileged services) interact so that other frontends for Anaconda could be developed in the future. Eliminating desktop frontends entirely is a painful regression. :( And this new model would let us have the installer UI operate unprivileged and use the appropriate mechanisms for privilege escalation like any other desktop app. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 15:42 +, Gary Buhrmaster wrote: > On Thu, Dec 8, 2022 at 12:31 PM Kevin Kofler via devel > wrote: > > > IMHO, the web-based UI is a major mistake and should never be > > shipped in > > Fedora. > > As I am sure you are aware, it seems a number of distros > are experimenting with their next gen installer. OpenSUSE > has their new web based D-installer, The OpenSUSE effort is very close on the library/tooling level - they also use PatternFly/React/Cockkpit to build their Web UI, pretty much like we do for the Anaconda Web UI. > and Canonical is > writing an installer in flutter. > > While I have not personally tried them out, so don't have > any real opinion in whether they are good approaches, > maybe eventually some of the good ideas from all the > distros next generations of installer will eventually cross > pollinate even though we likely all agree that there will > not be one installer to rule them all. Agreed! I would hope it could help with say extending PatternFly widgets or some Cockpit tooling to better cover some installation specific use cases, more people looking into library/tooling bugs affecting installers, etc. > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 08:53 +0100, Tomáš Popela wrote: > Hi Adam, > > On Thu, Dec 8, 2022 at 5:30 AM Adam Williamson > wrote: > > On Thu, 2022-12-08 at 03:28 +, Gary Buhrmaster wrote: > > > On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson > > > wrote: > > > > > > > > Hi folks! Today I woke up and found > > > > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which > > diverted me > > > > down a bit of an "installer environment size" rabbit hole. > > > > > > Does the "new and improved" web based installer help this > > > in any way? > > > > I haven't looked yet but I suspect it'll probably be a wash, > > mostly. If > > anything it's likely slightly negative because the new installer > > itself > > hard requires webkitgtk, so we can't really do anything to finesse > > that > > requirement any more. With the old installer, we could maybe try > > and > > figure out some way of being able to show the help pages without > > needing yelp/webkitgtk. Since the new installer itself needs > > webkitgtk, > > seems like there's no way we're getting rid of that ~40M > > compressed. > > > > > You should also try to do this - remove webkitgtk and yelp packages > and add firefox, because that's what the web based installer > should/will use in the end - see > https://bugzilla.redhat.com/show_bug.cgi?id=2142779. We already had a > meeting with Anaconda dev and told him to use Firefox instead of > WebKitGTK for the web installer - due to resources that are > internally (and frankly in upstream as well) allocated to WebKitGTK. While the currently published Web UI preview images use GTK3 WebKit to show the Web UI locally, I did some local image build using Firefox in kiosk mode (eq. using the --kiosk CLI option): - generally works fine - performance is good, does not exhibit the high CPU consumption for UI animations GTK WebKit has on VM with no GPU acceleration - no header bar ever shown (which is what we want at the moment) - no right click context menu, no right click copy paste menu (this is usable, be if possible we would like to show the web debug console option if anaconda runs in debug mode & the missing copa paste menu could be seen as a usability issue for users) - adds about 80 MB to image size (so about +40 MB in comparison to GTK WebKit as far as I can tell - *very rough/preliminary numbers*) - the memory consumption (measured at end of the installation) seems to be slightly higher when compared to GTK WebKit BTW, just an idea - would a "Firefox minimal" build make sense ? I don't think we (and other projects in need of a lightweight Web View) would ne advanced vide playback support, Web GL, VR support and other features that certainly make the Firefox binaries bigger than necessary for the usecase. > > Tom > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 7:31 AM Kevin Kofler via devel wrote: > > Tomáš Popela wrote: > > firefox, because that's what the web based installer should/will use in > > the end > > 臘 A full-blown, 71 MB compressed (!) web browser just to show the UI for the > installer! > > IMHO, the web-based UI is a major mistake and should never be shipped in > Fedora. It is good as a prototype, but way too resource-heavy to be shipped > in production. We need a rewrite of the UI design in a desktop toolkit. (If > GTK is not suitable, how about Qt? ☺) > While I agree with you on a Qt-based frontend over a GTK-based one, what they're doing is valuable because it splits Anaconda into a service that lets anyone implement whatever frontends they want. If we wound up having an enterprising developer interested in Fedora KDE to make a frontend for Anaconda leveraging Kirigami, we absolutely could. Will we? I don't know, probably not (at least for a while), but it might happen someday. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Dne 08. 12. 22 v 19:33 Adam Williamson napsal(a): On that note, /usr/share/doc, /usr/share/man, and /usr/share/info could be removed from the installer image if they are present. That likely won't free a whole lot of space, but it's not nothing. All of those are already stripped: https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-cleanup.tmpl#L69 Am I blind, or there is no removal of /var/cache/* ? Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Wed, 2022-12-07 at 20:19 -0800, Adam Williamson wrote: > On Thu, 2022-12-08 at 03:28 +, Gary Buhrmaster wrote: > > On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson > > wrote: > > > > > > Hi folks! Today I woke up and found > > > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which > > > diverted me > > > down a bit of an "installer environment size" rabbit hole. > > > > Does the "new and improved" web based installer help this > > in any way? > > I haven't looked yet but I suspect it'll probably be a wash, mostly. Yeah - even when we drop the actual Python/GTK3 GUI code, I don't expect the size to change much, as GTK itself will end up on the imaage anyway - GTK WebKit needs it & AFAIK Firefox also uses it to setup it window (?) and related stuff. > If > anything it's likely slightly negative because the new installer > itself > hard requires webkitgtk, so we can't really do anything to finesse > that > requirement any more. On the other hand we expect the help to be just a part of the Web UI, so no need for yelp and likely some other GUI tools (eq. that thing that shows keyboard layouts, possibly network connection editor, etc.). But yet again I would not expect huge savings there as the tools themselves are likely tiny, the main size comming from the GUI toolikt they use. > With the old installer, we could maybe try and > figure out some way of being able to show the help pages without > needing yelp/webkitgtk. Or perhaps drop even the help system ? The help content has actually not been updated in a while and does not really have an active "upstream" - the Fedora docs moved on to a very different format, that can no longer be used to produce the per-screen content the current help system needs. So maybe dropping the help support-yelp-webkit & getting 40 MB back for now could be worth it ? (With the built-in-help system in the Web UI, we plan to have the installer UI specific help content maintained as part of the Anaconda project, with easy access to documentatrists and contributors to avoid the issues with up-to-date help content. It will also solve localization issues & makes it possible to update as changes in the code happen.) > Since the new installer itself needs webkitgtk, > seems like there's no way we're getting rid of that ~40M compressed. One possible option that could work in some use cases is to also build headless images, where you would connect to the Web UI remotely - this could be very useful for SBC, as it would avoid any CPU/RAM intensive local rendering, yet having the full GUI experience available. The resulting headless image could possibly quite small without GTK,X/Wayland,WebKit/Firefox, Gnome Kiosk and their transitive deps. > -- > Adam Williamson > Fedora QA > IRC: adamw | Twitter: adamw_ha > https://www.happyassassin.net > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Hi, On Thu, Dec 8, 2022 at 2:55 PM Adam Williamson wrote: > This is the direction Daniel was thinking down. I'm waiting for someone > with more expertise to reply, but I suspect the reply is going to be > along the lines of "yes, we *can* do that, but it's somewhat tricky > work that involves thinking about lots of paths that aren't obvious, > and somebody would need to dedicate their time to working on that". Presumably we could package the firmware separately and just unpack it into place from a udev rule when the hardware is detected? But first, do we actually know this is a problem? I think you're saying squashfs loads the whole decompressed image into memory, but my expectation prior to your mail was that it performs I/O on the usb stick (with a cache in between). If my intuition was right and files only hit ram when accessed, then it seems like this is pretty much not an issue, right? Do you have stats on memory usage when running in a live environment? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Dne 08. 12. 22 v 17:52 Adam Williamson napsal(a): No, and we're looking at splitting those out, but the fact is they are a tiny amount of the overall firmware collection. You could even argue either way for something like bluetooth due to it sometimes being used by keyboards. We actually already strip a lot of those in lorax. ... then, in cleanup, we wipe a bunch of files that have not yet been split out from linux-firmware but which we know the installer env doesn't need: https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-cleanup.tmpl#L201 Thanks for the links. Putting aside the installer image... the firmware files mentioned in last link are not so small: [msuchy@dri//tmp/linux-firmware-20221109-144.fc38.noarch]$ du-hcusr/lib/firmware/dvb*usr/lib/firmware/*_12mhz*usr/lib/firmware/v4l*usr/lib/firmware/brcm/BCM-*usr/lib/firmware/ttusb- budget/dspbootcode.bin*usr/lib/firmware/emi26/*usr/lib/firmware/emi62/*usr/lib/firmware/cpia2/*usr/lib/firmware/dabusb/*usr/lib/firmware/vicam/*usr/lib/firmware/dsp56k/*usr/lib/firmw are/sun/*usr/lib/firmware/av7110/*usr/lib/firmware/usbdux*usr/lib/firmware/f2255usb.bin*usr/lib/firmware/lgs8g75.fw*usr/lib/firmware/tlg2300_firmware.bin*usr/lib/firmware/s5p-mfc*us r/lib/firmware/go7007/*usr/lib/firmware/intel/IntcSST2.bin*usr/lib/firmware/intel/fw_sst*usr/lib/firmware/intel/dsp*usr/lib/firmware/as102*usr/lib/firmware/qcom/apq8096/*usr/lib/firmw are/qcom/sdm845/*usr/lib/firmware/qcom/sm8250/*usr/lib/firmware/qcom/venus*/*usr/lib/firmware/qcom/vpu*/*usr/lib/firmware/meson/vdec/*usr/lib/firmware/phanfw.bin* 28M total This is about 20 % of size of linux-firmware package. Can this be moved to to separate subpackage? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 5:06 PM Adam Williamson wrote: > > On Thu, 2022-12-08 at 16:51 +, Gary Buhrmaster wrote: > > On Thu, Dec 8, 2022 at 12:58 PM Peter Robinson wrote: > > > > > I've done a few passes, dropping a bunch of older firmware upstream > > > that are no longer supported in any stable kernel release, also a > > > bunch of de-dupe and linking of files rather than shipping of multiple > > > copies of the same firmware. It's improved things a bit, unfortunately > > > a lot of the dead firmware was tiny compared to say average modern > > > devices like GPUs or WiFI. > > > > > > The problem with a lot of the firmware, and with the new nvidia "open > > > driver" which shoves a lot of stuff into firmware in order to have an > > > upstreamable driver apparently the firmwares there are going to be > > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > > even just install so I don't know how we can get around this and still > > > have a device work enough to be able to install the needed firmware > > > across the network. > > > > > > Ideas on how to solve that problem welcome. > > > > Does compressing the firmware using zstd (perhaps > > at an aggressive level) level help at all? > > It already *is* compressed, which is why it doesn't get any smaller in > the compressed filesystem image, unlike the other things I mentioned. > Check for yourself - look under /lib/firmware and you'll see only > things ending in .xz. And "aggressive level zstd" makes little difference TBH, and we went with XZ over zstd because of the support matrix needed actoss various pieces to support compression wasn't there, and may not be yet, I've not revisited the whole end to end process since it landed to audit all the bits since the first feature. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 09, 2022 at 12:04:24PM +0100, Florian Weimer wrote: > * Richard W. M. Jones: > > > You only need network / wifi firmware blobs (although I'm sure they > > are in themselves large) and then you can fetch anything else needed > > for the hardware including graphics, right? > > I think you need graphics to set up wifi. I long for old school text mode installers ... At least you knew that the tab key would always work. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 4:56 PM Adam Williamson wrote: > > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > I've done a few passes, dropping a bunch of older firmware upstream > > that are no longer supported in any stable kernel release, also a > > bunch of de-dupe and linking of files rather than shipping of multiple > > copies of the same firmware. It's improved things a bit, unfortunately > > a lot of the dead firmware was tiny compared to say average modern > > devices like GPUs or WiFI. > > > > The problem with a lot of the firmware, and with the new nvidia "open > > driver" which shoves a lot of stuff into firmware in order to have an > > upstreamable driver apparently the firmwares there are going to be > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > even just install so I don't know how we can get around this and still > > have a device work enough to be able to install the needed firmware > > across the network. > > > > Ideas on how to solve that problem welcome. > > Sorry if this is way off, but - do we need the GPU firmwares to run a > graphical install on the fallback path, just using the framebuffer set > up by the firmware? How crazy would it be to just do that - ship the > installer env with no GPU firmware? That has crossed my mind, and with simpledrm that may be more straight forward now, but TBH it's not something I am skilled enough to deal with, nor have the resources to test, or actually care enough about, but the big GPU firmwares are now all split out so that should be much more straightforward for someone with the resources to investigate. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
* Richard W. M. Jones: > You only need network / wifi firmware blobs (although I'm sure they > are in themselves large) and then you can fetch anything else needed > for the hardware including graphics, right? I think you need graphics to set up wifi. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 08, 2022 at 11:49:16AM -0800, Adam Williamson wrote: > On Thu, 2022-12-08 at 20:23 +0100, drago01 wrote: > > The problem I see here is not the presence of the firmware on the > > image, > > but the fact that it seems to be loaded into memory despite not being > > used. > > This is the direction Daniel was thinking down. I'm waiting for someone > with more expertise to reply, but I suspect the reply is going to be > along the lines of "yes, we *can* do that, but it's somewhat tricky > work that involves thinking about lots of paths that aren't obvious, > and somebody would need to dedicate their time to working on that". Split install.img into install.img + firmware.img? I think we already have support for multiple images (I see requests for updates.img when watching httpd logs while doing network installs), so the split should be easy. The somewhat more tricky part is probably to figure whenever we need the firmware or not. take care, Gerd ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Fri, Dec 09, 2022 at 08:09:42AM +, Daniel P. Berrangé wrote: > On Thu, Dec 08, 2022 at 12:54:16PM -0800, Adam Williamson wrote: > > On Thu, 2022-12-08 at 20:43 +0100, drago01 wrote: > > > On Thursday, December 8, 2022, Chris Adams wrote: > > > > > > > Once upon a time, Daniel P. Berrangé said: > > > > > On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote: > > > > > > That would be very crazy, as you will have a degraded user > > > > > > experience > > > > > > (laggy UI, wrong resolution, ...) to save a couple of megabytes that > > > > are a > > > > > > non issue for today's hardware. > > > > > > > > > > Please bear in mind the difference between bare metal and virtual > > > > > machines. The bare metal machine may have 32 GB of RAM, making a > > > > > 800 MB install image a non-issue. For a public cloud virtual machine > > > > > though, this could bump your VM sizing up 1 level from 2 GB quota > > > > > to a 4 GB RAM quota, with correspondingly higher price point. > > > > > > > > Also "today's hardware" increasingly includes small devices like > > > > Raspberry Pi. ARM devices don't typically use anaconda, but there are > > > > also small x86 based devices competing with the small ARM devices. > > > > > > > > I think the answer is "no", but I'll ask anyway: is there a way to evict > > > > all the firmware once the system is started? I'm guessing that as long > > > > as it's all in one disk image, that's not possible. Can we tack on a > > > > second disk image with use-once (at most) stuff and then drop the whole > > > > image after startup? > > > > > > > > > > Again there is no reason why everything on the disk image had to be loaded > > > into memory in the first place. Same way when you boot your installed > > > system, not everything on disk is loaded into memory. If you don't need > > > the > > > firmware, it should stay on the install media and never be loaded into > > > memory. > > > > The problem is, what is "the install media"? We don't *only* support > > installs from USB sticks and DVDs - things the installer could > > potentially access as local storage after starting up. We also do > > installs where everything is retrieved over the network - PXE installs, > > for instance. > > > > There are possible ways to finesse things even in those cases - as I > > said, Daniel started thinking them through a bit - but it's not as > > simple as just "put this stuff on the ISO and read it off that". > > It could potentially be almost that simple actually > > qemu-nbd -c https:///some.server/path/to/second.iso > mount /dev/nbd0 /mnt/second-iso > > This uses QEMU's curl driver, which will fetch blocks of the ISO > content only as they are accessed, so you're not pulling down the > whole ISO if you only read 2 files from it. > > The 'nbdkit' program can be used instead of qemu-nbd, and probably > a better choice since it can layer into all sorts of interesting > functionality that QEMU's curl layer can't offer. This, but using kernel nbd root instead of a qemu nbd file: https://rwmj.wordpress.com/2019/02/19/nbdkit-linuxdisk-plugin/ Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 08, 2022 at 02:12:22PM -0600, Chris Adams wrote: > Once upon a time, drago01 said: > > Again there is no reason why everything on the disk image had to be loaded > > into memory in the first place. Same way when you boot your installed > > system, not everything on disk is loaded into memory. If you don't need the > > firmware, it should stay on the install media and never be loaded into > > memory. > > That only works for cases where there is local install media. Network > installs require downloading and image and running it from RAM. That's not really true as long as the web server supports random access and/or you use NBD or NFS root. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com nbdkit - Flexible, fast NBD server with plugins https://gitlab.com/nbdkit/nbdkit ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 6:45 PM Adam Williamson wrote: > On Thu, 2022-12-08 at 08:53 +0100, Tomáš Popela wrote: > > Hi Adam, > > > > On Thu, Dec 8, 2022 at 5:30 AM Adam Williamson < > adamw...@fedoraproject.org> > > wrote: > > > > > On Thu, 2022-12-08 at 03:28 +, Gary Buhrmaster wrote: > > > > On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson > > > > wrote: > > > > > > > > > > Hi folks! Today I woke up and found > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which > diverted > > > me > > > > > down a bit of an "installer environment size" rabbit hole. > > > > > > > > Does the "new and improved" web based installer help this > > > > in any way? > > > > > > I haven't looked yet but I suspect it'll probably be a wash, mostly. If > > > anything it's likely slightly negative because the new installer itself > > > hard requires webkitgtk, so we can't really do anything to finesse that > > > requirement any more. With the old installer, we could maybe try and > > > figure out some way of being able to show the help pages without > > > needing yelp/webkitgtk. Since the new installer itself needs webkitgtk, > > > seems like there's no way we're getting rid of that ~40M compressed. > > > > > > > You should also try to do this - remove webkitgtk and yelp packages and > add > > firefox, because that's what the web based installer should/will use in > the > > end - see https://bugzilla.redhat.com/show_bug.cgi?id=2142779. We > already > > had a meeting with Anaconda dev and told him to use Firefox instead of > > WebKitGTK for the web installer - due to resources that are internally > (and > > frankly in upstream as well) allocated to WebKitGTK. > > I can try that. What will Firefox run on top of? Is that decided? A > bare X server? Weston? Something else? Thanks! > I think that it will run on top of gnome-kiosk (what anaconda uses now - implemented in https://github.com/rhinstaller/anaconda/pull/3307), but I will add @Radek Vykydal here to confirm. Tom ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Wed, Dec 07, 2022 at 04:42:05PM -0800, Adam Williamson wrote: > I've moaned about the sheer amount and size of firmware blobs in other > forums before, but 214M compressed is *really* obnoxious. We must be > able to do something to clean this up (further than it's already > cleaned up - this is *after* we dropped low-hanging fruit like > enterprise switch 'firmwares' and garbage like that; most of the > remaining size seems to be huge amounts of probably-very-similar > firmware files for AMD graphics adapters and Intel wireless adapters). > I know some folks were trying to work on this (there was talk that we > could drop quite a lot of files that would only be loaded by older > kernels no longer in Fedora); any news on how far along that effort is? You only need network / wifi firmware blobs (although I'm sure they are in themselves large) and then you can fetch anything else needed for the hardware including graphics, right? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 08, 2022 at 12:54:16PM -0800, Adam Williamson wrote: > On Thu, 2022-12-08 at 20:43 +0100, drago01 wrote: > > On Thursday, December 8, 2022, Chris Adams wrote: > > > > > Once upon a time, Daniel P. Berrangé said: > > > > On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote: > > > > > That would be very crazy, as you will have a degraded user experience > > > > > (laggy UI, wrong resolution, ...) to save a couple of megabytes that > > > are a > > > > > non issue for today's hardware. > > > > > > > > Please bear in mind the difference between bare metal and virtual > > > > machines. The bare metal machine may have 32 GB of RAM, making a > > > > 800 MB install image a non-issue. For a public cloud virtual machine > > > > though, this could bump your VM sizing up 1 level from 2 GB quota > > > > to a 4 GB RAM quota, with correspondingly higher price point. > > > > > > Also "today's hardware" increasingly includes small devices like > > > Raspberry Pi. ARM devices don't typically use anaconda, but there are > > > also small x86 based devices competing with the small ARM devices. > > > > > > I think the answer is "no", but I'll ask anyway: is there a way to evict > > > all the firmware once the system is started? I'm guessing that as long > > > as it's all in one disk image, that's not possible. Can we tack on a > > > second disk image with use-once (at most) stuff and then drop the whole > > > image after startup? > > > > > > > Again there is no reason why everything on the disk image had to be loaded > > into memory in the first place. Same way when you boot your installed > > system, not everything on disk is loaded into memory. If you don't need the > > firmware, it should stay on the install media and never be loaded into > > memory. > > The problem is, what is "the install media"? We don't *only* support > installs from USB sticks and DVDs - things the installer could > potentially access as local storage after starting up. We also do > installs where everything is retrieved over the network - PXE installs, > for instance. > > There are possible ways to finesse things even in those cases - as I > said, Daniel started thinking them through a bit - but it's not as > simple as just "put this stuff on the ISO and read it off that". It could potentially be almost that simple actually qemu-nbd -c https:///some.server/path/to/second.iso mount /dev/nbd0 /mnt/second-iso This uses QEMU's curl driver, which will fetch blocks of the ISO content only as they are accessed, so you're not pulling down the whole ISO if you only read 2 files from it. The 'nbdkit' program can be used instead of qemu-nbd, and probably a better choice since it can layer into all sorts of interesting functionality that QEMU's curl layer can't offer. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 10:46 PM Kevin Kofler via devel wrote: > > Adam Williamson wrote: > > On Thu, 2022-12-08 at 07:57 +0100, Florian Weimer wrote: > >> It has all the targets in it. As it's for JIT, we'd only need one > >> target. > > > > That sounds interesting, though of course the details of how to > > implement it could be a bit tricky, I guess... > > Build /usr/lib64/libLLVM-15.so with only the host as the target, and a > renamed /usr/lib64/libLLVM-cross-15.so with all the targets. Link the > compilers against the latter. We'd have to build host + amdgpu backends into it. Dave. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 08, 2022 at 09:28:48AM -0800, Adam Williamson wrote: > It shouldn't be too hard to try this out - it's just one setting in > lorax somewhere, but I gave up on this alleyway before figuring out > exactly where to set it. You can use an un-documented lorax config file to set things, defaults are here: https://github.com/weldr/lorax/blob/master/src/pylorax/__init__.py#L111 pass the new config with '-c lorax.cfg' A while back (year or two) I fiddled with settings and didn't find much of an improvement. The real problem here, as you have pointed out, is too much data. There's just not much that can be done when more and more files are crammed into the boot.iso At one time I even experimented with a text-only installer image. IIRC it only ended up being around 40M smaller so I gave up on that idea. But now that this has come up again I wonder if there is some usefulness to a no-firmware image for virt installs where none(?) of the firmware files are used. Brian -- Brian C. Lane (PST8PDT) - weldr.io - lorax - parted - pykickstart ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Adam Williamson wrote: > On Thu, 2022-12-08 at 12:50 -0500, David Cantrell wrote: >> If mesa-dri-drivers is not required for installation, it could be removed >> from the installer environment. > > It is required - we don't get any graphics without it. Is that because of the use of GNOME Shell? A non-compositing X11 window manager or a plain fbdev framebuffer would probably work fine without it, would they not? Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
David Cantrell wrote: > Broadly speaking, a lot of the growth came from converging the runtime > environment for the installer with the installed system. In Fedora Core 8 > and previous releases, the "installer environment" was a unique and > stripped down install. This was frustrating because it was effectively > maintaining a small mini distro for the purposes of running the distro > installer. But this "converging" unfortunately means the installer image now runs things like GDM which are definitely not needed to just bring up an installer. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Gary Buhrmaster wrote: > As I am sure you are aware, it seems a number of distros > are experimenting with their next gen installer. OpenSUSE > has their new web based D-installer, and Canonical is > writing an installer in flutter. And most of the others have stopped reinventing the wheel and are all shipping Calamares instead. Which is written in a toolkit actually designed for desktop applications (Qt). Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 20:43 +0100, drago01 wrote: > On Thursday, December 8, 2022, Chris Adams wrote: > > > Once upon a time, Daniel P. Berrangé said: > > > On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote: > > > > That would be very crazy, as you will have a degraded user experience > > > > (laggy UI, wrong resolution, ...) to save a couple of megabytes that > > are a > > > > non issue for today's hardware. > > > > > > Please bear in mind the difference between bare metal and virtual > > > machines. The bare metal machine may have 32 GB of RAM, making a > > > 800 MB install image a non-issue. For a public cloud virtual machine > > > though, this could bump your VM sizing up 1 level from 2 GB quota > > > to a 4 GB RAM quota, with correspondingly higher price point. > > > > Also "today's hardware" increasingly includes small devices like > > Raspberry Pi. ARM devices don't typically use anaconda, but there are > > also small x86 based devices competing with the small ARM devices. > > > > I think the answer is "no", but I'll ask anyway: is there a way to evict > > all the firmware once the system is started? I'm guessing that as long > > as it's all in one disk image, that's not possible. Can we tack on a > > second disk image with use-once (at most) stuff and then drop the whole > > image after startup? > > > > Again there is no reason why everything on the disk image had to be loaded > into memory in the first place. Same way when you boot your installed > system, not everything on disk is loaded into memory. If you don't need the > firmware, it should stay on the install media and never be loaded into > memory. The problem is, what is "the install media"? We don't *only* support installs from USB sticks and DVDs - things the installer could potentially access as local storage after starting up. We also do installs where everything is retrieved over the network - PXE installs, for instance. There are possible ways to finesse things even in those cases - as I said, Daniel started thinking them through a bit - but it's not as simple as just "put this stuff on the ISO and read it off that". -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Once upon a time, Adam Williamson said: > This is the direction Daniel was thinking down. I'm waiting for someone > with more expertise to reply, but I suspect the reply is going to be > along the lines of "yes, we *can* do that, but it's somewhat tricky > work that involves thinking about lots of paths that aren't obvious, > and somebody would need to dedicate their time to working on that". One other thing that I noticed a while back that takes up a chunk of space is the kernel... it's included inside install.img (in two places even, although I assume it's hardlinked?), even though it has to have already been loaded before install.img can be read. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Once upon a time, drago01 said: > Again there is no reason why everything on the disk image had to be loaded > into memory in the first place. Same way when you boot your installed > system, not everything on disk is loaded into memory. If you don't need the > firmware, it should stay on the install media and never be loaded into > memory. That only works for cases where there is local install media. Network installs require downloading and image and running it from RAM. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 20:23 +0100, drago01 wrote: > > Please bear in mind the difference between bare metal and virtual > > machines. The bare metal machine may have 32 GB of RAM, making a > > 800 MB install image a non-issue. For a public cloud virtual > > machine > > though, this could bump your VM sizing up 1 level from 2 GB quota > > to a 4 GB RAM quota, with correspondingly higher price point. Now > > most people probably don't run the installer in a public cloud, > > preferring pre-built disk images. Even in a local machine though, > > you may be using most of your 32 GB of RAM for other things (well > > firefox/chrome), so allowing extra for the VM is not without > > resource cost. If we could figure out a way to knock a few 100 MB > > off the installer RAM requirements that is valuable. > > > > > The problem I see here is not the presence of the firmware on the > image, > but the fact that it seems to be loaded into memory despite not being > used. This is the direction Daniel was thinking down. I'm waiting for someone with more expertise to reply, but I suspect the reply is going to be along the lines of "yes, we *can* do that, but it's somewhat tricky work that involves thinking about lots of paths that aren't obvious, and somebody would need to dedicate their time to working on that". -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thursday, December 8, 2022, Chris Adams wrote: > Once upon a time, Daniel P. Berrangé said: > > On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote: > > > That would be very crazy, as you will have a degraded user experience > > > (laggy UI, wrong resolution, ...) to save a couple of megabytes that > are a > > > non issue for today's hardware. > > > > Please bear in mind the difference between bare metal and virtual > > machines. The bare metal machine may have 32 GB of RAM, making a > > 800 MB install image a non-issue. For a public cloud virtual machine > > though, this could bump your VM sizing up 1 level from 2 GB quota > > to a 4 GB RAM quota, with correspondingly higher price point. > > Also "today's hardware" increasingly includes small devices like > Raspberry Pi. ARM devices don't typically use anaconda, but there are > also small x86 based devices competing with the small ARM devices. > > I think the answer is "no", but I'll ask anyway: is there a way to evict > all the firmware once the system is started? I'm guessing that as long > as it's all in one disk image, that's not possible. Can we tack on a > second disk image with use-once (at most) stuff and then drop the whole > image after startup? > Again there is no reason why everything on the disk image had to be loaded into memory in the first place. Same way when you boot your installed system, not everything on disk is loaded into memory. If you don't need the firmware, it should stay on the install media and never be loaded into memory. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Once upon a time, Daniel P. Berrangé said: > On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote: > > That would be very crazy, as you will have a degraded user experience > > (laggy UI, wrong resolution, ...) to save a couple of megabytes that are a > > non issue for today's hardware. > > Please bear in mind the difference between bare metal and virtual > machines. The bare metal machine may have 32 GB of RAM, making a > 800 MB install image a non-issue. For a public cloud virtual machine > though, this could bump your VM sizing up 1 level from 2 GB quota > to a 4 GB RAM quota, with correspondingly higher price point. Also "today's hardware" increasingly includes small devices like Raspberry Pi. ARM devices don't typically use anaconda, but there are also small x86 based devices competing with the small ARM devices. I think the answer is "no", but I'll ask anyway: is there a way to evict all the firmware once the system is started? I'm guessing that as long as it's all in one disk image, that's not possible. Can we tack on a second disk image with use-once (at most) stuff and then drop the whole image after startup? That wouldn't help the initial RAM usage, but it would free up a chunk (so that dnf can then use it). That wouldn't help with the other large space users, but it would probably make the firmware much less of an issue. -- Chris Adams ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thursday, December 8, 2022, Daniel P. Berrangé wrote: > On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote: > > On Thursday, December 8, 2022, Adam Williamson < > adamw...@fedoraproject.org> > > wrote: > > > > > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > > > > > I've done a few passes, dropping a bunch of older firmware upstream > > > > that are no longer supported in any stable kernel release, also a > > > > bunch of de-dupe and linking of files rather than shipping of > multiple > > > > copies of the same firmware. It's improved things a bit, > unfortunately > > > > a lot of the dead firmware was tiny compared to say average modern > > > > devices like GPUs or WiFI. > > > > > > > > The problem with a lot of the firmware, and with the new nvidia "open > > > > driver" which shoves a lot of stuff into firmware in order to have an > > > > upstreamable driver apparently the firmwares there are going to be > > > > 30+Mb each, is that they're needed to bring up graphics/network etc > to > > > > even just install so I don't know how we can get around this and > still > > > > have a device work enough to be able to install the needed firmware > > > > across the network. > > > > > > > > Ideas on how to solve that problem welcome. > > > > > > Sorry if this is way off, but - do we need the GPU firmwares to run a > > > graphical install on the fallback path, just using the framebuffer set > > > up by the firmware? How crazy would it be to just do that - ship the > > > installer env with no GPU firmware? > > > > > > That would be very crazy, as you will have a degraded user experience > > (laggy UI, wrong resolution, ...) to save a couple of megabytes that are > a > > non issue for today's hardware. > > Please bear in mind the difference between bare metal and virtual > machines. The bare metal machine may have 32 GB of RAM, making a > 800 MB install image a non-issue. For a public cloud virtual machine > though, this could bump your VM sizing up 1 level from 2 GB quota > to a 4 GB RAM quota, with correspondingly higher price point. Now > most people probably don't run the installer in a public cloud, > preferring pre-built disk images. Even in a local machine though, > you may be using most of your 32 GB of RAM for other things (well > firefox/chrome), so allowing extra for the VM is not without > resource cost. If we could figure out a way to knock a few 100 MB > off the installer RAM requirements that is valuable. > > The problem I see here is not the presence of the firmware on the image, but the fact that it seems to be loaded into memory despite not being used. > With regards, > Daniel > -- > |: https://berrange.com -o-https://www.flickr.com/photos/ > dberrange :| > |: https://libvirt.org -o- > https://fstop138.berrange.com :| > |: https://entangle-photo.org-o-https://www.instagram.com/ > dberrange :| > ___ > kernel mailing list -- ker...@lists.fedoraproject.org > To unsubscribe send an email to kernel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject. > org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/kernel@ > lists.fedoraproject.org > Do not reply to spam, report it: https://pagure.io/fedora- > infrastructure/new_issue > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 08, 2022 at 07:59:20PM +0100, drago01 wrote: > On Thursday, December 8, 2022, Adam Williamson > wrote: > > > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > > > I've done a few passes, dropping a bunch of older firmware upstream > > > that are no longer supported in any stable kernel release, also a > > > bunch of de-dupe and linking of files rather than shipping of multiple > > > copies of the same firmware. It's improved things a bit, unfortunately > > > a lot of the dead firmware was tiny compared to say average modern > > > devices like GPUs or WiFI. > > > > > > The problem with a lot of the firmware, and with the new nvidia "open > > > driver" which shoves a lot of stuff into firmware in order to have an > > > upstreamable driver apparently the firmwares there are going to be > > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > > even just install so I don't know how we can get around this and still > > > have a device work enough to be able to install the needed firmware > > > across the network. > > > > > > Ideas on how to solve that problem welcome. > > > > Sorry if this is way off, but - do we need the GPU firmwares to run a > > graphical install on the fallback path, just using the framebuffer set > > up by the firmware? How crazy would it be to just do that - ship the > > installer env with no GPU firmware? > > > That would be very crazy, as you will have a degraded user experience > (laggy UI, wrong resolution, ...) to save a couple of megabytes that are a > non issue for today's hardware. Please bear in mind the difference between bare metal and virtual machines. The bare metal machine may have 32 GB of RAM, making a 800 MB install image a non-issue. For a public cloud virtual machine though, this could bump your VM sizing up 1 level from 2 GB quota to a 4 GB RAM quota, with correspondingly higher price point. Now most people probably don't run the installer in a public cloud, preferring pre-built disk images. Even in a local machine though, you may be using most of your 32 GB of RAM for other things (well firefox/chrome), so allowing extra for the VM is not without resource cost. If we could figure out a way to knock a few 100 MB off the installer RAM requirements that is valuable. With regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 19:59 +0100, drago01 wrote: > On Thursday, December 8, 2022, Adam Williamson > wrote: > > > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > > > I've done a few passes, dropping a bunch of older firmware upstream > > > that are no longer supported in any stable kernel release, also a > > > bunch of de-dupe and linking of files rather than shipping of multiple > > > copies of the same firmware. It's improved things a bit, unfortunately > > > a lot of the dead firmware was tiny compared to say average modern > > > devices like GPUs or WiFI. > > > > > > The problem with a lot of the firmware, and with the new nvidia "open > > > driver" which shoves a lot of stuff into firmware in order to have an > > > upstreamable driver apparently the firmwares there are going to be > > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > > even just install so I don't know how we can get around this and still > > > have a device work enough to be able to install the needed firmware > > > across the network. > > > > > > Ideas on how to solve that problem welcome. > > > > Sorry if this is way off, but - do we need the GPU firmwares to run a > > graphical install on the fallback path, just using the framebuffer set > > up by the firmware? How crazy would it be to just do that - ship the > > installer env with no GPU firmware? > > > That would be very crazy, as you will have a degraded user experience > (laggy UI, wrong resolution, ...) to save a couple of megabytes that are a > non issue for today's hardware. I mean, the modern systems that *need* GPU firmware generally seem to do pretty well with using native resolution and don't perform too badly, especially in the simple installer UI. When I test the fallback path on my bare metal test box on UEFI it uses the monitor's native resolution and performs fine (even, honestly, in GNOME), and that motherboard is nearly a decade old even. Don't know if this is the same for everyone, of course. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thursday, December 8, 2022, Adam Williamson wrote: > On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > > > I've done a few passes, dropping a bunch of older firmware upstream > > that are no longer supported in any stable kernel release, also a > > bunch of de-dupe and linking of files rather than shipping of multiple > > copies of the same firmware. It's improved things a bit, unfortunately > > a lot of the dead firmware was tiny compared to say average modern > > devices like GPUs or WiFI. > > > > The problem with a lot of the firmware, and with the new nvidia "open > > driver" which shoves a lot of stuff into firmware in order to have an > > upstreamable driver apparently the firmwares there are going to be > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > even just install so I don't know how we can get around this and still > > have a device work enough to be able to install the needed firmware > > across the network. > > > > Ideas on how to solve that problem welcome. > > Sorry if this is way off, but - do we need the GPU firmwares to run a > graphical install on the fallback path, just using the framebuffer set > up by the firmware? How crazy would it be to just do that - ship the > installer env with no GPU firmware? That would be very crazy, as you will have a degraded user experience (laggy UI, wrong resolution, ...) to save a couple of megabytes that are a non issue for today's hardware. > Adam Williamson > Fedora QA > IRC: adamw | Twitter: adamw_ha > https://www.happyassassin.net > > ___ > desktop mailing list -- desk...@lists.fedoraproject.org > To unsubscribe send an email to desktop-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject. > org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/desktop@ > lists.fedoraproject.org > Do not reply to spam, report it: https://pagure.io/fedora- > infrastructure/new_issue > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 12:50 -0500, David Cantrell wrote: > On Wed, Dec 07, 2022 at 04:42:05PM -0800, Adam Williamson wrote: > > Hi folks! Today I woke up and found > > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me > > down a bit of an "installer environment size" rabbit hole. > > > > As of today, with that new dep in webkitgtk, Rawhide's network install > > images are 703M in size. Here's a potted history of network install > > image sizes: > > > > Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M) > > Fedora 13: 208M > > Fedora 17: 162M (last "old UI") > > Fedora 18: 294M (first "new UI") > > Fedora 23: 415M > > Fedora 28: 583M > > Fedora 33: 686M > > Fedora 37: 665M > > Fedora Rawhide: 703M > > > > The installer does not really do much more in Rawhide than it did in > > FC8. Even after the UI rewrite in F18, we were only at 294M. Now the > > image is well over 2x as big and does...basically the same. > > I take issue with this. It is not accurate to say that the installer now does > not do much more than it did for Fedora Core 8. There is more to the > installer than the UI. > > Broadly speaking, a lot of the growth came from converging the runtime > environment for the installer with the installed system. In Fedora Core 8 and > previous releases, the "installer environment" was a unique and stripped down > install. This was frustrating because it was effectively maintaining a small > mini distro for the purposes of running the distro installer. I meant it doesn't do much more in terms of what it achieves for the user. But we can also just take F18 as the base point, if you like. We're still over 2x as big as that was. > I think some curation on firmware could happen. I think probinson@ mentions > it later in this thread. If there were a way to identify the firmware > necessary for the installer environment that could probably simplify things. We already do a lot of "identifying the firmware necessary for the installer environment", see my earlier mail about all the stuff lorax does here. We've done the easy part, unfortunately. The stuff that's left is stuff that is, in some sense, needed - graphics card and wireless adapter firmwares, mainly. > > > Other obvious things that take up a lot of space: > > > > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is > > 224M uncompressed. A quick test just compressing the file with xz on my > > system shows it compresses to around 11M, though, so that's probably > > all it adds up to after compression (the image is an xz-compressed > > squashfs) > > Can this be installed compressed? I'm not sure it can. The "compressed" size is the effective size we're concerned about, because the installer filesystem image is an xz-compressed squashfs. So 11M is already the "effective weight" of this file, I think - if I ran an image build with it removed, it'd probably be ~11M smaller. > > > 2. /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to > > 23M. We are, I think, basically stuck with this for mesa-dri-drivers , > > but does it have to be so *big*? > > If mesa-dri-drivers is not required for installation, it could be removed from > the installer environment. It is required - we don't get any graphics without it. > > 4. /usr/share/locale - 112M in total (uncompressed, not sure how much > > compressed) of translated strings from a ton of packages. No idea how > > many of these are really *needed* in the installer environment. We can > > maybe come up with a way to have lorax strip some, if we can come up > > with a viable way to figure out which. Obviously-fairly-large ones are > > from gnupg2 and libgweather4. I do recall we have some logic somewhere > > to decide which languages have a certain level of translation in > > anaconda; perhaps we could only include the strings for these > > languages? > > On that note, /usr/share/doc, /usr/share/man, and /usr/share/info could be > removed from the installer image if they are present. That likely won't free > a whole lot of space, but it's not nothing. All of those are already stripped: https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-cleanup.tmpl#L69 -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Wed, Dec 07, 2022 at 04:42:05PM -0800, Adam Williamson wrote: > Hi folks! Today I woke up and found > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me > down a bit of an "installer environment size" rabbit hole. > > As of today, with that new dep in webkitgtk, Rawhide's network install > images are 703M in size. Here's a potted history of network install > image sizes: > > Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M) > Fedora 13: 208M > Fedora 17: 162M (last "old UI") > Fedora 18: 294M (first "new UI") > Fedora 23: 415M > Fedora 28: 583M > Fedora 33: 686M > Fedora 37: 665M > Fedora Rawhide: 703M > > The installer does not really do much more in Rawhide than it did in > FC8. Even after the UI rewrite in F18, we were only at 294M. Now the > image is well over 2x as big and does...basically the same. I take issue with this. It is not accurate to say that the installer now does not do much more than it did for Fedora Core 8. There is more to the installer than the UI. Broadly speaking, a lot of the growth came from converging the runtime environment for the installer with the installed system. In Fedora Core 8 and previous releases, the "installer environment" was a unique and stripped down install. This was frustrating because it was effectively maintaining a small mini distro for the purposes of running the distro installer. > Why does this matter? Well, the images being large is moderately > annoying in itself just in terms of transfer times and so on. But more > importantly, AIUI at least, the entire installer environment is loaded > into RAM at startup - it kinda has to be, we don't have anywhere else > to put it. The bigger it is, the more RAM you need to install Fedora. > The size of the installer environment (for which the size of the > network install image is more or less a perfect proxy) is one of the > two key factors in this, the other being how much RAM DNF uses during > package install. > > So, I did a bit of poking about into *what* is taking up all that > space. There's a variety of answers, but there's two major culprits: > > 1. firmware > 2. yelp (which pulls in webkitgtk and its deps) > > I've been using du and baobab (the GNOME visual disk usage analyzer, > which is great) to examine the filesystems, but I ran a couple of test > builds to confirm these suspects, especially after the impact of > compression (it's hard to check the *compressed* size of things in the > installer environment directly). > > I did a scratch build of lorax which does not pull in firmware > packages, and had openQA build a netinst using that lorax. It came out > at 489M - 214M smaller than current netinsts, a size we last managed in > Fedora 26. I did a scratch build of anaconda with its requirement of > yelp dropped (which would break help pages), and built a netinst with > that; it came out at 662M - 41M smaller than current images. I haven't > run a combined test yet, but it ought to come out around 448M, around > the size of Fedora 24. > > Even then we'd still be about 50% larger than the Fedora 18 image, for > not really any added functionality. > > I've moaned about the sheer amount and size of firmware blobs in other > forums before, but 214M compressed is *really* obnoxious. We must be > able to do something to clean this up (further than it's already > cleaned up - this is *after* we dropped low-hanging fruit like > enterprise switch 'firmwares' and garbage like that; most of the > remaining size seems to be huge amounts of probably-very-similar > firmware files for AMD graphics adapters and Intel wireless adapters). > I know some folks were trying to work on this (there was talk that we > could drop quite a lot of files that would only be loaded by older > kernels no longer in Fedora); any news on how far along that effort is? I think some curation on firmware could happen. I think probinson@ mentions it later in this thread. If there were a way to identify the firmware necessary for the installer environment that could probably simplify things. > Other obvious things that take up a lot of space: > > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is > 224M uncompressed. A quick test just compressing the file with xz on my > system shows it compresses to around 11M, though, so that's probably > all it adds up to after compression (the image is an xz-compressed > squashfs) Can this be installed compressed? I'm not sure it can. I guess a more important question is whether or not this file is used at install time. That I do not know. > 2. /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to > 23M. We are, I think, basically stuck with this for mesa-dri-drivers , > but does it have to be so *big*? If mesa-dri-drivers is not required for installation, it could be removed from the installer environment. > 3. libicudata.so.71.1 - 30.4M, compresses to 7M. This is in the > webkitgtk dep chain but seems to still be
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 5:29 PM Adam Williamson wrote: > It shouldn't be too hard to try this out - it's just one setting in > lorax somewhere, but I gave up on this alleyway before figuring out > exactly where to set it. Well, a *very* unscientific (a few random files) showed a mostly small difference between xz and zstd at ultra compression levels (sometimes xz won, sometimes zstd, mostly by small margins), so it is probably mostly a wash (there are probably a few outliers, as there are always outliers). So it seems like zstd was not a good idea. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 08:53 +0100, Tomáš Popela wrote: > Hi Adam, > > On Thu, Dec 8, 2022 at 5:30 AM Adam Williamson > wrote: > > > On Thu, 2022-12-08 at 03:28 +, Gary Buhrmaster wrote: > > > On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson > > > wrote: > > > > > > > > Hi folks! Today I woke up and found > > > > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted > > me > > > > down a bit of an "installer environment size" rabbit hole. > > > > > > Does the "new and improved" web based installer help this > > > in any way? > > > > I haven't looked yet but I suspect it'll probably be a wash, mostly. If > > anything it's likely slightly negative because the new installer itself > > hard requires webkitgtk, so we can't really do anything to finesse that > > requirement any more. With the old installer, we could maybe try and > > figure out some way of being able to show the help pages without > > needing yelp/webkitgtk. Since the new installer itself needs webkitgtk, > > seems like there's no way we're getting rid of that ~40M compressed. > > > > You should also try to do this - remove webkitgtk and yelp packages and add > firefox, because that's what the web based installer should/will use in the > end - see https://bugzilla.redhat.com/show_bug.cgi?id=2142779. We already > had a meeting with Anaconda dev and told him to use Firefox instead of > WebKitGTK for the web installer - due to resources that are internally (and > frankly in upstream as well) allocated to WebKitGTK. I can try that. What will Firefox run on top of? Is that decided? A bare X server? Weston? Something else? Thanks! -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 17:10 +, Gary Buhrmaster wrote: > On Thu, Dec 8, 2022 at 5:06 PM Adam Williamson > wrote: > > > It already *is* compressed, which is why it doesn't get any smaller in > > the compressed filesystem image, unlike the other things I mentioned. > > Check for yourself - look under /lib/firmware and you'll see only > > things ending in .xz. > > Right, but zstd *may* have a better compression > ratio than .xz (that was at least one of the reasons > given for the changes to support it). I actually spent half an hour looking into that yesterday as I got diverted into the details of how the filesystem compression stuff works, but all the references I found say xz consistently compresses better than zstd. zstd's advantages over xz are in *performance* (time taken to compress and decompress). So switching from xz to zstd seems like it would make things bigger, not smaller. It shouldn't be too hard to try this out - it's just one setting in lorax somewhere, but I gave up on this alleyway before figuring out exactly where to set it. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 5:06 PM Adam Williamson wrote: > It already *is* compressed, which is why it doesn't get any smaller in > the compressed filesystem image, unlike the other things I mentioned. > Check for yourself - look under /lib/firmware and you'll see only > things ending in .xz. Right, but zstd *may* have a better compression ratio than .xz (that was at least one of the reasons given for the changes to support it). ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Wed, Dec 07, 2022 at 04:42:05PM -0800, Adam Williamson wrote: > Hi folks! Today I woke up and found > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me > down a bit of an "installer environment size" rabbit hole. snip > Why does this matter? Well, the images being large is moderately > annoying in itself just in terms of transfer times and so on. But more > importantly, AIUI at least, the entire installer environment is loaded > into RAM at startup - it kinda has to be, we don't have anywhere else > to put it. The bigger it is, the more RAM you need to install Fedora. > The size of the installer environment (for which the size of the > network install image is more or less a perfect proxy) is one of the > two key factors in this, the other being how much RAM DNF uses during > package install. Is there something that can be done to optimize the RAM usage, in spite of the large installer env size ? If we're installing off DVD media, it shouldn't be required to pull all of the content into RAM, since it can be fetched on demand from the media. IOW, 99% of the firmware never need leave the ISO, so shouldn't matter if firmware is GBs in size [1] if we never load it off the media. Same for languages, only the one we actually want to use should ever get into RAM. If we're installing off a network source, we need to pull content into RAM, but that doesn't mean we should pull everything in at once upfront. Is it possible to delay pulling in non-NIC firmware until we have a NIC configured, and just rely on the basic generic framebuffer setup by UEFI/BIOS until we get far enugh to pull in video card firmware ? For localization, is it possible to split the localization into per-language bundles, and delay loading off the network until we know what language we want to load, instead of pre-loading all languages ? With regards, Daniel [1] Yes, I know it matters for user media download size in reality -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 16:51 +, Gary Buhrmaster wrote: > On Thu, Dec 8, 2022 at 12:58 PM Peter Robinson wrote: > > > I've done a few passes, dropping a bunch of older firmware upstream > > that are no longer supported in any stable kernel release, also a > > bunch of de-dupe and linking of files rather than shipping of multiple > > copies of the same firmware. It's improved things a bit, unfortunately > > a lot of the dead firmware was tiny compared to say average modern > > devices like GPUs or WiFI. > > > > The problem with a lot of the firmware, and with the new nvidia "open > > driver" which shoves a lot of stuff into firmware in order to have an > > upstreamable driver apparently the firmwares there are going to be > > 30+Mb each, is that they're needed to bring up graphics/network etc to > > even just install so I don't know how we can get around this and still > > have a device work enough to be able to install the needed firmware > > across the network. > > > > Ideas on how to solve that problem welcome. > > Does compressing the firmware using zstd (perhaps > at an aggressive level) level help at all? It already *is* compressed, which is why it doesn't get any smaller in the compressed filesystem image, unlike the other things I mentioned. Check for yourself - look under /lib/firmware and you'll see only things ending in .xz. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 08:26 -0500, Stephen Smoogen wrote:
> >
> The only ideas I have seen which 'work'* is to ship a minimal set of
> drivers for some 'chosen' hardware and then you have a bloated kitchen-sink
> iso which has all the drivers in it. The chosen hardware could be a
> 'defined' virtual environment which needs a minimal set of firmware,
> languages, etc. Everything else can be done in the install for getting
> languages, extra firmware, etc. The kitchen-sink.iso is going to be one
> which we know is going to be large.
>
> Now I have doubled the QA, releng, and product work.. I would say we would
> focus most of the work on the mini-installer, but we all know that all the
> work will be in the kitchen-sink one.
Well, if the two images are just "with firmware" and "without firmware"
I suppose in a way the testing load shouldn't be too awful, because we
can be pretty confident of the possible behaviours - nothing except the
kernel should do anything with kernel firmware files, after all, so
you're kinda limited to "it works fine", "some hardware doesn't work
because the firmware isn't there", and the occasional "kernel blew up
because firmware was missing which it really shouldn't do", which is
bad but at least easy to spot and workaround ("you gotta boot with the
firmware, sorry"). We could probably rely mostly on automated testing
to confirm that both images at least work properly in expected cases.
Also it's general not too onerous for us to test "some random alternate
path through the installer" because we have to run several install
tests on bare metal anyway so we can just kinda add it to the 'matrix'
there - "OK, for the firmware RAID install test I'll try booting the
no-firmware route", that knocks out two tests in one. (Of course you're
assuming that firmware RAID handling isn't somehow broken when booting
with the firmware, but that seems sufficiently unlikely not to worry
about :>)
If we want to get fancy, I suppose we could ship a single ISO, but with
two filesystem images - one the main installer environment, one
containing /lib/firmware - and just have a boot arg that (tells dracut
to) mounts the firmware one, and a boot menu option to *not* do it (not
pass the boot arg), which saves memory as long as the system doesn't
need the firmwares. But then of course someone will say "hey, why don't
we build a second ISO without the firmware image included at all, so we
have a small ISO for people who know they don't need it?" and you're
back at option 1. We also I guess have to think about how things work
for things like PXE installs, and maybe update the documentation
there...
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 12:58 +, Peter Robinson wrote: > > I've done a few passes, dropping a bunch of older firmware upstream > that are no longer supported in any stable kernel release, also a > bunch of de-dupe and linking of files rather than shipping of multiple > copies of the same firmware. It's improved things a bit, unfortunately > a lot of the dead firmware was tiny compared to say average modern > devices like GPUs or WiFI. > > The problem with a lot of the firmware, and with the new nvidia "open > driver" which shoves a lot of stuff into firmware in order to have an > upstreamable driver apparently the firmwares there are going to be > 30+Mb each, is that they're needed to bring up graphics/network etc to > even just install so I don't know how we can get around this and still > have a device work enough to be able to install the needed firmware > across the network. > > Ideas on how to solve that problem welcome. Sorry if this is way off, but - do we need the GPU firmwares to run a graphical install on the fallback path, just using the framebuffer set up by the firmware? How crazy would it be to just do that - ship the installer env with no GPU firmware? -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 12:58 PM Peter Robinson wrote: > I've done a few passes, dropping a bunch of older firmware upstream > that are no longer supported in any stable kernel release, also a > bunch of de-dupe and linking of files rather than shipping of multiple > copies of the same firmware. It's improved things a bit, unfortunately > a lot of the dead firmware was tiny compared to say average modern > devices like GPUs or WiFI. > > The problem with a lot of the firmware, and with the new nvidia "open > driver" which shoves a lot of stuff into firmware in order to have an > upstreamable driver apparently the firmwares there are going to be > 30+Mb each, is that they're needed to bring up graphics/network etc to > even just install so I don't know how we can get around this and still > have a device work enough to be able to install the needed firmware > across the network. > > Ideas on how to solve that problem welcome. Does compressing the firmware using zstd (perhaps at an aggressive level) level help at all? Yes, I know I could probably test this myself, but then it would not be just an idea (so I am offering only what was asked for ). ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 16:04 +, Peter Robinson wrote: > On Thu, Dec 8, 2022 at 3:54 PM Miroslav Suchý wrote: > > > > Dne 08. 12. 22 v 13:58 Peter Robinson napsal(a): > > > Ideas on how to solve that problem welcome. > > > > Do we need - at install time - firmware for: > > > > * v4l > > > > * dvb > > > > * cameras > > No, and we're looking at splitting those out, but the fact is they are > a tiny amount of the overall firmware collection. You could even argue > either way for something like bluetooth due to it sometimes being used > by keyboards. We actually already strip a lot of those in lorax. There are two bits of lorax templates dealing with firmware files. First, we install "*- firmware" but exclude a bunch of packages that have already been split out to help with this problem: https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-install.tmpl#L29 then, in cleanup, we wipe a bunch of files that have not yet been split out from linux-firmware but which we know the installer env doesn't need: https://github.com/weldr/lorax/blob/master/share/templates.d/99-generic/runtime-cleanup.tmpl#L201 so as I said, we're already dealing with the low-hanging fruit :( Splitting the things currently dealt with in runtime-cleanup out from linux-firmware would make things cleaner (I really hate runtime-cleanup - it's sadly necessary, but we should minimize usage of it as much as possible), but not smaller. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 3:54 PM Miroslav Suchý wrote: > > Dne 08. 12. 22 v 13:58 Peter Robinson napsal(a): > > Ideas on how to solve that problem welcome. > > Do we need - at install time - firmware for: > > * v4l > > * dvb > > * cameras No, and we're looking at splitting those out, but the fact is they are a tiny amount of the overall firmware collection. You could even argue either way for something like bluetooth due to it sometimes being used by keyboards. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Dne 08. 12. 22 v 13:58 Peter Robinson napsal(a): Ideas on how to solve that problem welcome. Do we need - at install time - firmware for: * v4l * dvb * cameras ? ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 12:31 PM Kevin Kofler via devel wrote: > IMHO, the web-based UI is a major mistake and should never be shipped in > Fedora. As I am sure you are aware, it seems a number of distros are experimenting with their next gen installer. OpenSUSE has their new web based D-installer, and Canonical is writing an installer in flutter. While I have not personally tried them out, so don't have any real opinion in whether they are good approaches, maybe eventually some of the good ideas from all the distros next generations of installer will eventually cross pollinate even though we likely all agree that there will not be one installer to rule them all. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 8 Dec 2022 at 08:15, Peter Robinson wrote: > On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson > wrote: > > > > Hi folks! Today I woke up and found > > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me > > down a bit of an "installer environment size" rabbit hole. > > > > As of today, with that new dep in webkitgtk, Rawhide's network install > > images are 703M in size. Here's a potted history of network install > > image sizes: > > > > Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M) > > Fedora 13: 208M > > Fedora 17: 162M (last "old UI") > > Fedora 18: 294M (first "new UI") > > Fedora 23: 415M > > Fedora 28: 583M > > Fedora 33: 686M > > Fedora 37: 665M > > Fedora Rawhide: 703M > > > > The installer does not really do much more in Rawhide than it did in > > FC8. Even after the UI rewrite in F18, we were only at 294M. Now the > > image is well over 2x as big and does...basically the same. > > > > Why does this matter? Well, the images being large is moderately > > annoying in itself just in terms of transfer times and so on. But more > > importantly, AIUI at least, the entire installer environment is loaded > > into RAM at startup - it kinda has to be, we don't have anywhere else > > to put it. The bigger it is, the more RAM you need to install Fedora. > > The size of the installer environment (for which the size of the > > network install image is more or less a perfect proxy) is one of the > > two key factors in this, the other being how much RAM DNF uses during > > package install. > > > > So, I did a bit of poking about into *what* is taking up all that > > space. There's a variety of answers, but there's two major culprits: > > > > 1. firmware > > 2. yelp (which pulls in webkitgtk and its deps) > > > > I've been using du and baobab (the GNOME visual disk usage analyzer, > > which is great) to examine the filesystems, but I ran a couple of test > > builds to confirm these suspects, especially after the impact of > > compression (it's hard to check the *compressed* size of things in the > > installer environment directly). > > > > I did a scratch build of lorax which does not pull in firmware > > packages, and had openQA build a netinst using that lorax. It came out > > at 489M - 214M smaller than current netinsts, a size we last managed in > > Fedora 26. I did a scratch build of anaconda with its requirement of > > yelp dropped (which would break help pages), and built a netinst with > > that; it came out at 662M - 41M smaller than current images. I haven't > > run a combined test yet, but it ought to come out around 448M, around > > the size of Fedora 24. > > > > Even then we'd still be about 50% larger than the Fedora 18 image, for > > not really any added functionality. > > > > I've moaned about the sheer amount and size of firmware blobs in other > > forums before, but 214M compressed is *really* obnoxious. We must be > > able to do something to clean this up (further than it's already > > cleaned up - this is *after* we dropped low-hanging fruit like > > enterprise switch 'firmwares' and garbage like that; most of the > > remaining size seems to be huge amounts of probably-very-similar > > firmware files for AMD graphics adapters and Intel wireless adapters). > > I know some folks were trying to work on this (there was talk that we > > could drop quite a lot of files that would only be loaded by older > > kernels no longer in Fedora); any news on how far along that effort is? > > I've done a few passes, dropping a bunch of older firmware upstream > that are no longer supported in any stable kernel release, also a > bunch of de-dupe and linking of files rather than shipping of multiple > copies of the same firmware. It's improved things a bit, unfortunately > a lot of the dead firmware was tiny compared to say average modern > devices like GPUs or WiFI. > > The problem with a lot of the firmware, and with the new nvidia "open > driver" which shoves a lot of stuff into firmware in order to have an > upstreamable driver apparently the firmwares there are going to be > 30+Mb each, is that they're needed to bring up graphics/network etc to > even just install so I don't know how we can get around this and still > have a device work enough to be able to install the needed firmware > across the network. > > Ideas on how to solve that problem welcome. > > The only ideas I have seen which 'work'* is to ship a minimal set of drivers for some 'chosen' hardware and then you have a bloated kitchen-sink iso which has all the drivers in it. The chosen hardware could be a 'defined' virtual environment which needs a minimal set of firmware, languages, etc. Everything else can be done in the install for getting languages, extra firmware, etc. The kitchen-sink.iso is going to be one which we know is going to be large. Now I have doubled the QA, releng, and product work.. I would say we would focus most of the work on the mini-installer, but we all know that all the work will be
Re: Small rant: installer environment size
On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson wrote: > > Hi folks! Today I woke up and found > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted me > down a bit of an "installer environment size" rabbit hole. > > As of today, with that new dep in webkitgtk, Rawhide's network install > images are 703M in size. Here's a potted history of network install > image sizes: > > Fedora Core 8: 103.2M (boot.iso 9.2M + stage2.img 94M) > Fedora 13: 208M > Fedora 17: 162M (last "old UI") > Fedora 18: 294M (first "new UI") > Fedora 23: 415M > Fedora 28: 583M > Fedora 33: 686M > Fedora 37: 665M > Fedora Rawhide: 703M > > The installer does not really do much more in Rawhide than it did in > FC8. Even after the UI rewrite in F18, we were only at 294M. Now the > image is well over 2x as big and does...basically the same. > > Why does this matter? Well, the images being large is moderately > annoying in itself just in terms of transfer times and so on. But more > importantly, AIUI at least, the entire installer environment is loaded > into RAM at startup - it kinda has to be, we don't have anywhere else > to put it. The bigger it is, the more RAM you need to install Fedora. > The size of the installer environment (for which the size of the > network install image is more or less a perfect proxy) is one of the > two key factors in this, the other being how much RAM DNF uses during > package install. > > So, I did a bit of poking about into *what* is taking up all that > space. There's a variety of answers, but there's two major culprits: > > 1. firmware > 2. yelp (which pulls in webkitgtk and its deps) > > I've been using du and baobab (the GNOME visual disk usage analyzer, > which is great) to examine the filesystems, but I ran a couple of test > builds to confirm these suspects, especially after the impact of > compression (it's hard to check the *compressed* size of things in the > installer environment directly). > > I did a scratch build of lorax which does not pull in firmware > packages, and had openQA build a netinst using that lorax. It came out > at 489M - 214M smaller than current netinsts, a size we last managed in > Fedora 26. I did a scratch build of anaconda with its requirement of > yelp dropped (which would break help pages), and built a netinst with > that; it came out at 662M - 41M smaller than current images. I haven't > run a combined test yet, but it ought to come out around 448M, around > the size of Fedora 24. > > Even then we'd still be about 50% larger than the Fedora 18 image, for > not really any added functionality. > > I've moaned about the sheer amount and size of firmware blobs in other > forums before, but 214M compressed is *really* obnoxious. We must be > able to do something to clean this up (further than it's already > cleaned up - this is *after* we dropped low-hanging fruit like > enterprise switch 'firmwares' and garbage like that; most of the > remaining size seems to be huge amounts of probably-very-similar > firmware files for AMD graphics adapters and Intel wireless adapters). > I know some folks were trying to work on this (there was talk that we > could drop quite a lot of files that would only be loaded by older > kernels no longer in Fedora); any news on how far along that effort is? I've done a few passes, dropping a bunch of older firmware upstream that are no longer supported in any stable kernel release, also a bunch of de-dupe and linking of files rather than shipping of multiple copies of the same firmware. It's improved things a bit, unfortunately a lot of the dead firmware was tiny compared to say average modern devices like GPUs or WiFI. The problem with a lot of the firmware, and with the new nvidia "open driver" which shoves a lot of stuff into firmware in order to have an upstreamable driver apparently the firmwares there are going to be 30+Mb each, is that they're needed to bring up graphics/network etc to even just install so I don't know how we can get around this and still have a device work enough to be able to install the needed firmware across the network. Ideas on how to solve that problem welcome. Peter ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Kevin Kofler via devel wrote: > 臘 A full-blown, 71 MB compressed (!) web browser just to show the UI for > the installer! PS: And I guess we will then also need to ship the langpacks, which are another 42 MB compressed! (And in one monolithic package for all languages.) Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Adam Williamson wrote: > On Thu, 2022-12-08 at 07:57 +0100, Florian Weimer wrote: >> It has all the targets in it. As it's for JIT, we'd only need one >> target. > > That sounds interesting, though of course the details of how to > implement it could be a bit tricky, I guess... Build /usr/lib64/libLLVM-15.so with only the host as the target, and a renamed /usr/lib64/libLLVM-cross-15.so with all the targets. Link the compilers against the latter. Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Tomáš Popela wrote: > firefox, because that's what the web based installer should/will use in > the end 臘 A full-blown, 71 MB compressed (!) web browser just to show the UI for the installer! IMHO, the web-based UI is a major mistake and should never be shipped in Fedora. It is good as a prototype, but way too resource-heavy to be shipped in production. We need a rewrite of the UI design in a desktop toolkit. (If GTK is not suitable, how about Qt? ☺) Kevin Kofler ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
Hi Adam, On Thu, Dec 8, 2022 at 5:30 AM Adam Williamson wrote: > On Thu, 2022-12-08 at 03:28 +, Gary Buhrmaster wrote: > > On Thu, Dec 8, 2022 at 12:42 AM Adam Williamson > > wrote: > > > > > > Hi folks! Today I woke up and found > > > https://bugzilla.redhat.com/show_bug.cgi?id=2151495 , which diverted > me > > > down a bit of an "installer environment size" rabbit hole. > > > > Does the "new and improved" web based installer help this > > in any way? > > I haven't looked yet but I suspect it'll probably be a wash, mostly. If > anything it's likely slightly negative because the new installer itself > hard requires webkitgtk, so we can't really do anything to finesse that > requirement any more. With the old installer, we could maybe try and > figure out some way of being able to show the help pages without > needing yelp/webkitgtk. Since the new installer itself needs webkitgtk, > seems like there's no way we're getting rid of that ~40M compressed. > You should also try to do this - remove webkitgtk and yelp packages and add firefox, because that's what the web based installer should/will use in the end - see https://bugzilla.redhat.com/show_bug.cgi?id=2142779. We already had a meeting with Anaconda dev and told him to use Firefox instead of WebKitGTK for the web installer - due to resources that are internally (and frankly in upstream as well) allocated to WebKitGTK. Tom ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: Small rant: installer environment size
On Thu, 2022-12-08 at 07:57 +0100, Florian Weimer wrote: > * Adam Williamson: > > > 1. /usr/lib/locale/locale-archive , from glibc-all-langpacks - this is > > 224M uncompressed. A quick test just compressing the file with xz on my > > system shows it compresses to around 11M, though, so that's probably > > all it adds up to after compression (the image is an xz-compressed > > squashfs) > > Isn't the compression block-based? I think it would be interesting to > measure the image size with the file removed. I'll try it tomorrow, it's not too hard. > > For the non-live installer, we can *significantly* cut down its size, > without degrading localization of the installer itself. > > > 2. /usr/lib64/libLLVM-15.so, which is 114M on its own, compresses to > > 23M. We are, I think, basically stuck with this for mesa-dri-drivers , > > but does it have to be so *big*? > > It has all the targets in it. As it's for JIT, we'd only need one > target. That sounds interesting, though of course the details of how to implement it could be a bit tricky, I guess... Thanks for the ideas! -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
