[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
This fix will be included in grub2-unsigned/2.06-2ubuntu14.1 for kinetic, jammy, focal, bionic. Binaries have been built in https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/ppa/+packages for kinetic. Signing request: https://answers.launchpad.net/canonical-signing- jobs/+question/704589 ** Also affects: initramfs-tools (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: grub2-signed (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: grub2-unsigned (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: initramfs-tools (Ubuntu Jammy) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Jammy) Importance: Undecided Status: New ** Also affects: grub2-signed (Ubuntu Jammy) Importance: Undecided Status: New ** Also affects: grub2-unsigned (Ubuntu Jammy) Importance: Undecided Status: New ** Also affects: initramfs-tools (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: grub2-signed (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: grub2-unsigned (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: initramfs-tools (Ubuntu Kinetic) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Kinetic) Importance: Undecided Status: New ** Also affects: grub2-signed (Ubuntu Kinetic) Importance: Undecided Status: New ** Also affects: grub2-unsigned (Ubuntu Kinetic) Importance: Undecided Status: New ** No longer affects: initramfs-tools (Ubuntu Kinetic) ** No longer affects: initramfs-tools (Ubuntu Jammy) ** No longer affects: initramfs-tools (Ubuntu Focal) ** No longer affects: initramfs-tools (Ubuntu Bionic) ** No longer affects: linux (Ubuntu) ** No longer affects: initramfs-tools (Ubuntu) ** No longer affects: linux (Ubuntu Bionic) ** No longer affects: linux (Ubuntu Focal) ** No longer affects: linux (Ubuntu Jammy) ** No longer affects: linux (Ubuntu Kinetic) ** Description changed: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
The terrible thing with compression is how we know of no universal rule. I'm sure you can even find non-pathological cases where lz4 compresses better than zpaq (and does so 100 times faster). And that's without taking I/O into account (or filters). An important thing to keep in mind here is that zstd currently uses its -1 preset while xz uses its default of -6. That means that these two tools which are often in the same ballpark are configured at vastly different compression/speed setting. What's important is that you get something that works for your setup and as I said above: we don't have a rule for optimal settings (and it's probably impossible to find one). One last note: xz decompression speed depends on the compressed size, not uncompressed size. This means that a more-compressed file decompresses faster. This might be the same with zstd since they're related compressors but I'm not 100% sure. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Released Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF 0002 000F BS_Data2B633000-2B63CFFF 000A 000F Available 2B63D000-2B649FFF 000D 000F BS_Data2B64A000-2B64EFFF 0005 000F Available 2B64F000-2B666FFF 0018 000F BS_Data2B667000-2D8D5FFF 226F 000F LoaderCode 2D8D6000-2D8E9FFF 0014 000F BS_Data2D8EA000-2D925FFF 003C 000F LoaderCode 2D926000-2D932FFF 000D 000F BS_Data2D933000-2D969FFF 0037 000F BS_Code
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
@madzohan from my comment around #41 above and related workarounds from there downward, it seems you tried a different compression scheme and found a smaller size in this case from zstd to xz. I looked over your settings and options for xz compression, but also reading the man pages for XZ it shows that it is already at a -6 setting by default that makes the dictionary size 8M and uses lzma2 compression. XZ settings are already at default in /etc/sbin/mkinitramfs . With your options it was a few MB larger than from xz default in my setup. I tried both xz settings. Finally, I switched back to zstd and my compression changes I made in /etc/sbin/mkinitramfs as my workaround. I got a good compression size, but people note online that zstd has great read speeds over xz no matter the compression level. So I decided to stay at zstd with only a 7 MB difference: from 125 MB on default xz to 131 MB on zstd with -q 22 compression in /etc/sbin/mkinitramfs . -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Released Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF 0002 000F BS_Data2B633000-2B63CFFF 000A 000F Available 2B63D000-2B649FFF 000D 000F BS_Data2B64A000-2B64EFFF 0005 000F Available 2B64F000-2B666FFF 0018 000F BS_Data2B667000-2D8D5FFF 226F 000F LoaderCode 2D8D6000-2D8E9FFF 0014 000F BS_Data2D8EA000-2D925FFF 003C 000F LoaderCode 2D926000-2D932FFF 000D 000F BS_Data2D933000-2D969FFF 0037 000F
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hi guys, I want to share my success story: It happened after I've updated to CUDA repo's nvidia drivers :) my `initrd.img-5.15.0-58-generic` (`ls -lah /boot`) was about 192M and using tuning option of the `/etc/initramfs-tools/initramfs.conf` file I've decreased it (`sudo update-initramfs -c -k 5.15.0-58-generic`) to 116M and now it works =) my `/etc/initramfs-tools/initramfs.conf` ``` # # initramfs.conf # Configuration file for mkinitramfs(8). See initramfs.conf(5). # # Note that configuration options from this file can be overridden # by config files in the /etc/initramfs-tools/conf.d directory. # # MODULES: [ most | netboot | dep | list ] # # most - Add most filesystem and all harddrive drivers. # # dep - Try and guess which modules to load. # # netboot - Add the base modules, network modules, but skip block devices. # # list - Only include modules from the 'additional modules' list # # original #MODULES=most # patch MODULES=dep # # BUSYBOX: [ y | n | auto ] # # Use busybox shell and utilities. If set to n, klibc utilities will be used. # If set to auto (or unset), busybox will be used if installed and klibc will # be used otherwise. # BUSYBOX=auto # # COMPRESS: [ gzip | bzip2 | lz4 | lzma | lzop | xz | zstd ] # # original # COMPRESS=zstd COMPRESS=xz # # DEVICE: ... # # Specify a specific network interface, like eth0 # Overridden by optional ip= or BOOTIF= bootarg # DEVICE= # # NFSROOT: [ auto | HOST:MOUNT ] # NFSROOT=auto # # RUNSIZE: ... # # The size of the /run tmpfs mount point, like 256M or 10% # Overridden by optional initramfs.runsize= bootarg # RUNSIZE=10% # # FSTYPE: ... # # The filesystem type(s) to support, or "auto" to use the current root # filesystem type # FSTYPE=auto # patch export XZ_OPT='--lzma2=preset=0,dict=8M' ``` Hope it will help you (and don't forget about `sudo update-initramfs -c -k YOUR_VERSION-generic`). Keep yourself safe (from kernel panics too) and Glory to Ukraine! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Released Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
The entire upload will be uploaded to all older releases but we do have a security upload already in the unapproved queues that should go out first, but that needs approving those first, them passing the SRU verification, a resigning against the new signing key the new shim needs, and finally someone to release them. So that the security update can be released to security pocket. This won't be ready before Monday the week after next week at the earliest (7 day testing period in proposed). People that can approve are still out and only return next week, so I don't think that's entirely realistic we get the existing uploads into proposed on Monday. If we could get them in by Thursday they could go out Monday the 23rd and we may be able to then upload this fix and release it on like 31st or so. We're strictly limited by the point release - it needs to be ready for that. ** Changed in: grub2-signed (Ubuntu) Milestone: None => ubuntu-22.04.2 ** Changed in: grub2-unsigned (Ubuntu) Milestone: None => ubuntu-22.04.2 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Released Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF 0002 000F BS_Data2B633000-2B63CFFF 000A 000F Available 2B63D000-2B649FFF 000D 000F BS_Data2B64A000-2B64EFFF 0005 000F Available 2B64F000-2B666FFF 0018 000F BS_Data2B667000-2D8D5FFF 226F 000F LoaderCode 2D8D6000-2D8E9FFF 0014 000F BS_Data2D8EA000-2D925FFF 003C 000F LoaderCode 2D926000-2D932FFF 000D 000F
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Backporting to LTS is a good idea IMHO. A few additional use cases to consider - Users on the threshold who experience slight initrd image growth due to any module update - Users who thought they might be able to continue just using older images may fail to recognize grub will eventually remove those older images during regular updates. - Users that take their laptops on the road, but then plug them into their dock with larger displays to get the real work done only to have a boot failure. And yet when they take it into Best Buy it just works somehow. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Released Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF 0002 000F BS_Data2B633000-2B63CFFF 000A 000F Available 2B63D000-2B649FFF 000D 000F BS_Data2B64A000-2B64EFFF 0005 000F Available 2B64F000-2B666FFF 0018 000F BS_Data2B667000-2D8D5FFF 226F 000F LoaderCode 2D8D6000-2D8E9FFF 0014 000F BS_Data2D8EA000-2D925FFF 003C 000F LoaderCode 2D926000-2D932FFF 000D 000F BS_Data2D933000-2D969FFF 0037 000F BS_Code2D96A000-2D973FFF 000A 000F BS_Data2D974000-2E377FFF 0A04 000F Available 2E378000-2E37AFFF 0003 000F ... Reserved 3C08B000-4192 58A5 000F ACPI_NVS 4193-41B2 0200
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I second backporting this to jammy at least as it is an LTS. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Released Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF 0002 000F BS_Data2B633000-2B63CFFF 000A 000F Available 2B63D000-2B649FFF 000D 000F BS_Data2B64A000-2B64EFFF 0005 000F Available 2B64F000-2B666FFF 0018 000F BS_Data2B667000-2D8D5FFF 226F 000F LoaderCode 2D8D6000-2D8E9FFF 0014 000F BS_Data2D8EA000-2D925FFF 003C 000F LoaderCode 2D926000-2D932FFF 000D 000F BS_Data2D933000-2D969FFF 0037 000F BS_Code2D96A000-2D973FFF 000A 000F BS_Data2D974000-2E377FFF 0A04 000F Available 2E378000-2E37AFFF 0003 000F ... Reserved 3C08B000-4192 58A5 000F ACPI_NVS 4193-41B2 0200 000F ACPI_Recl 41B3-41BFEFFF 00CF 000F BS_Data41BFF000-41BF 0001 000F Available 0001-0002AB7F 001AB800 000F Reserved 000A-000F 0060 Reserved 41C0-43FF 2400 Reserved 4400-47FF 4000 000F
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Tags added: originate-from-2000298 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Released Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF 0002 000F BS_Data2B633000-2B63CFFF 000A 000F Available 2B63D000-2B649FFF 000D 000F BS_Data2B64A000-2B64EFFF 0005 000F Available 2B64F000-2B666FFF 0018 000F BS_Data2B667000-2D8D5FFF 226F 000F LoaderCode 2D8D6000-2D8E9FFF 0014 000F BS_Data2D8EA000-2D925FFF 003C 000F LoaderCode 2D926000-2D932FFF 000D 000F BS_Data2D933000-2D969FFF 0037 000F BS_Code2D96A000-2D973FFF 000A 000F BS_Data2D974000-2E377FFF 0A04 000F Available 2E378000-2E37AFFF 0003 000F ... Reserved 3C08B000-4192 58A5 000F ACPI_NVS 4193-41B2 0200 000F ACPI_Recl 41B3-41BFEFFF 00CF 000F BS_Data41BFF000-41BF 0001 000F Available 0001-0002AB7F 001AB800 000F Reserved 000A-000F 0060 Reserved 41C0-43FF 2400 Reserved 4400-47FF 4000 000F Reserved
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hi, There are a few OEM platforms that also requires these memory patches in grub2 on jammy, could we also port 2.06-2ubuntu16 back to jammy? ** Tags added: originate-from-1998995 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Released Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF 0002 000F BS_Data2B633000-2B63CFFF 000A 000F Available 2B63D000-2B649FFF 000D 000F BS_Data2B64A000-2B64EFFF 0005 000F Available 2B64F000-2B666FFF 0018 000F BS_Data2B667000-2D8D5FFF 226F 000F LoaderCode 2D8D6000-2D8E9FFF 0014 000F BS_Data2D8EA000-2D925FFF 003C 000F LoaderCode 2D926000-2D932FFF 000D 000F BS_Data2D933000-2D969FFF 0037 000F BS_Code2D96A000-2D973FFF 000A 000F BS_Data2D974000-2E377FFF 0A04 000F Available 2E378000-2E37AFFF 0003 000F ... Reserved 3C08B000-4192 58A5 000F ACPI_NVS 4193-41B2 0200 000F ACPI_Recl 41B3-41BFEFFF 00CF 000F BS_Data41BFF000-41BF 0001 000F Available 0001-0002AB7F 001AB800 000F Reserved 000A-000F 0060 Reserved 41C0-43FF
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
This bug was fixed in the package grub2-unsigned - 2.06-2ubuntu16 --- grub2-unsigned (2.06-2ubuntu16) lunar; urgency=medium * Cherry-pick all memory patches from rhboot - Allocate initrd > 4 GB (LP: #1842320) - Allocate kernels as code, not data (needed for newer firmware) * ubuntu: Fix casts on i386-efi target * Cherry-pick all the 2.12 memory management changes (LP: #1842320) * Allocate executables as CODE, not DATA in chainloader and arm64 * Source package generated from src:grub2 using make -f ./debian/rules generate-grub2-unsigned -- Julian Andres Klode Fri, 09 Dec 2022 17:11:44 +0100 ** Changed in: grub2-unsigned (Ubuntu) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Released Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF 0002 000F BS_Data2B633000-2B63CFFF 000A 000F Available 2B63D000-2B649FFF 000D 000F BS_Data2B64A000-2B64EFFF 0005 000F Available 2B64F000-2B666FFF 0018 000F BS_Data2B667000-2D8D5FFF 226F 000F LoaderCode 2D8D6000-2D8E9FFF 0014 000F BS_Data2D8EA000-2D925FFF 003C 000F LoaderCode 2D926000-2D932FFF 000D 000F BS_Data2D933000-2D969FFF 0037 000F BS_Code2D96A000-2D973FFF 000A 000F BS_Data2D974000-2E377FFF 0A04 000F Available 2E378000-2E37AFFF 0003
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Can confirm on 22.10, Lenovo Yoga 720 (4k display). Is there no fix for this other than trying to mess with initramfs compression settings? Is that expected to be the normal user case? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Committed Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory region starvation in <1G addresses: Type StartEnd # Pages Attributes Available -00086FFF 0087 000F BS_Data00087000-00087FFF 0001 000F Available 00088000-0009EFFF 0017 000F Reserved 0009F000-0009 0001 000F Available 0010-00FF 0F00 000F LoaderCode 0100-01021FFF 0022 000F Available 01022000-238A7FFF 00022886 000F BS_Data238A8000-23927FFF 0080 000F Available 23928000-28860FFF 4F39 000F BS_Data28861000-2AB09FFF 22A9 000F LoaderCode 2AB0A000-2ACF8FFF 01EF 000F BS_Data2ACF9000-2B2FAFFF 0602 000F Available 2B2FB000-2B611FFF 0317 000F BS_Data2B612000-2B630FFF 001F 000F Available 2B631000-2B632FFF 0002 000F BS_Data2B633000-2B63CFFF 000A 000F Available 2B63D000-2B649FFF 000D 000F BS_Data2B64A000-2B64EFFF 0005 000F Available 2B64F000-2B666FFF 0018 000F BS_Data2B667000-2D8D5FFF 226F 000F LoaderCode 2D8D6000-2D8E9FFF 0014 000F BS_Data2D8EA000-2D925FFF 003C 000F LoaderCode 2D926000-2D932FFF 000D 000F BS_Data2D933000-2D969FFF 0037 000F BS_Code2D96A000-2D973FFF 000A 000F BS_Data2D974000-2E377FFF 0A04 000F Available 2E378000-2E37AFFF 0003 000F ... Reserved 3C08B000-4192 58A5 000F ACPI_NVS 4193-41B2 0200 000F ACPI_Recl 41B3-41BFEFFF 00CF 000F BS_Data41BFF000-41BF 0001 000F Available 0001-0002AB7F 001AB800 000F Reserved 000A-000F 0060 Reserved
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Description changed: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` - Based on grub_mm_dump, we can see the memory fragment (some parts seem - likely be used because of 4K resolution?) and doesn’t have available - contiguous memory for larger file as: - - ``` - grub_real_malloc(...) + Based on grub_mm_dump, we can see the memory region starvation in <1G + addresses: + + Type StartEnd # Pages Attributes + Available -00086FFF 0087 000F + BS_Data00087000-00087FFF 0001 000F + Available 00088000-0009EFFF 0017 000F + Reserved 0009F000-0009 0001 000F + Available 0010-00FF 0F00 000F + LoaderCode 0100-01021FFF 0022 000F + Available 01022000-238A7FFF 00022886 000F + BS_Data238A8000-23927FFF 0080 000F + Available 23928000-28860FFF 4F39 000F + BS_Data28861000-2AB09FFF 22A9 000F + LoaderCode 2AB0A000-2ACF8FFF 01EF 000F + BS_Data2ACF9000-2B2FAFFF 0602 000F + Available 2B2FB000-2B611FFF 0317 000F + BS_Data2B612000-2B630FFF 001F 000F + Available 2B631000-2B632FFF 0002 000F + BS_Data2B633000-2B63CFFF 000A 000F + Available 2B63D000-2B649FFF 000D 000F + BS_Data2B64A000-2B64EFFF 0005 000F + Available 2B64F000-2B666FFF 0018 000F + BS_Data2B667000-2D8D5FFF 226F 000F + LoaderCode 2D8D6000-2D8E9FFF 0014 000F + BS_Data2D8EA000-2D925FFF 003C 000F + LoaderCode 2D926000-2D932FFF 000D 000F + BS_Data2D933000-2D969FFF 0037 000F + BS_Code2D96A000-2D973FFF 000A 000F + BS_Data2D974000-2E377FFF 0A04 000F + Available 2E378000-2E37AFFF 0003 000F ... - if (cur->size >= n + extra) - ``` + Reserved 3C08B000-4192 58A5 000F + ACPI_NVS 4193-41B2 0200 000F + ACPI_Recl 41B3-41BFEFFF 00CF 000F + BS_Data41BFF000-41BF 0001 000F + Available 0001-0002AB7F 001AB800 000F + Reserved 000A-000F 0060 + Reserved 41C0-43FF 2400 + Reserved 4400-47FF 4000 000F + Reserved 4940-495F 0200 000F + Reserved 4C00-4FFF 4000 0009 + Reserved 5000-547F 4800 + MMIO C000-CFFF 0001
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
juliank, edited and removed more notes as you advised. I hope my edits per your advice help clarify things and help anyone to see if a new bug is necessary to report. Thanks for your input. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Committed Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
The term you're looking for is workaround, not triage or fixes. Triage is the process of understanding the bug and marking it the correct things. Not using the same language as everyone else and writing very long comments makes it hard to understand. First I thought you reported an out of memory issue on Pis. Then I thought you were saying that the fix doesn't help. It took me a long while to realize it was completely irrelevant for the process of resolving that bug. If the initrd grows significantly please open a separate bug against linux so that is tracked - maybe something went wrong - and leave a comment here. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Committed Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hi juliank, I understand completely you want me to have insights related to verification of the update in lunar-proposed and that I am confusing you with my #131 post. I was mentioning that to help people who need the triage, since it is not possible to say locked out nor without updates until lunar is released. The reasons for #131 is to note the original issue that brought out our understanding of this problem, liked in my #45, that when Raspberry Pi needs changed the compression, we suddenly realized that for everyone else grub had a memory issue on some systems since we were not compressing it as much. One of the triages I shared out of 2 fixed things for some time, but after the latest kernel update to 5.19.0-26 from 5.19.0-23, the initrd image grows significantly and both triages--compression and module dependencies, are necessary on my system. So in cases of knowing that it is fixed properly, great. In cases where we don't have an update that may be backported, we are lost in 2 triages that people need to find instructions and help on that are here--- therefore my post on #131 you noted was not relevant. When you install 5.19.0-26, you will need the triages on initrd. On 5.19.0-23, the initrd is much, much smaller. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Committed Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 /
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hi ybdjkfd I don't really understand what you are doing and how it's relevant. This issue should be fixed in lunar-proposed 2.06-2ubuntu16, please only add comments if you have meaningful insights regarding the verification of that update. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Committed Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Bringing one the original triage fixes I posted in #41 and #45 after gathering some information on changes to grub2 for Raspberry Pi that exposed this issue (i.e., I know the fix to get things to boot), after upgrading to 22.10, I now have an updated set of events on a specific kernel. I still have the fixes in my system before restarting (after the upgrade) so that the compression and module dependency setup are triaged and run with an update grub command as the final stop. Later on, after an upgrade to the newest kernel 5.19.0-2x-generic, I get on grub: error: out of memory Press any key to boot . . . However, pressing a key will boot forward normally. My /boot/initrd.img-5.19.0-26-generic size is 131 Mb. This happens only with this new kernel. The previous one did not have this similar error show. I thought I would be locked out as before since the triage did not prevent the error message. It may be a new cause for the error than what is originally bothering this bug. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Committed Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Changed in: grub2-unsigned (Ubuntu) Status: Triaged => Fix Committed ** Changed in: grub2-signed (Ubuntu) Status: Triaged => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Fix Committed Status in grub2-unsigned package in Ubuntu: Fix Committed Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 *
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I'm insanely late on this, but I couldn't help but notice that nobody mentioned Ventoy (at least ctrl+f doesn't return any results in the full activity log for "ventoy") Since installing via Rufus set to ISO mode has been a suggested work- around, wouldn't this by definition mean that using Ventoy would "just work"? Normally I would test this myself and report back, but I'm not actually running into this issue myself, so... Link for those that aren't aware of what Ventoy is: https://github.com/ventoy/Ventoy -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I have now picked all the rhboot patches in 2.06-2ubuntu16~ppa1 in https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/ppa/+packages and will boot that before uploading to the archive. This *should* allow initrds over 4GB but obviously bugs could be there. The important bit is that we are now close to the rhboot branch again in that file such that the whole thing is maintainable to some extent at least. Once this is uploaded to lunar people can test this easily on secure boot systems without having to install PPA signing keys in MOK. I'm gonna see if I can run some tests myself and reproduce this. Also it turns out it fails to build the i386-efi target due to pointer size mismatches. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Tags added: originate-from-1998320 somerville -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Jeremy, there are duplicate firmware files. Replacing duplicates with symlinks is probably the easiest and most efficient way to improve the situation. I get the following: > % jdupes -mrS /lib/firmware > Scanning: 2830 files, 286 items (in 1 specified) > 405 duplicate files (in 212 sets), occupying 37 MB With "jdupes -rl", "tar c /lib/firmware | zstd -1" drops from 428MB to 411MB. The effect on initrds is maybe slightly lower iirc but it's still noticeable and it should be very easy and safe for us. I can't tell exactly how many more people will be able to boot with that change but I think the cost is small enough that it's worth trying. Worst case, every kernel update is faster. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Workaround] Some workarounds have been suggested in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Description changed: + [Workaround] + + Some workarounds have been suggested in + https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/125 + [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case: `/var/cache/pbuilder/build/276481/build/grub2-2.06/obj/monolithic/grub- efi-amd64/grubx64.efi` * My build command: `sudo PBSHELL=1 pbuilder build --hookdir ~/hook-dir ubuntu-grub/grub2_2.06-2ubuntu7+jeremydev2.dsc 2>&1 | tee build.log` * My qemu command: `qemu-system-x86_64 -bios edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd -hda Templates/grub.qcow2 -m 6G -vga cirrus -smp 8 -machine
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Ah, didn't realize Debian had a different limit. Thanks @juliank. @anourzad, I'm not sure I would want to start adding custom steps to the kernel update path. I'm much more inclined to add an /etc option for a supported automatic mechanism, like compression or module selection. @adrien-n, great notes in #125, I'm sure this will help others. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Is it possible if we have an overwrite in debian/rule by using something like DEB_BUILD_ARCH to specify higher compression rate for amd64? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I put together some notes and work-arounds in order to provide a simpler reference for people hitting this issue. I didn't test everything below but nothing should be risky. # Summary Grub attempts to read the initrd into a memory location that is too small. This issue is caused by a combination of several factors: - Grub not setting aside enough space for the initrd in memory and not at the right location, - Initrds having grown due to holding more modules and more firmware. This is especially the case with the nvidia proprietary driver. - Typical screen resolutions and their associated buffers in memory having increased. A related issue is not having enough space in /boot to hold enough initrds. # Solutions There are four ways to tackle these issues: - Patch grub, - Have fewer modules, - Compress initrds more, - Lower screen resolution at boot. ## Patch grub This is being worked on. It’s the trickier option so I’m not going to provide details but you can find that in the discussion above. ## Have fewer modules The default configuration creates initrds that are meant to be universal in order to accommodate as many hardware variants as possible. Unsurprisingly this makes initrds much bigger. It is possible to change the value of MODULES to ‘dep’ in /etc/initramfs-tools/initramfs.conf . This will drastically reduce the initrd size in virtually every configuration at the cost of not supporting during boot hardware that is not plugged when the initrd generation takes place. ## Compress initrds more Change the value of COMPRESS in /etc/initramfs-tools/initramfs.conf . In terms of compression level, you will have something along the lines of the following: lz4: 74066711 zstd: 54260379 gzip: 53829310 xz: 29665544 NB: these values are only valid on one specific machine and configuration; they are only meant to give an idea of compression ratio that can be obtained but the initrd uses MODULES=dep as outlined above. With a small hack, it is also possible to make xz compression much faster at a small cost in compression by adding the following at the end of initramfs.conf: export XZ_OPT='--lzma2=preset=0,dict=8M' In the future, it will be possible to directly set compression levels for every compression method. ## Lower screen resolution at boot You can use a lower resolution screen when booting. You can also edit /etc/default/grub and use the following: GRUB_GFXMODE=640x480 Remember to run update-grub afterwards. If you need to do it at boot-time, you will use GFXMODE instead (no ‘GRUB_’ prefix). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
@Craig The limit in Debian is actually much lower than in Ubuntu even, but fixing it there is even harder as that misses a lot more patches. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
can this be a solution ? https://stackoverflow.com/questions/2895816/how-do-i-strip-local- symbols-from-linux-kernel-module-without-breaking-it -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Confirmed that reducing the size of the initrd.img slightly allowed my machine to boot. I just enabled `COMPRESSLEVEL=19` with `COMPRESS=zstd` (default on debian) in initramfs.conf. That reduced my initrd.img size from 72MB to 62MB, which allowed my machine to boot. Ultimately, I set `MODULES=dep` (instead of `MODULES=most`) and reverted the compression to defaults. This reduced my initrd.img to 22MB, which includes Intel microcode (specific binary), nvme, crypto, and lvm support, among other things. --- For the record: My machine is running Debian bookworm with the Nvidia proprietary drivers with 32GB. I first noticed the issue when, coincidentally, upgrading from kernel 5.19 to 6.0. ``` Loading Linux 6.0.0-4amd64 ... Loading initial ramdisk ... error: out of memory. Press any key to continue... ``` -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
The issue happened for me after installing newer nvidia proprietary drivers (515 and also 520). The only way to recover was to boot using a previous installed kernel (I used rescue mode. The rescue mode in current kernel still gives the same error) Not sure how this helps this bug investigation but was real pain knowing that I couldn't install the nvidia drivers. The only thing that worked was installing the nvidia drivers 470 version. https://forums.developer.nvidia.com/t/ubuntu-22-04-1-nvidia-driver-open- kernel-nvidia-driver-515-open-issue/231356/12?u=yohn.krichton -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Changed in: grub2-unsigned (Ubuntu) Assignee: (unassigned) => Julian Andres Klode (juliank) ** Changed in: grub2-signed (Ubuntu) Assignee: (unassigned) => Julian Andres Klode (juliank) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Tags added: fr-2934 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case: `/var/cache/pbuilder/build/276481/build/grub2-2.06/obj/monolithic/grub-
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
-> If an user somehow met this issue, then the system is not able to boot. It can boot, if it did previously, and this issue will get fixed pretty quick IMHO in the updates repository given it impacts current 20.04, 22.04 and 22.10. Here is just the workaround part, for current 22.10 or 22.04 to help people stuck on boot that used to have a working system. 1) Hold the SHIFT key on boot, and select the 2nd newest kernel from the advanced list in the grub menu: i.e. for myself, 5.19 broke, so using the older 5.15 allowed me to boot. 2) Open a terminal, sudo -i to root, and run these commands to download, install, and prevent upgrades to last version of initramfs-tools before the compression rates were changed, which allows you to regenerate your initrd image with the previously working smaller size. Download: wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools-bin_0.140ubuntu12_amd64.deb wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools-core_0.140ubuntu12_all.deb wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools_0.140ubuntu12_all.deb Install: dpkg -i initramfs-tools-bin_0.140ubuntu12_amd64.deb dpkg -i initramfs-tools-core_0.140ubuntu12_all.deb dpkg -i initramfs-tools_0.140ubuntu12_all.deb Hold Upgrades: apt-mark hold initramfs-tools-bin apt-mark hold initramfs-tools-core apt-mark hold initramfs-tools Re-generate current initrd image: update-initramfs -u A reboot should just bring you up in 5.19. To restore the system to a normal state once the problem has been fixed in the current versions, unhold those same packages using the steps below, and the new ones will automatically be downloaded and installed. apt-mark unhold initramfs-tools-bin apt-mark unhold initramfs-tools-core apt-mark unhold initramfs-tools Some additional notes, not for the workaround above follow. Do NOT issue the "update-initramfs -k all" command, if you have been thinking about it as a first triage step. The reason being, if you have not applied some kind of fix or workaround prior, this will rebuild all your old previously working kernel initrd images with the new low compression directive, which might render them all unbootable, further complicating matters. If you get to this state, and you need a working live ISO, to chroot and fix, make sure to choose one prior with the older, increased initrd compresssion which must be 20.04.4 LTS or older. Current 20.05.5, and all 22.04 and 22.10 releases use the lower compression that can cause this issue and may not work. Impatient new installers may find this info useful as well. Another simpler workaround for more advanced users with 4K screens is to pass GFXMODE=640x480 in grub defaults and update-grub as mentioned here: http://savannah.gnu.org/bugs/?61058 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2],
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I don't think "MODULES=dep" is the proper workaround for this. If an user somehow met this issue, then the system is not able to boot. Users need to do many extra works to find out this workaround and do apply unless we make "MODULES=dep" as default setting in initramfs configs (but I don't think it's doable). Based on #117, we also confirm the stock ubuntu not able to boot on some machines. The fix needs to be included in the packages from ubuntu-archive as common solution. Julian, Can we go back to consider #105, #106 to try to land it in Lunar or Kinetic first? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access.
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
My XPS 13 9380 (no luks) also triggers this on 5.19 after the 22.10 upgrade; I can see the initrd is about 7% bigger in 22.10 over 22.04. -rw-r--r-- 1 root root 111203941 Oct 30 18:16 initrd.img-5.15.0-52-generic -rw-r--r-- 1 root root 120222346 Oct 31 21:39 initrd.img-5.19.0-23-generic #103 lays out some interesting blockers for grub2, so as of right now there is no simple way to get upstream through ubuntu repositories as grub2 2.06-2ubuntu12 does not have a fix. I think it is interesting that in https://lists.gnu.org/archive/html/grub-devel/2017-03/msg00037.html there was pushback on autodetection to fix this, citing "that machine that nobody heard of that will fail" however it ended up being a bunch of not-yet-built systems that failed Looking at initramfs-tools in Jammy, the very last changelog entry was the change to lower compression, that is triggering this issue: http://changelogs.ubuntu.com/changelogs/pool/main/i/initramfs- tools/initramfs-tools_0.140ubuntu13/changelog initramfs-tools (0.140ubuntu13) jammy; urgency=medium * Lower the compression levels for zstd and lz4 (LP: #1958148) Following the discussion on the mailing list, we have reached a conclusion to lower the default compression levels: - For lz4, the compression level is lowered to 2 from 9 - For zstd, the compression level is lowered to 1 from 19 Simple fix IMHO is to just go back one version. Ubuntu main archive doesn't include all old versions, but other mirrors do. Here are some quick steps that let you run official binaries from Jammy under Kinetic to work around this issue until grub2 catches up and/or they tweak compression, or make compression more easily tunable. wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools-bin_0.140ubuntu12_amd64.deb wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools-core_0.140ubuntu12_all.deb wget https://ubuntu.repo.cure.edu.uy/mirror/pool/main/i/initramfs-tools/initramfs-tools_0.140ubuntu12_all.deb dpkg -i initramfs-tools-bin_0.140ubuntu12_amd64.deb dpkg -i initramfs-tools-core_0.140ubuntu12_all.deb dpkg -i initramfs-tools_0.140ubuntu12_all.deb apt-mark hold initramfs-tools-bin apt-mark hold initramfs-tools-core apt-mark hold initramfs-tools update-initramfs -u Afterwards, big drop in initrd size. -rw-r--r-- 1 root root 111203941 Oct 30 18:16 initrd.img-5.15.0-52-generic -rw-r--r-- 1 root root 79034965 Oct 31 22:50 initrd.img-5.19.0-23-generic Sometime down the road, when this bug is closed, you will want to un- hold these to restore the lastest packages apt-mark unhold initramfs-tools-bin apt-mark unhold initramfs-tools-core apt-mark unhold initramfs-tools -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Download ubuntu-20.04.5-desktop-amd64.iso and try to install on a HP laptop, I can't boot into installation menu. If initrd.img lower than 100MB then everything is fine. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
If possible, I still expect we can make grub supports to boot from bigger initramfs. There will still have users to meet this issue in the future if their initramfs somehow bigger as long as we don't choose use higher compression as workaround (for ubuntu desktop / server). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Tags added: originate-from-1994098 stella -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
initramfs-tools 0.142 added support for specifying the compression level. Ubuntu will probably merge this version for Ubuntu 23.04. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I upgraded from 22.04.1 to 22.10. Issue returned as /usr/sbin/mkinitramfs was changed to default. Had to follow my previous steps shown after #41 . . . zstd had defaulted to a lower compression value in /usr/sbin/mkinitramfs at 1 instead of 19. I changed line 196 to 19 compression zstd) compress="zstd -q -19 -T0" ;; Instead of zstd) compress="zstd -q -1 -T0" ;; Problem solved from proper chroot and updating grub2 after changing line 196 of /usr/sbin/mkinitramfs for my system. I can boot normally again and any upgrades to initramfs don't cause this failure. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access.
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
As I've said elsewhere, if we dedup firmware files through symlinks, we can save 10MB in initrds. Compression does not help because the compressors have very small compression windows and cannot see redundancy in practice (this applies to xz to a lesser extent but even for xz there is an improvement to be gained). I don't know if it will be enough but it could tip us just below some threshold and it's easy to implement. It should also save memory at runtime. And for reference, there's a working but weird work-around that only touches initramfs.conf, keeps MODULES=most and is reasonably fast. Use 'xz' as the compress program and then add the following on a new line: export XZ_OPT=--lzma2=preset=0,dict=16M (yes, it relies on implementation details of mkinitramfs) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Really the workaround is to set MODULES=dep in /etc/initramfs- tools/initramfs.conf, this yields a much smaller initramfs (but you can't take out the disk and boot it in another machine), disabling secure boot or sgx should not be doing anything. ** Also affects: grub2-unsigned (Ubuntu) Importance: Undecided Status: New ** Changed in: grub2-unsigned (Ubuntu) Status: New => Triaged ** Changed in: grub2-unsigned (Ubuntu) Importance: Undecided => Critical -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in grub2-unsigned package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I can replicate this on my Dell XPS13 9380 (no encryption). Occurs as well with Ubuntu 20.04 LTS, but not with 18.04 LTS, so not sure if it is related to the same reason? -> Disabling Secure Boot and SGX did not resolve the issue -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Duplicate bug 1970402 mentions that: > Workaround: Disable Secure Boot and SGX. works for multiple people. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Another workaround might be to use Ubuntu's own compressed image with a preinstalled Ubuntu Server (although not officially released) according to the following link https://ubuntuforums.org/showthread.php?t=2474692 Scroll down to read also the following posts, that describe how to install Lubuntu. In the same way you can install Ubuntu Desktop as well as all the community flavours. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Workaround: You can usually install Ubuntu on a drive in one machine and then move the drive into another machine. At least I think that's what I did on the affected machine with 22.04, which is why I never discovered this bug until 22.10 development started. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package,
Re: [Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
When is this going to be fixed and is there a work-around. I have tried both updating from 20.04 and direct installation of 22.04 and have found this problem preventing me from using 22.04 [it never boots]. I have two computers. One is a Dell with nvidia drivers [the perfect target machine to show this problem] - and it fails, but I also have a desktop I built myself that should not have problems [and obviously some having installed 22.04 do NOT have problems, but I do - and can not tell you what component is the problem [hardware or software]. Somewhere I read that making some spaces larger for the possible 'expansion' to not cause problems, but can not find it now with all of the many entries. I have spent MONTHS trying to install 22.04without success on either machine. I do not think you are correct that it only effects a small number of machines. Anyway, for those of us who it does, the problem is IMMENSE and total. On 03/08/2022 10:44, Julian Andres Klode wrote: > Yes, grub will be fixed eventually, but we are blocked by the security > update not being out yet. This is not a blocker for enabling upgrades to > 22.04.1, as it only affects a small number of systems and grub fixes > itself by using the previous kernel version if the new one fails. > -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
OK the patch set here is broken, we gotta do this from scratch properly. So I'm going to start cherry picking the rhboot patches for memory management. I have applied so far from bug 1989446 the backport of "Try to pick better locations for kernel and initrd" and cherry-picked from rhboot the "x86-efi: Use bounce buffers for reading to addresses > 4GB" patch and will continue picking up the memory changes from the rhboot branch and then we can address any gaps remaining. Now the problem with the patch set is clearly that the rhboot patches miss half their content and are applied after a custom patch that changes most of the memory management, which does not help. 0129 I thought was "Try to pick better locations for kernel and initrd" but it's something custom that's deep into the memory management code and just looks like it. I thought only increasing the memory limit in the last patch was the custom bit. I do not want to change general memory allocation in grub vs our upstreams. Scary. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Changed in: grub2-signed (Ubuntu) Status: Confirmed => Triaged ** Changed in: initramfs-tools (Ubuntu) Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Triaged Status in initramfs-tools package in Ubuntu: Won't Fix Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Our goal should be to merge the entire patch series into kinetic, worst case, kinetic will not be installable for some users. This means we will have decent results from people trying that in the next 4 weeks (by Oct 27 the release has been out 1 week). In the meantime, next week we should push out the security update again. Then we can take dannf's patches and the first patch here for that SRU to unblock the other OEM thing which certainly is less controversial. And on like Oct 27 if we don't have serious regressions do a final SRU with all the patches. I'll see what I can do to get those into kinetic so that we get that bit moving but I'll be travelling or meeting people most of next week. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I don't have such "there is allways that machine (or bad firmware) that nobody heard of that will fail" case from my hand but it's frustrated if it becomes a blocker for fixing the bug for new devices from market. However, I understand how users feel helpless if a SRU introduces a regression especially the bootloader. I think it's worth to mention that, here are many users reported their certain real cases (as opposed to "there is allways that machine (or bad firmware) that nobody heard of that will fail") which causes there system not able to boot. As we all known, Fedora and Arch linux are using higher compression rate (for initramfs) to avoid this issue. I could try the upstream grub as soon as I can and to see if I can prepare such scheme as what the mailing thread mentioned but I afraid it'll definitely take times. Anyway, if we can have some solutions for existing impacted user, then it'll very helpful. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I've been asked to prepare a summary of the current status of this bug: - there's a grub2 security update that's been published and then pulled: https://launchpad.net/ubuntu/+source/grub2-unsigned/2.06-2ubuntu10/+publishinghistory https://bugs.launchpad.net/ubuntu/+source/grub2-signed/+bug/1990684 - apt's dependency solver is being modified to handle updates that need to phase identically - once the apt update is released, then the grub security fixes can be republished - once the grub security fixes are republished, then this issue can be addressed. It appears there's an open question about the risks of jeremyszu's changes possibly causing problems for older systems. The closest thing I found in the linked thread was on this message: https://lists.gnu.org/archive/html/grub-devel/2017-03/msg00033.html > I seem to recall that the x86_64 port was being restricted due to > known bad firmware encountered in the past. It could be that it would > be worth adding an option to configure for enabling access to higher > addresses, alternatively for retaining compatibility with the broken > systems. I haven't read through the patches nor the upstream issue tracker to find out if these are recent problems or not, but this sounds like the usual warning that grub is difficult to test, lives in firmwares that may be ignored or otherwise horrible, etc. I hope we have a representative sample of machines to test in our labs, as well as our home offices, and in our wider community. Was there a more specific problem that I missed? Are there outstanding tasks that need doing that could be done before the apt+security update steps are complete? Refreshing patches, or skimming through issue trackers to find regressions from the patches, etc? Thanks -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs +
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hi Julian, I didn't see the 'boot failures on older system' from comment#90. Would you please point it out more specifically? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
@jeremyszu (os369510) Did you see #90, it seems your patch would cause boot failures on older system unable to handle this memory range. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I tested latest daily build https://cdimage.ubuntu.com/jammy/daily-live/20220920/jammy-desktop-amd64.iso and out of memory stil exist. I hope someone can provide an iso for download with a livesystem and install for jammy with changed compression. Ubuntu 21.10 (out of support) and Fedora 36 works fine. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Unfortunately the steps in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/comments/89 aren't working for me (I'm a professional developer, am confident i've followed the steps properly, have tried three times) Does anyone else have any other workarounds? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Add one more case caused by this issue: When SEV is enabled on the host, the guest also enables SEV and running 5.15 kernel with this commit: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e998879d4fb7991856916972168cf27c0d86ed12 The SWIOTLB driver in the guest could allocate 1G contiguous memory and fail: [0.005854] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB [0.042923] software IO TLB: Cannot allocate buffer [0.821973] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [0.822542] software IO TLB: No low mem With Try-to-pick-better-locations-for-kernel-and-initrd.patch, this will be solved Test procedure: Enable SEV on a AMD machine, refer to https://docs.ovh.com/us/en/dedicated/enable-and-use-amd-sme-sev/#references-and-additional-resources_1 create a ubuntu VM with SEV enabled (--launchSecurity sev) and 18G memory as below: virt-install --name --memory 18874368 --memtune hard_limit=36507216 --boot uefi --disk /var/lib/libvirt/images/,device=disk,bus=scsi --disk /var/lib/libvirt/images/-config.iso,device=cdrom --os-type linux --os-variant --import --controller type=scsi,model=virtio-scsi,driver.iommu=on --controller type=virtio-serial,driver.iommu=on --network network=default,model=virtio,driver.iommu=on --memballoon driver.iommu=on --graphics none --launchSecurity sev --noautoconsole Make sure the running kernel in VM is 5.15 Then check if it can boot successfully with the above patch dmesg should show SWIOTLB is correctly mapped to 1G memory [ 0.005713] software IO TLB: SWIOTLB bounce buffer size adjusted to 1024MB [ 0.821746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 0.822210] software IO TLB: mapped [mem 0x14fb4000-0x54fb4000] (1024MB) [ 0.933346] software IO TLB: Memory encryption is active and system is using DMA bounce buffers more details are in lp:1989446 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Tags added: foundations-todo -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case: `/var/cache/pbuilder/build/276481/build/grub2-2.06/obj/monolithic/grub- efi-amd64/grubx64.efi` * My build command: `sudo
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hey Jeremy, Understood; thanks for clarifying! I thought that maybe having upstream input could be helpful (as Julian mentioned to avoid diverging further), but if it is so much different nowadays, as you mentioned, it likely won't be that helpful anyway. By the way, appreciate that "newbie" statement.. after all this work/analysis/patches, I couldn't think that of you :) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hi Mauricio, From the patch set in this ticket: ``` +0129-Try-to-pick-better-locations-for-kernel-and-initrd.patch +0130-x86-efi-Use-bounce-buffers-for-reading-to-addresses-.patch +0131-x86-efi-Re-arrange-grub_cmd_linux-a-little-bit.patch +0132-x86-efi-Make-our-own-allocator-for-kernel-stuff.patch +0133-x86-efi-Allow-initrd-params-cmdline-allocations-abov.patch +0134-linuxefi-fail-kernel-validation-without-shim-protoco.patch +0135-Fix-4GB-memory-be-filtered-out-by-filter_memory_map.patch ``` 0129-0133 are the key patches to support the >4G memory allocation which made by vathpela and I didn't see they are upstreamed. From my point of view, it won't upstream because upstream has different approach to allocate memory for kernel and initramfs. Therefore, the last time I debug this issue is based on ubuntu code base. Indeed, I should take a look to see if upstream version whether has this issue. However, I learned from Julian that upstream refactored the MM part. (says, the patches will likely be dropped after 2.12 grub) I can give it a try but it takes time because I'm a newbie in the grub. Even if there is a patch on upstream, it need to based on the refactored MM which might not easy to land in LTS version as users expected. I suggest to consider to carry these patches in ubuntu to unblock the ubuntu users first. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Earlier today, I upgraded my Mint 20.3 system to Mint 21.0 and got this "out of memory" issue (Asus ROG G752VT laptop.). The fix in comment #41 and #89 worked (changing the compression to 19), but a lot of the steps were not needed. This is the minimal steps needed, (maybe fewer possible, anybody up for some code golf?): $ sudo su - # mkdir root He probably had his disk encrypted, I do not: # cryptsetup luksOpen /dev/nvme0n1p3 nvme0n1p3_crypt This one was very different for me due to mint vs Ubuntu. # mount /dev/mapper/vgmint-root ./root was # mount /dev/mapper/vgubuntu-root ./root This one makes the computer think the normal root is really root to fake out update-initramfs: # chroot ./root Yes, do this: # ### Now edit line 196 of /usr/sbin/mkinitramfs to to use -19 instead of -1 This wasn't completely happy with me, but it worked out ok for me. # update-initramfs -u -k all This command completely failed for me, but it didn't seem to hate me afterwards: # update-grub # exit # reboot I hope you have a happy reboot too. Then re-run # update-initramfs -u -k all and # update-grub To clean up any oddities. I wish there were a way to recompress this from grub though. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I found if the Max TOLUD is set to 1GB or higher in the BIOS (anything other than dynamic) then Ubuntu 22.04 ISO will not get this out of memory error -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hi renton, If you want to build it by yourself, then you should clone this: https://people.canonical.com/~jeremysu/lp1842320/ and built it locally and don't clear the build environment, you can found the efi binary on "obj/monolithic/grub-efi-amd64/grubx64.efi". I personally use pbuilder with some hooks when debugging. Alternatively, I upload the binary built by me on https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320v2 and for sure, the efi binary relates to security, then build it by yourself makes sense (but TBH it's not easy as other packages.) If you want to give it a try, then I suggest don't replace the original efi binary. Instead, you can use something like efibootmgr -c -L 'lp1842320' -d ${u-r-boot-dev} -p ${partition} -l '\EFI\ubuntu\grubx64.efi.lp1842320v2' for example: efibootmgr -c -L 'lp1842320' -d /dev/vda -p 2 -l '\EFI\ubuntu\grubx64.efi.lp1842320v2' and use something like 'efibootmgr -n ${your-boot-order-of-lp1842320}' or press F9 on HP machine or press F12 on Lenovo machine Please only do that if you know what you are doing. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
I am getting this error and cannot even boot Ubuntu 22.04.1 from a live USB. I am using an HP Elite Dragonfly G2 with 16GB of RAM. I get an error: out of memory message then a kernel panic. Please fix this, I would very much like to try running Ubuntu but I have to run Fedora 36 instead due to the fact that I can't even boot Ubuntu because of this bug. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 *
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hi, @jeremyszu How can I build grub from your repository https://github.com/os369510/grub2 with your patches for testing my linux box? Thanks in advance! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Hi Jeremy, It seems the PR in rhboot's grub2 hasn't been reviewed yet; just mentions from other folks who also found it helpful. Would you consider sending this to the grub-devel list? (i.e., upstream gnu's grub2) It looks like it wasn't yet (search results only cover rhboot). By the way, this type of change had an interesting discussion about the implementation approach a few years ago [1], but it seems it ceased as the sender didn't provide further replies. Your understanding of the tech details seems very deep, so it might hopefully be a good opportunity to propose that again :) Thanks! [1] https://lists.gnu.org/archive/html/grub-devel/2017-03/msg00032.html -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get >
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
A couple of days ago I upgraded my 21.10 system to 22.04 and got this "out of memory" issue (Dell XPS 15 9550 with 3840x2160 display). The fix in comment #41 worked (changing the compression to 19), and the sequence of commands in #44 helped. However, just noting that there are several typos in #44 that had me stumped for a while (it's the first time I've used chroot to update grub). So here are (I think!) the corrected commands to update grub: $ sudo su - # mkdir root # mkdir boot # mkdir efi # cryptsetup luksOpen /dev/nvme0n1p3 nvme0n1p3_crypt # mount /dev/mapper/vgubuntu-root ./root # mount /dev/nvme0n1p1 ./efi # was "nvme01p1" # mount /dev/nvme0n1p2 ./boot # was "nvme01p2" # mount -B ./boot ./root/boot # mount -B ./efi ./root/boot/efi # mount -B /dev ./root/dev # was: mount -B ./dev ./root/boot/dev # mount -B /dev/pts ./root/dev/pts # was: mount -B ./dev/pts ./root/boot/dev/pts # mount -B /proc ./root/proc # was: mount -B ./proc ./root/proc # mount -B /sys ./root/sys # was: mount -B ./sys ./root/sys # chroot ./root # ### Now edit line 196 of /usr/sbin/mkinitramfs to to use -19 instead of -1 # update-initramfs -u -k all # update-grub # exit # reboot -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Confirmed affecting GIGABYTE Z690 AORUS ELITE DDR4 (LGA1700, USB 3.2, PCIe 5.0) after installing nvidia-driver-515 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Confirmed affecting Samsung Galaxy Book2 (i7-1255U, Intel Iris Xe Graphics, 16GB LPDDR4x / NP750XED-KC4SE). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
This is also affecting the MSI MEG Z690I Unify motherboard; can not boot any live disk images of Ubuntu 22.04 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Ben, XPS 13 should have no problems with the factory-installed certified (by us) images: https://ubuntu.com/certified?q=XPS+13=20=22.04+LTS Certainly when we were certifying the latest XPS 13 Plus (with 4K), the only blocking issue I saw was a multi-monitor bug that we fixed before launch. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Joseph, out of interest, what model of XPS 13 do you have? I'm getting new XPS 13 soon and wondering if it'll have the same issue (I guess it's likely given the 4k screen). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
@juliank: I was affected by this bug when upgrading to 22.04.1 from 20.04 at the weekend. (originally installed as 18.04) Unfortunately the system did *not* auto-recover. Manually picking the previous kernel options from grub gave the same "error: out of memory." I was able to recover the system through live cd -> decrypt -> chroot -> MODULES=dep. This got me to initramfs on next boot, where I was able to `cryptsetup luksOpen` and mounting /root manually before continuing the boot. After getting back into the system, a final `update-initramfs` gave me a cleanly booting system. I've seen various other reports in Google of similar problems on Dell XPS 13 hardware like my own with 22.04. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2],
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
This patch is not from upstream or rhboot but which has been submitted as a MP to rhboot https://github.com/rhboot/grub2/pull/102 +0129-Try-to-pick-better-locations-for-kernel-and-initrd.patch +0130-x86-efi-Use-bounce-buffers-for-reading-to-addresses-.patch +0131-x86-efi-Re-arrange-grub_cmd_linux-a-little-bit.patch +0132-x86-efi-Make-our-own-allocator-for-kernel-stuff.patch +0133-x86-efi-Allow-initrd-params-cmdline-allocations-abov.patch +0134-linuxefi-fail-kernel-validation-without-shim-protoco.patch +0135-Fix-4GB-memory-be-filtered-out-by-filter_memory_map.patch 0129-0134 are from rhboot. ** Patch added: "0135-Fix-4GB-memory-be-filtered-out-by-filter_memory_map.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/+attachment/5607436/+files/0135-Fix-4GB-memory-be-filtered-out-by-filter_memory_map.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Patch added: "0134-linuxefi-fail-kernel-validation-without-shim-protoco.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/+attachment/5607435/+files/0134-linuxefi-fail-kernel-validation-without-shim-protoco.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Patch added: "0133-x86-efi-Allow-initrd-params-cmdline-allocations-abov.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/+attachment/5607434/+files/0133-x86-efi-Allow-initrd-params-cmdline-allocations-abov.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Patch added: "0131-x86-efi-Re-arrange-grub_cmd_linux-a-little-bit.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/+attachment/5607432/+files/0131-x86-efi-Re-arrange-grub_cmd_linux-a-little-bit.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Patch added: "0132-x86-efi-Make-our-own-allocator-for-kernel-stuff.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/+attachment/5607433/+files/0132-x86-efi-Make-our-own-allocator-for-kernel-stuff.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
** Patch added: "0130-x86-efi-Use-bounce-buffers-for-reading-to-addresses-.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/+attachment/5607431/+files/0130-x86-efi-Use-bounce-buffers-for-reading-to-addresses-.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
As my MP from comment#69 doesn't have response from rhboot team. Shared the debdiff here for review to consider to carry it in Ubuntu. ** Patch added: "0129-Try-to-pick-better-locations-for-kernel-and-initrd.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842320/+attachment/5607429/+files/0129-Try-to-pick-better-locations-for-kernel-and-initrd.patch -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi:
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
Yes, grub will be fixed eventually, but we are blocked by the security update not being out yet. This is not a blocker for enabling upgrades to 22.04.1, as it only affects a small number of systems and grub fixes itself by using the previous kernel version if the new one fails. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
In case it's relevant... I bricked my old PC while trying to work around this bug(!). After putting the same RAM and plugging the same monitor into a new machine, the bug does not occur. So the trigger for the bug was more to do with the old machine's firmware. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the
[Kernel-packages] [Bug 1842320] Re: Can't boot: "error: out of memory." immediately after the grub menu
As with comment #41, #44, and #45, has the decision been made to fix grub2's memory issues or just the compression level change that was with initramfs (as I believe it was a choice made to increase Raspberry Pie boot times)? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1842320 Title: Can't boot: "error: out of memory." immediately after the grub menu Status in grub: Unknown Status in OEM Priority Project: Triaged Status in grub2-signed package in Ubuntu: Confirmed Status in initramfs-tools package in Ubuntu: Confirmed Status in linux package in Ubuntu: Confirmed Bug description: [Impact] * In some cases, if the users’ initramfs grow bigger, then it’ll likely not be able to be loaded by grub2. * Some real cases from OEM projects: In many built-in 4k monitor laptops with nvidia drivers, the u-d-c puts the nvidia*.ko to initramfs which grows the initramfs to ~120M. Also the gfxpayload=auto will remain to use 4K resolution since it’s what EFI POST passed. In this case, the grub isn't able to load initramfs because the grub_memalign() won't be able to get suitable memory for the larger file: ``` #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376 #1 0x7dd7b074 in grub_malloc (size=592214020) at ../../../grub-core/kern/mm.c:408 #2 0x7dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076) at ../../../grub-core/kern/verifiers.c:150 #3 0x7dd801d4 in grub_file_open (name=0x7bc02f00 "/boot/initrd.img-5.17.0-1011-oem", type=131076) at ../../../grub-core/kern/file.c:121 #4 0x7bcd5a30 in ?? () #5 0x7fe21247 in ?? () #6 0x7bc030c8 in ?? () #7 0x00017fe21238 in ?? () #8 0x7bcd5320 in ?? () #9 0x7fe21250 in ?? () #10 0x in ?? () ``` Based on grub_mm_dump, we can see the memory fragment (some parts seem likely be used because of 4K resolution?) and doesn’t have available contiguous memory for larger file as: ``` grub_real_malloc(...) ... if (cur->size >= n + extra) ``` Based on UEFI Specification Section 7.2[1] and UEFI driver writers’ guide 4.2.3[2], we can ask 32bits+ on AllocatePages(). As most X86_64 platforms should support 64 bits addressing, we should extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available memory. * When users grown the initramfs, then probably will get initramfs not found which really annoyed and impact the user experience (system not able to boot). [Test Plan] * detailed instructions how to reproduce the bug: 1. Any method to grow the initramfs, such as install nvidia-driver. 2. If developers would like to reproduce, then could dd if=/dev/random of=... bs=1M count=500, something like: ``` $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file #!/bin/sh PREREQ="" prereqs() { echo "$PREREQ" } case $1 in # get pre-requisites prereqs) prereqs exit 0 ;; esac . /usr/share/initramfs-tools/hook-functions dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500 ``` And then update-initramfs * After applying my patches, the issue is gone. * I did also test my test grubx64.efi in: 1. X86_64 qemu with 1.1. 60M initramfs + 5.15.0-37-generic kernel 1.2. 565M initramfs + 5.17.0-1011-oem kernel 2. Amd64 HP mobile workstation with 2.1. 65M initramfs + 5.15.0-39-generic kernel 2.2. 771M initramfs + 5.17.0-1011-oem kernel All working well. [Where problems could occur] * The changes almost in i386/efi, thus the impact will be in the i386 / x86_64 EFI system. The other change is to modify the “grub-core/kern/efi/mm.c” but I use the original addressing for “arm/arm64/ia64/riscv32/riscv64”. Thus it should not impact them. * There is a “#if defined(__x86_64__)” which intent to limit the > 32bits code in i386 system and also ``` #if defined (__code_model_large__) -#define GRUB_EFI_MAX_USABLE_ADDRESS 0x +#define GRUB_EFI_MAX_USABLE_ADDRESS __UINTPTR_MAX__ +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x7fff #else #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fff +#define GRUB_EFI_MAX_ALLOCATION_ADDRESS 0x3fff #endif ``` If everything works as expected, then i386 should working good. If not lucky, based on “UEFI writers’ guide”[2], the i386 will get > 4GB memory region and never be able to access. [Other Info] * Upstream grub2 bug #61058 https://savannah.gnu.org/bugs/index.php?61058 * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320 * Test grubx64.efi: https://people.canonical.com/~jeremysu/lp1842320/grubx64.efi.lp1842320 * Test source code: https://github.com/os369510/grub2/tree/lp1842320 * If you built the package, then test grubx64.efi is under