I put together some notes and work-arounds in order to provide a simpler
reference for people hitting this issue. I didn't test everything below
but nothing should be risky.

# Summary

Grub attempts to read the initrd into a memory location that is too

This issue is caused by a combination of several factors:
- Grub not setting aside enough space for the initrd in memory and not at the 
right location,
- Initrds having grown due to holding more modules and more firmware. This is 
especially the case with the nvidia proprietary driver.
- Typical screen resolutions and their associated buffers in memory having 

A related issue is not having enough space in /boot to hold enough

# Solutions

There are four ways to tackle these issues:
- Patch grub,
- Have fewer modules,
- Compress initrds more,
- Lower screen resolution at boot.

## Patch grub
This is being worked on. It’s the trickier option so I’m not going to provide 
details but you can find that in the discussion above.

## Have fewer modules
The default configuration creates initrds that are meant to be universal in 
order to accommodate as many hardware variants as possible. Unsurprisingly this 
makes initrds much bigger.

It is possible to change the value of MODULES to ‘dep’ in
/etc/initramfs-tools/initramfs.conf . This will drastically reduce the
initrd size in virtually every configuration at the cost of not
supporting during boot hardware that is not plugged when the initrd
generation takes place.

## Compress initrds more
Change the value of COMPRESS in /etc/initramfs-tools/initramfs.conf .
In terms of compression level, you will have something along the lines of the 
    lz4: 74066711
    zstd: 54260379
    gzip: 53829310
    xz: 29665544

NB: these values are only valid on one specific machine and
configuration; they are only meant to give an idea of compression ratio
that can be obtained but the initrd uses MODULES=dep as outlined above.

With a small hack, it is also possible to make xz compression much faster at a 
small cost in compression by adding the following at the end of initramfs.conf:
    export XZ_OPT='--lzma2=preset=0,dict=8M'

In the future, it will be possible to directly set compression levels
for every compression method.

## Lower screen resolution at boot
You can use a lower resolution screen when booting.

You can also edit /etc/default/grub and use the following:
Remember to run update-grub afterwards.
If you need to do it at boot-time, you will use GFXMODE instead (no ‘GRUB_’ 

You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to initramfs-tools in Ubuntu.

  Can't boot: "error: out of memory." immediately after the grub menu

Status in grub:
Status in OEM Priority Project:
Status in grub2-signed package in Ubuntu:
Status in grub2-unsigned package in Ubuntu:
Status in initramfs-tools package in Ubuntu:
  Won't Fix
Status in linux package in Ubuntu:

Bug description:

   * In some cases, if the users’ initramfs grow bigger, then it’ll
  likely not be able to be loaded by grub2.

   * Some real cases from OEM projects:

  In many built-in 4k monitor laptops with nvidia drivers, the u-d-c
  puts the nvidia*.ko to initramfs which grows the initramfs to ~120M.
  Also the gfxpayload=auto will remain to use 4K resolution since it’s
  what EFI POST passed.

  In this case, the grub isn't able to load initramfs because the
  grub_memalign() won't be able to get suitable memory for the larger

  #0 grub_memalign (align=1, size=592214020) at ../../../grub-core/kern/mm.c:376
  #1 0x000000007dd7b074 in grub_malloc (size=592214020) at 
  #2 0x000000007dd7a2c8 in grub_verifiers_open (io=0x7bc02d80, type=131076)
      at ../../../grub-core/kern/verifiers.c:150
  #3 0x000000007dd801d4 in grub_file_open (name=0x7bc02f00 
      type=131076) at ../../../grub-core/kern/file.c:121
  #4 0x000000007bcd5a30 in ?? ()
  #5 0x000000007fe21247 in ?? ()
  #6 0x000000007bc030c8 in ?? ()
  #7 0x000000017fe21238 in ?? ()
  #8 0x000000007bcd5320 in ?? ()
  #9 0x000000007fe21250 in ?? ()
  #10 0x0000000000000000 in ?? ()

  Based on grub_mm_dump, we can see the memory fragment (some parts seem
  likely be used because of 4K resolution?) and doesn’t have available
  contiguous memory for larger file as:

  if (cur->size >= n + extra)

  Based on UEFI Specification Section 7.2[1] and UEFI driver writers’
  guide 4.2.3[2], we can ask 32bits+ on AllocatePages().

  As most X86_64 platforms should support 64 bits addressing, we should
  extend GRUB_EFI_MAX_USABLE_ADDRESS to 64 bits to get more available

   * When users grown the initramfs, then probably will get initramfs
  not found which really annoyed and impact the user experience (system
  not able to boot).

  [Test Plan]

   * detailed instructions how to reproduce the bug:

  1. Any method to grow the initramfs, such as install nvidia-driver.

  2. If developers would like to reproduce, then could dd if=/dev/random
  of=... bs=1M count=500, something like:

  $ cat /usr/share/initramfs-tools/hooks/zzz-touch-a-file


          echo "$PREREQ"

  case $1 in
  # get pre-requisites
          exit 0

  . /usr/share/initramfs-tools/hook-functions
  dd if=/dev/random of=${DESTDIR}/test-500M bs=1M count=500

  And then update-initramfs

   * After applying my patches, the issue is gone.

   * I did also test my test grubx64.efi in:

  1. X86_64 qemu with
  1.1. 60M initramfs + 5.15.0-37-generic kernel
  1.2. 565M initramfs + 5.17.0-1011-oem kernel

  2. Amd64 HP mobile workstation with
  2.1. 65M initramfs + 5.15.0-39-generic kernel
  2.2. 771M initramfs + 5.17.0-1011-oem kernel

  All working well.

  [Where problems could occur]

  * The changes almost in i386/efi, thus the impact will be in the i386 / 
x86_64 EFI system.
  The other change is to modify the “grub-core/kern/efi/mm.c” but I use the 
original addressing for “arm/arm64/ia64/riscv32/riscv64”.
  Thus it should not impact them.

  * There is a “#if defined(__x86_64__)” which intent to limit the >
  32bits code in i386 system and also

   #if defined (__code_model_large__)
  -#define GRUB_EFI_MAX_USABLE_ADDRESS 0xffffffff
   #define GRUB_EFI_MAX_USABLE_ADDRESS 0x7fffffff

  If everything works as expected, then i386 should working good.

  If not lucky, based on “UEFI writers’ guide”[2], the i386 will get >
  4GB memory region and never be able to access.

  [Other Info]

   * Upstream grub2 bug #61058

   * Test PPA: https://launchpad.net/~os369510/+archive/ubuntu/lp1842320

   * Test grubx64.efi:

   * Test source code: https://github.com/os369510/grub2/tree/lp1842320

   * If you built the package, then test grubx64.efi is under
  “obj/monolithic/grub-efi-amd64/grubx64.efi”, in my case:

   * My build command: `sudo PBSHELL=1 pbuilder build --hookdir ~/hook-
  dir ubuntu-grub/grub2_2.06-2ubuntu7+jeremydev2.dsc 2>&1 | tee

   * My qemu command: `qemu-system-x86_64 -bios
  edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF.fd -hda Templates/grub.qcow2 -m
  6G -vga cirrus -smp 8 -machine type=q35,accel=kvm -cpu host -enable-
  kvm -boot menu=on` (I built an edk2 binary with debugging log)

   * You can use my grubx64.efi with debug symbols from
  with-debug-symbols and source code is from
  https://github.com/os369510/grub2/tree/jeremy-dev .

  After built the package from source code, then you can use gdb to
  attach the qemu session as:

  ubuntu@ubuntu-HP-ZBook-Fury-16-G9-Mobile-Workstation-PC [ 
  $ gdb -x gdb_grub # with “add-symbol-file kernel.img ${address}

  The address above can read from qemu serial port and found the last
  “Loading driver at 0x000xxxxxxxxxx EntryPoint=0x000xxxxxxxabc”

  In above case, fill “0x000xxxxxxxabc” to ${address}.



  Upgraded from 19.04 to current 19.10 using "do-release-upgrade -d".
  Can still boot using the previous 5.0.0-25-generic kernel, but the
  5.2.0-15-generic fails to start.

  On selecting Ubuntu from Grub, the message "error: out of memory." is
  immediately shown. Pressing a key attempts to start boot-up but fails
  to mount root fs.

  Machine is HP Spectre X360 with 8GB RAM. Under kernel 5.0.0, free
  shows the following (run from Gnome terminal):

                total        used        free      shared  buff/cache   
  Mem:        7906564     1761196     3833240     1020216     2312128     
  Swap:       1003516           0     1003516

  Kernel packages installed:

  linux-generic                     amd64
  linux-headers-5.2.0-15                     5.2.0-15.16 all
  linux-headers-5.2.0-15-generic             5.2.0-15.16 amd64
  linux-headers-generic             amd64
  linux-image-5.0.0-25-generic               5.0.0-25.26 amd64
  linux-image-5.2.0-15-generic               5.2.0-15.16+signed1 amd64
  linux-image-generic               amd64
  linux-modules-5.0.0-25-generic             5.0.0-25.26 amd64
  linux-modules-5.2.0-15-generic             5.2.0-15.16 amd64
  linux-modules-extra-5.0.0-25-generic       5.0.0-25.26 amd64
  linux-modules-extra-5.2.0-15-generic       5.2.0-15.16 amd64

  Photo of kernel panic attached.

  NVMe drive partition layout (GPT):

  Device           Start        End   Sectors   Size Type
  /dev/nvme0n1p1    2048    1050623   1048576   512M EFI System
  /dev/nvme0n1p2 1050624    2549759   1499136   732M Linux filesystem
  /dev/nvme0n1p3 2549760 1000214527 997664768 475.7G Linux filesystem

  $ sudo pvs
    PV                          VG        Fmt  Attr PSize    PFree
    /dev/mapper/nvme0n1p3_crypt ubuntu-vg lvm2 a--  <475.71g    0

  $ sudo lvs
    LV     VG        Attr       LSize   Pool Origin Data%  Meta%  Move Log 
Cpy%Sync Convert
    root   ubuntu-vg -wi-ao---- 474.75g
    swap_1 ubuntu-vg -wi-ao---- 980.00m

  Partition 3 is LUKS encrypted. Root LV is ext4.
  ProblemType: Bug
  ApportVersion: 2.20.11-0ubuntu7
  Architecture: amd64
   /dev/snd/controlC0:  gmckeown   1647 F.... pulseaudio
  CurrentDesktop: ubuntu:GNOME
  DistroRelease: Ubuntu 19.10
  InstallationDate: Installed on 2019-08-15 (18 days ago)
  InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
   Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
   Bus 001 Device 003: ID 8087:0a2b Intel Corp.
   Bus 001 Device 002: ID 04f2:b593 Chicony Electronics Co., Ltd HP Wide Vision 
FHD Camera
   Bus 001 Device 004: ID 046d:c52b Logitech, Inc. Unifying Receiver
   Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  MachineType: HP HP Spectre x360 Convertible 13-ae0xx
  Package: linux (not installed)
  ProcFB: 0 inteldrmfb
  ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.0.0-25-generic 
root=/dev/mapper/ubuntu--vg-root ro quiet splash
  ProcVersionSignature: Ubuntu 5.0.0-25.26-generic 5.0.18
   linux-restricted-modules-5.0.0-25-generic N/A
   linux-backports-modules-5.0.0-25-generic  N/A
   linux-firmware                            1.181
  Tags:  eoan
  Uname: Linux 5.0.0-25-generic x86_64
  UpgradeStatus: Upgraded to eoan on 2019-09-02 (0 days ago)
  UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
  _MarkForUpload: True
  dmi.bios.date: 05/17/2019
  dmi.bios.vendor: AMI
  dmi.bios.version: F.25
  dmi.board.asset.tag: Base Board Asset Tag
  dmi.board.name: 83B9
  dmi.board.vendor: HP
  dmi.board.version: 56.43
  dmi.chassis.type: 31
  dmi.chassis.vendor: HP
  dmi.chassis.version: Chassis Version
  dmi.product.family: 103C_5335KV HP Spectre
  dmi.product.name: HP Spectre x360 Convertible 13-ae0xx
  dmi.product.sku: 2QH38EA#ABU
  dmi.sys.vendor: HP

To manage notifications about this bug go to:

Mailing list: https://launchpad.net/~touch-packages
Post to     : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to