Public bug reported: [Summary]
After unattended-upgrades installed linux-aws 7.0.0-1009.9~24.04.1 (upgrade from 6.17.0-1019.19~24.04.1) on an EC2 t3a.nano (512MB RAM) running Ubuntu 24.04.4 LTS, the first boot of the new kernel came up with only ~75MB of usable memory: Memory: 74904K/503328K available (23130K kernel code, 4948K rwdata, 16388K rodata, 5412K init, 5744K bss, 422584K reserved, 0K cma-reserved) 422MB of the 503MB present were reserved at boot. Although the boot banner says "0K cma-reserved", at runtime ~316MB of the "free" memory sat in a CMA pool, unusable for regular kernel allocations (from the OOM report Mem-Info): Node 0 DMA32 free:317020kB ... managed:428984kB mlocked:27252kB ... free_cma:315604kB active_anon:0 inactive_anon:15 ... slab_reclaimable:2139 slab_unreclaimable:9479 ... all_unreclaimable? yes Within ~2 minutes of boot the global OOM killer started killing every process on the system (cron, systemd-logind, systemd-networkd, systemd-resolved, rsyslog, agetty, amazon-ssm-agent, ...). Every OOM victim had near-zero anon RSS, e.g.: Out of memory: Killed process 532 (python3) total-vm:131592kB, anon-rss:16kB, file-rss:2968kB, shmem-rss:0kB, UID:0 pgtables:160kB oom_score_adj:0 i.e. no userspace process was consuming memory; the memory was reserved by the kernel itself. The instance remained in an OOM kill loop (unreachable/unusable) until it was rebooted ~7 hours later. [The trigger appears to be the initrdless boot path] Ubuntu cloud images boot via GRUB_FORCE_PARTUUID (initrdless fast boot: kernel + /boot/microcode.cpio only, with panic=-1). That was the failing boot: Command line: BOOT_IMAGE=/boot/vmlinuz-7.0.0-1009-aws root=PARTUUID=bf0d338c-01 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1 RAMDISK: [mem 0x1cdc5000-0x1dc2cfff] <- 15,106,048 bytes = microcode.cpio Memory: 74904K/503328K available (... 422584K reserved, 0K cma-reserved) The subsequent manual reboot took GRUB's fallback path (initrdfail=1 was still set, so it booted with the full initramfs and without panic=-1). The exact same kernel then reserved a normal amount of memory and runs fine: Command line: BOOT_IMAGE=/boot/vmlinuz-7.0.0-1009-aws root=PARTUUID=bf0d338c-01 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 RAMDISK: [mem 0x18fc9000-0x1dc3dfff] <- full 78MB initramfs Memory: 349536K/503324K available (... 148204K reserved, 0K cma-reserved) On the healthy boot /proc/meminfo shows CmaTotal: 0 kB. So: same kernel, same machine, same day — initrdless boot reserves 422MB (~316MB of it ending up as CMA), full-initramfs boot reserves 148MB. The previous 6.17.0-1019-aws kernel booted initrdless on this instance for weeks without problems, so this is a regression in the 7.0 kernel. On a 512MB instance this is fatal: the OOM killer kills init-adjacent services faster than they can respawn and the instance never becomes usable. [Environment] - Ubuntu 24.04.4 LTS (noble), cloud image - EC2 t3a.nano (AMD, 512MB RAM, nvme root, ENA), eu-central-1 - linux-image-7.0.0-1009-aws 7.0.0-1009.9~24.04.1 (from noble-updates/noble-security) - Previous kernel: 6.17.0-1019-aws (no issue) - GRUB_FORCE_PARTUUID=bf0d338c-01 (stock cloud-image /etc/default/grub.d/40-force-partuuid.cfg) - Linux version 7.0.0-1009-aws (buildd@lcy02-amd64-104) #9~24.04.1-Ubuntu SMP PREEMPT Mon Jul 13 22:43:41 UTC 2026 (Ubuntu 7.0.0-1009.9~24.04.1-aws 7.0.12) [Steps to reproduce] 1. Ubuntu 24.04 cloud image on a t3a.nano with the linux-aws (HWE) kernel meta 2. Upgrade to linux-aws 7.0.0-1009.9~24.04.1 and reboot 3. Let GRUB take its default initrdless path (GRUB_FORCE_PARTUUID set, first boot attempt, panic=-1 + microcode.cpio only) Expected: kernel boots with roughly the same usable memory as with an initramfs (~350MB of 503MB). Actual: 422584K reserved at boot, ~75MB usable, ~316MB stuck as free_cma at runtime, system-wide OOM kill storm within 2 minutes, instance unusable. [Workarounds found] - Boot with the full initramfs (disable GRUB_FORCE_PARTUUID initrdless boot), or - Switch to the GA kernel track linux-aws-lts-24.04 (6.8 series) [Additional data] Full kern.log of the failing boot (including the complete first OOM report with Mem-Info and process dump) is available on request / attached. First OOM stack (modprobe loading a module, order-0 GFP_KERNEL allocation): modprobe invoked oom-killer: gfp_mask=0xcc2(GFP_KERNEL|__GFP_HIGHMEM), order=0, oom_score_adj=0 CPU: 1 UID: 0 PID: 693 Comm: modprobe Not tainted 7.0.0-1009-aws #9~24.04.1-Ubuntu PREEMPTLAZY Hardware name: Amazon EC2 t3a.nano/, BIOS 1.0 10/16/2017 Call Trace: <TASK> dump_stack_lvl+0x76/0xa0 out_of_memory+0x101/0x350 __alloc_pages_may_oom+0x109/0x1d0 __alloc_pages_slowpath.constprop.0+0x42d/0x9f0 __alloc_frozen_pages_noprof+0x329/0x350 module_zstd_decompress+0x111/0x2c0 module_decompress+0x65/0xf0 init_module_from_file+0xc5/0x160 __x64_sys_finit_module+0x73/0xf0 ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: linux-image-7.0.0-1009-aws 7.0.0-1009.9~24.04.1 ProcVersionSignature: Ubuntu 7.0.0-1009.9~24.04.1-aws 7.0.12 Uname: Linux 7.0.0-1009-aws x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.28.2-0ubuntu0.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' CRDA: N/A CasperMD5CheckResult: unknown CloudArchitecture: x86_64 CloudBuildName: server CloudID: aws CloudName: aws CloudPlatform: ec2 CloudRegion: eu-central-1 CloudSerial: 20200729 CloudSubPlatform: metadata (http://169.254.169.254) Date: Tue Jul 28 14:45:43 2026 Ec2Architecture: x86_64 Ec2Imageid: ami-0c2b1c303a2e4cb49 Ec2Instancetype: t3a.nano Ec2Region: eu-central-1 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Error: command ['lsusb'] failed with exit code 1: Lsusb-t: Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1: MachineType: Amazon EC2 t3a.nano PciMultimedia: ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-7.0.0-1009-aws root=PARTUUID=bf0d338c-01 ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 RelatedPackageVersions: linux-restricted-modules-7.0.0-1009-aws N/A linux-backports-modules-7.0.0-1009-aws N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux-aws-7.0 UpgradeStatus: Upgraded to noble on 2025-02-24 (519 days ago) dmi.bios.date: 10/16/2017 dmi.bios.release: 1.0 dmi.bios.vendor: Amazon EC2 dmi.bios.version: 1.0 dmi.board.asset.tag: i-03627e2ad21ebb52d dmi.board.vendor: Amazon EC2 dmi.chassis.asset.tag: Amazon EC2 dmi.chassis.type: 1 dmi.chassis.vendor: Amazon EC2 dmi.modalias: dmi:bvnAmazonEC2:bvr1.0:bd10/16/2017:br1.0:svnAmazonEC2:pnt3a.nano:pvr:rvnAmazonEC2:rn:rvr:cvnAmazonEC2:ct1:cvr:sku:pfa: dmi.product.name: t3a.nano dmi.sys.vendor: Amazon EC2 ** Affects: linux-aws-7.0 (Ubuntu) Importance: Undecided Status: New ** Tags: amd64 apport-bug cloud-image noble -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2161986 Title: 7.0.0-1009-aws reserves 422MB of 503MB at boot on t3a.nano when booting initrdless (GRUB_FORCE_PARTUUID), causing immediate system- wide OOM To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-aws-7.0/+bug/2161986/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
