Attached is the persistent panic log from the affected 7.0.0-1017-azure boot. The existing CurrentDmesg.txt attachment is from the working comparison kernel 6.17.0-1022-azure. The panic log contains the final OOM memory diagnostic and panic stack trace. The beginning of the OOM sequence was not retained. The file was reviewed for identifying information.
** Attachment added: "panic-07.txt" https://bugs.launchpad.net/ubuntu/+source/linux-azure-7.0/+bug/2173756/+attachment/6016305/+files/panic-07.txt ** Description changed: Summary ------- Ubuntu 24.04.5 LTS repeatedly experiences an OOM cascade and kernel panic under 7.0.0-1017-azure on Hyper-V with Dynamic Memory enabled: - Out of memory and no killable processes... - Kernel panic - not syncing: System is deadlocked on memory + Out of memory and no killable processes... + Kernel panic - not syncing: System is deadlocked on memory The affected kernel reports CMA-classified memory despite CmaTotal=0. At the captured panic, most reported free DMA32 memory was counted as CMA, while almost all swap remained free. The problem has not been observed so far under 6.17.0-1022-azure with Dynamic Memory still enabled. This suggests a kernel regression; the exact faulty component has not been identified. Environment ----------- Guest: - Ubuntu 24.04.5 LTS, x86_64 - Affected kernel: 7.0.0-1017-azure - Affected package version: 7.0.0-1017.17~24.04.1 - Source package: linux-azure-7.0 - Working comparison kernel: 6.17.0-1022-azure - Comparison package version: 6.17.0-1022.22 - 4 GiB active swap - vm.panic_on_oom=0; vm.overcommit_memory=0 Host: - Physical Hyper-V host, no nested virtualization - Windows Server 2025 Datacenter, 24H2, build 26100.33650 Dynamic Memory: - Enabled - Startup: 2048 MB - Minimum: 1024 MB - Maximum: 4096 MB - Memory buffer: 20% Reproduction ------------ 1. Cold-start or reboot the guest with 7.0.0-1017-azure and the above - Dynamic Memory settings. + Dynamic Memory settings. 2. Run the normal guest workload. 3. An OOM cascade occurs, followed by a kernel panic. Approximately five failures have been observed, generally within a few hours. The captured panic occurred about 54 seconds after boot. No synthetic stress test was required to trigger these failures. Expected: The guest remains operational as Hyper-V adjusts its memory. Actual: The OOM killer terminates multiple system services and the kernel eventually panics. Evidence from the affected boot ------------------------------ Shortly before OOM, the journal recorded: - hv_balloon: Max. dynamic memory size: 4096 MB - workqueue: balloon_up hogged CPU for >10000us ... - gmain invoked oom-killer: ... + hv_balloon: Max. dynamic memory size: 4096 MB + workqueue: balloon_up hogged CPU for >10000us ... + gmain invoked oom-killer: ... The persistent panic log contains: - Balloon: 931664kB + Balloon: 931664kB - Node 0 DMA32: - free: 848540kB - min: 73092kB - free_cma: 776084kB + Node 0 DMA32: + free: 848540kB + min: 73092kB + free_cma: 776084kB - Free swap = 4186964kB - Total swap = 4194300kB - 0 pages cma reserved + Free swap = 4186964kB + Total swap = 4194300kB + 0 pages cma reserved After terminating multiple services, the kernel reports no remaining killable processes and panics. The final failing allocation is in pgd_alloc during process execution (execve). Unexpected CMA classification ----------------------------- The affected kernel's boot log reports: - Memory: 1068104K/2096056K available - (... 1019608K reserved, 0K cma-reserved) + Memory: 1068104K/2096056K available + (... 1019608K reserved, 0K cma-reserved) A later live snapshot under the same kernel shows: - CmaTotal: 0 kB - CmaFree: 182200 kB - Balloon: 843776 kB + CmaTotal: 0 kB + CmaFree: 182200 kB + Balloon: 843776 kB /proc/zoneinfo reports DMA32 nr_free_cma=45550, matching 182200 KiB. /proc/pagetypeinfo shows nonempty CMA free lists and 379 DMA32 pageblocks classified as CMA. With 512 pages per block and 4 KiB pages, this represents 758 MiB, despite CmaTotal=0. Relevant kernel configuration: - CONFIG_CMA=y - CONFIG_DMA_CMA=y - CONFIG_CMA_SIZE_MBYTES=0 - CONFIG_HYPERV_BALLOON=y - CONFIG_BALLOON_MIGRATION=y + CONFIG_CMA=y + CONFIG_DMA_CMA=y + CONFIG_CMA_SIZE_MBYTES=0 + CONFIG_HYPERV_BALLOON=y + CONFIG_BALLOON_MIGRATION=y Comparison and additional testing --------------------------------- Under 6.17.0-1022-azure, Dynamic Memory remains enabled and the problem has not been observed so far. Testing duration is still limited. A snapshot under 6.17 shows: - CmaTotal: 0 kB - CmaFree: 0 kB - Balloon: 188416 kB + CmaTotal: 0 kB + CmaFree: 0 kB + Balloon: 188416 kB A 512 MiB / 30-second stress-ng test under 7.0 completed successfully. Guest RAM increased from approximately 2.8 to 3.5 GiB and later decreased to approximately 2.6 GiB without a panic. The balloon_up warning also occurred without a panic, so that warning alone does not identify a failure. Workaround ---------- Boot 6.17.0-1022-azure while leaving Dynamic Memory enabled. - Diagnostic data --------------- The Apport report was collected under the working comparison kernel 6.17.0-1022-azure, targeting the affected 7.0.0-1017-azure package. Identifying information and selected unrelated fields were removed before submission. - A persistent panic log from the affected 7.0 boot is available locally - and can be provided separately. It contains the final OOM memory - diagnostic and panic stack trace; the beginning of the OOM sequence - was not retained. + The persistent panic log from the affected 7.0 boot is attached as + panic-07.txt. It contains the final OOM memory diagnostic and panic + stack trace; the beginning of the OOM sequence was not retained. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: linux-image-7.0.0-1017-azure 7.0.0-1017.17~24.04.1 ProcVersionSignature: Ubuntu 6.17.0-1022.22-azure 6.17.13 Uname: Linux 6.17.0-1022-azure x86_64 AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory AnonymizationNotes: Local sanitization: internal host/domain/user names, home paths, UUIDs, filesystem IDs, MACs and context-identified IPv4 addresses replaced. Omitted fields: ProcEnviron, UdevDb, WifiSyslog, acpidump, dmi.board.asset.tag, dmi.chassis.asset.tag. Original data retained privately. Automatic sanitization is not a guarantee that all identifying information has been removed. AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.28.3-0ubuntu0.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' CRDA: N/A CasperMD5CheckResult: pass CloudArchitecture: x86_64 CloudID: none CloudName: none CloudPlatform: none CloudSubPlatform: config Date: Mon Nov 9 14:08:01 2026 InstallationDate: Installed on 2024-01-15 (1029 days ago) InstallationMedia: Ubuntu-Server 20.04.6 LTS "Focal Fossa" - Release amd64 (20230314.1) IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lspci: - + Lspci-vt: - + Lsusb: Error: command ['lsusb'] failed with exit code 1: Lsusb-t: - + Lsusb-v: Error: command ['lsusb', '-v'] failed with exit code 1: MachineType: Microsoft Corporation Virtual Machine PciMultimedia: - + ProcFB: 0 hyperv_drmdrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-6.17.0-1022-azure root=UUID=REDACTED_UUID_2 ro security=apparmor apparmor=1 iommu=force l1tf=flush mce=0 slab_nomerge=yes spec_store_bypass_disable=prctl ipv6.disable=1 RegressionContext: Collected under working comparison kernel 6.17.0-1022-azure. Affected kernel: 7.0.0-1017-azure. Hyper-V Dynamic Memory enabled. The affected boot's persistent panic log must be supplied separately. RelatedPackageVersions: - linux-restricted-modules-6.17.0-1022-azure N/A - linux-backports-modules-6.17.0-1022-azure N/A - linux-firmware 20240318.git3b128b60.0ubuntu3.1 + linux-restricted-modules-6.17.0-1022-azure N/A + linux-backports-modules-6.17.0-1022-azure N/A + linux-firmware 20240318.git3b128b60.0ubuntu3.1 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux-azure-7.0 UpgradeStatus: Upgraded to noble on 2025-01-13 (665 days ago) dmi.bios.date: 09/25/2025 dmi.bios.release: 4.1 dmi.bios.vendor: Microsoft Corporation dmi.bios.version: Hyper-V UEFI Release v4.1 dmi.board.name: Virtual Machine dmi.board.vendor: Microsoft Corporation dmi.board.version: Hyper-V UEFI Release v4.1 dmi.chassis.type: 3 dmi.chassis.vendor: Microsoft Corporation dmi.chassis.version: Hyper-V UEFI Release v4.1 dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev4.1:bd09/25/2025:br4.1:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev4.1:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev4.1:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev4.1:skuNone: dmi.product.family: Virtual Machine dmi.product.name: Virtual Machine dmi.product.sku: None dmi.product.version: Hyper-V UEFI Release v4.1 dmi.sys.vendor: Microsoft Corporation -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2173756 Title: [Regression] 7.0.0-1017-azure: Hyper-V Dynamic Memory causes OOM panic with unexpected CMA pageblocks; 6.17.0-1022-azure works To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-azure-7.0/+bug/2173756/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
