** Description changed: + [ Impact ] + + * When generating a memory-only dump of a running KVM guest using virsh dump <guest> vmcore --memory-only, the resulting dump cannot be analyzed with the crash utility. + * The fix patch ( https://github.com/crash-utility/crash/commit/d34554f3437b424028c5aa06ece5006e91718fb9 ) will fallback to `vmcoreinfo_read_from_memory`, when the vmcoreinfo_data is not found within the format handler. This patch is already upstream in crash and should appear in the next version. + + [ Test Plan ] + + * Take a kdump of a resolute kernel using `virsh dump <guest> vmcore --memory-only`. + * Install the debug symbols for the kernel. + * Run crash on the kernel debug symbols and the kdump. + * Confirm that crash loads successfully. + + [ Where problems could occur ] + + * This patch adds a fallback option. The previous expanded version of + this patch caused a regression on x86_64 but this version should not. + There is potential for a similar issue affecting opening a kdump with + crash. + + + Original Description: + == Comment: #0 - Misbah Anjum N <[email protected]> - 2026-01-19 08:38:14 == ======================= Problem Description ======================= When generating a memory-only dump of a running KVM guest using virsh dump <guest> vmcore --memory-only, the resulting dump cannot be analyzed with the crash utility. - Attempting to open the dump with the matching debug kernel (/usr/lib/debug/boot/vmlinux-6.18.0-8-generic) results in a fatal crash: + Attempting to open the dump with the matching debug kernel (/usr/lib/debug/boot/vmlinux-6.18.0-8-generic) results in a fatal crash: crash: invalid kernel virtual address: 410898500a96531c type: "first vmlist addr" Thank you, Misbah Anjum N - ======================= Steps to Reproduce ======================== 1. Start Ubuntu KVM guest 2. Collect the dump of the guest using "virsh dump" command using any of the following command: - a. virsh dump ubuntu2604 vmcore-memonly --memory-only - b. virsh dump ubuntu2604 vmcore-memonly-byc --memory-only --bypass-cache - c. virsh dump ubuntu2604 vmcore-memonly-elf --memory-only --format=elf - d. virsh dump ubuntu2604 vmcore-memonly-zlib --memory-only --format=kdump-zlib + a. virsh dump ubuntu2604 vmcore-memonly --memory-only + b. virsh dump ubuntu2604 vmcore-memonly-byc --memory-only --bypass-cache + c. virsh dump ubuntu2604 vmcore-memonly-elf --memory-only --format=elf + d. virsh dump ubuntu2604 vmcore-memonly-zlib --memory-only --format=kdump-zlib 3. Analyse the vmcore using crash utility tool: crash /usr/lib/debug/boot/vmlinux-6.18.0-8-generic <vmcore-generated> - ===================== System Configuration ======================== Host OS: Ubuntu 26.04 (Resolute Raccoon) Host Firmware: FW1110.11 (NH1110_102) Host Kernel: 6.18.0-8-generic Host CPUs: 40 Host Memory: Available 48Gi, Free 23 Gi Qemu Version: QEMU emulator version 10.1.0 (Debian 1:10.1.0+ds-5ubuntu4) Libvirt Version: libvirtd (libvirt) 11.6.0 Guest OS: Ubuntu 26.04 (Resolute Raccoon) Guest Kernel: 6.18.0-8-generic Guest CPUs: 10 Guest Memory: Available 18Gi, Free 17Gi Guest cmdline: BOOT_IMAGE=/vmlinux-6.18.0-8-generic root=UUID=cc380ee0-e6fa-4dd8-8fa7-b550d73e4adb ro security=selinux crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M - Crash Utility Version: + Crash Utility Version: crash 8.0.6 Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 10.2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. - ======================== Guest XML Created ======================== virt-install Command: virt-install --connect=qemu:///system --hvm --accelerate --name 'ubuntu2604' --machine pseries --memory=20000 --vcpu=10,sockets=1,cores=10,threads=1 --import --nographics --os- variant ubuntu25.10 --serial pty --memballoon model=virtio --controller type=scsi,model=virtio-scsi --disk path=/home/ubuntu2604.qcow2,bus=scsi,format=qcow2,size=80 --network=bridge=virbr0,model=virtio,mac=52:54:00:fc:7d:4a --boot emulator=/usr/bin/qemu-system-ppc64 <domain type='kvm'> - <name>ubuntu2604</name> - <uuid>5fafb52d-bee1-415c-baa3-24726613074a</uuid> - <metadata> - <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0";> - <libosinfo:os id="http://ubuntu.com/ubuntu/25.10"/> - </libosinfo:libosinfo> - </metadata> - <memory unit='KiB'>20480000</memory> - <currentMemory unit='KiB'>20480000</currentMemory> - <vcpu placement='static'>10</vcpu> - <os> - <type arch='ppc64le' machine='pseries-questing'>hvm</type> - <boot dev='hd'/> - </os> - <cpu mode='custom' match='exact' check='none'> - <model fallback='forbid'>POWER11</model> - <topology sockets='1' dies='1' clusters='1' cores='10' threads='1'/> - </cpu> - <clock offset='utc'/> - <on_poweroff>destroy</on_poweroff> - <on_reboot>restart</on_reboot> - <on_crash>destroy</on_crash> - <devices> - <emulator>/usr/bin/qemu-system-ppc64</emulator> - <disk type='file' device='disk'> - <driver name='qemu' type='qcow2'/> - <source file='/home/ubuntu2604.qcow2'/> - <target dev='sda' bus='scsi'/> - <address type='drive' controller='0' bus='0' target='0' unit='0'/> - </disk> - <controller type='scsi' index='0' model='virtio-scsi'> - <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> - </controller> - <controller type='usb' index='0' model='qemu-xhci' ports='15'> - <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> - </controller> - <controller type='pci' index='0' model='pci-root'> - <model name='spapr-pci-host-bridge'/> - <target index='0'/> - </controller> - <controller type='virtio-serial' index='0'> - <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> - </controller> - <interface type='bridge'> - <mac address='52:54:00:fc:7d:4a'/> - <source bridge='virbr0'/> - <model type='virtio'/> - <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> - </interface> - <serial type='pty'> - <target type='spapr-vio-serial' port='0'> - <model name='spapr-vty'/> - </target> - <address type='spapr-vio' reg='0x30000000'/> - </serial> - <console type='pty'> - <target type='serial' port='0'/> - <address type='spapr-vio' reg='0x30000000'/> - </console> - <channel type='unix'> - <target type='virtio' name='org.qemu.guest_agent.0'/> - <address type='virtio-serial' controller='0' bus='0' port='1'/> - </channel> - <audio id='1' type='none'/> - <memballoon model='virtio'> - <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> - </memballoon> - <rng model='virtio'> - <backend model='random'>/dev/urandom</backend> - <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> - </rng> - <panic model='pseries'/> - </devices> + <name>ubuntu2604</name> + <uuid>5fafb52d-bee1-415c-baa3-24726613074a</uuid> + <metadata> + <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0";> + <libosinfo:os id="http://ubuntu.com/ubuntu/25.10"/> + </libosinfo:libosinfo> + </metadata> + <memory unit='KiB'>20480000</memory> + <currentMemory unit='KiB'>20480000</currentMemory> + <vcpu placement='static'>10</vcpu> + <os> + <type arch='ppc64le' machine='pseries-questing'>hvm</type> + <boot dev='hd'/> + </os> + <cpu mode='custom' match='exact' check='none'> + <model fallback='forbid'>POWER11</model> + <topology sockets='1' dies='1' clusters='1' cores='10' threads='1'/> + </cpu> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-ppc64</emulator> + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2'/> + <source file='/home/ubuntu2604.qcow2'/> + <target dev='sda' bus='scsi'/> + <address type='drive' controller='0' bus='0' target='0' unit='0'/> + </disk> + <controller type='scsi' index='0' model='virtio-scsi'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> + </controller> + <controller type='usb' index='0' model='qemu-xhci' ports='15'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> + </controller> + <controller type='pci' index='0' model='pci-root'> + <model name='spapr-pci-host-bridge'/> + <target index='0'/> + </controller> + <controller type='virtio-serial' index='0'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> + </controller> + <interface type='bridge'> + <mac address='52:54:00:fc:7d:4a'/> + <source bridge='virbr0'/> + <model type='virtio'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> + </interface> + <serial type='pty'> + <target type='spapr-vio-serial' port='0'> + <model name='spapr-vty'/> + </target> + <address type='spapr-vio' reg='0x30000000'/> + </serial> + <console type='pty'> + <target type='serial' port='0'/> + <address type='spapr-vio' reg='0x30000000'/> + </console> + <channel type='unix'> + <target type='virtio' name='org.qemu.guest_agent.0'/> + <address type='virtio-serial' controller='0' bus='0' port='1'/> + </channel> + <audio id='1' type='none'/> + <memballoon model='virtio'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> + </memballoon> + <rng model='virtio'> + <backend model='random'>/dev/urandom</backend> + <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> + </rng> + <panic model='pseries'/> + </devices> </domain> - ====================== Crash Analysis ============================= # virsh dump ubuntu2604 vmcore-memonly --memory-only Domain 'ubuntu2604' dumped to vmcore-memonly # crash /usr/lib/debug/boot/vmlinux-6.18.0-8-generic vmcore-memonly crash 8.0.6 Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 10.2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: - <http://www.gnu.org/software/gdb/documentation/>. + <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... crash: invalid kernel virtual address: 410898500a96531c type: "first vmlist addr" Errors like the one above typically occur when the kernel and memory source do not match. These are the files being used: - KERNEL: /usr/lib/debug/boot/vmlinux-6.18.0-8-generic - DUMPFILE: vmcore-memonly - + KERNEL: /usr/lib/debug/boot/vmlinux-6.18.0-8-generic + DUMPFILE: vmcore-memonly # virsh dump ubuntu2604 vmcore-memonly-zlib --memory-only --format=kdump-zlib Domain 'ubuntu2604' dumped to vmcore-memonly-zlib # crash /usr/lib/debug/boot/vmlinux-6.18.0-8-generic vmcore-memonly-zlib crash 8.0.6 Copyright (C) 2002-2025 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011, 2020-2024 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. Copyright (C) 2015, 2021 VMware, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 10.2 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64le-unknown-linux-gnu". Type "show configuration" for configuration details. Find the GDB manual and other documentation resources online at: - <http://www.gnu.org/software/gdb/documentation/>. + <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... crash: invalid kernel virtual address: 410898500a96531c type: "first vmlist addr" Errors like the one above typically occur when the kernel and memory source do not match. These are the files being used: - KERNEL: /usr/lib/debug/boot/vmlinux-6.18.0-8-generic - DUMPFILE: vmcore-memonly-zlib - + KERNEL: /usr/lib/debug/boot/vmlinux-6.18.0-8-generic + DUMPFILE: vmcore-memonly-zlib =================================================================== == Comment: #2 - Aditya Gupta <[email protected]> - 2026-02-10 11:55:27 == Update on this: The patch was merged upstream, but due to a breakage it has been reverted. Shivang has sent a fix patch for it, it's currently in the mailing list: https://lists.crash-utility.osci.io/archives/list/[email protected] utility.osci.io/thread/FJGIVQ5K5NSTT2U7TY4GUQZKEDUBGZRV/ This issue should get fixed once bug 212620 is fixed. Assigning the issue to shivang since he's currently working on it. Thanks
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2141507 Title: [Ubuntu2604] [FW1110] Unable to analyse vmcore generated via "virsh dump" using crash utility To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/crash/+bug/2141507/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
