Public bug reported: kdump cannot generate coredump file on bluefield with 5.4 kernel
Bug description: Following the instruction in https://ubuntu.com/server/docs/kernel-crash-dump, the coredump file cannot be generated. Bluefield is running 5.4 kernel bf2:~$ uname -a Linux sw-mtx-008-bf2 5.4.0-1060-bluefield #66-Ubuntu SMP PREEMPT Mon Mar 27 15:52:50 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux crashkernel parameter is configured bf2:~$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.4.0-1060-bluefield root=UUID=52ddbe2c-ee4f-48d4-b7d4-ab76e264e438 ro console=hvc0 console=ttyAMA0 earlycon=pl011,0x01000000 fixrtc net.ifnames=0 biosdevname=0 iommu.passthrough=1 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M bf2:~$ dmesg | grep -i crash [ 0.000000] crashkernel reserved: 0x00000000cfe00000 - 0x00000000efe00000 (512 MB) [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1060-bluefield root=UUID=52ddbe2c-ee4f-48d4-b7d4-ab76e264e438 ro console=hvc0 console=ttyAMA0 earlycon=pl011,0x01000000 fixrtc net.ifnames=0 biosdevname=0 iommu.passthrough=1 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M [ 8.070921] pstore: Using crash dump compression: deflate kdump-config is as below: bf2:~$ kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash crashkernel addr: 0x /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1060-bluefield kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1060-bluefield current state: ready to kdump kexec command: /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1060-bluefield root=UUID=52ddbe2c-ee4f-48d4-b7d4-ab76e264e438 ro console=hvc0 console=ttyAMA0 earlycon=pl011,0x01000000 fixrtc net.ifnames=0 biosdevname=0 iommu.passthrough=1 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz sysrq: bf2:/# cat /proc/sys/kernel/sysrq 176 After trigged the crash manually with "echo c > /proc/sysrq-trigger", the system could not come up because of OOM. And after change the crashkernel with 1024M memory it still hangs. With default 512M, it hangs at "Killed process 674" [ 8.718188] systemd-journald[368]: File /var/log/journal/8244d38b2f804fc692f3f2dbf8206f57/system.journal corrupted or uncleanly shut down, renaming and re. [ 30.252513] Out of memory: Killed process 651 (systemd-resolve) total-vm:24380kB, anon-rss:3812kB, file-rss:1828kB, shmem-rss:0kB, UID:101 pgtables:80kB o0 ... [ 34.651927] Out of memory: Killed process 674 (dbus-daemon) total-vm:7884kB, anon-rss:552kB, file-rss:1380kB, shmem-rss:0kB, UID:103 pgtables:52kB oom_sco0 With 1024M, it hangs at following [ 8.733323] systemd-journald[369]: File /var/log/journal/8244d38b2f804fc692f3f2dbf8206f57/system.journal corrupted or uncleanly shut down, renaming and re. After soft reboot the Bluefield, there's no coredump file generated. bf2:~$ ls /var/crash/ -la total 52 drwxrwxrwt 3 root root 4096 May 31 01:43 . drwxr-xr-x 14 root root 4096 Apr 30 11:26 .. drwxrwxr-x 2 ubuntu ubuntu 4096 May 31 01:43 202305310143 -rw-r----- 1 root root 34307 May 31 01:18 _usr_share_netplan_netplan.script.0.crash -rw-r--r-- 1 root root 0 May 31 03:47 kdump_lock -rw-r--r-- 1 root root 358 May 31 03:48 kexec_cmd bf2:~$ ls /var/crash/202305310143/ -la total 8 drwxrwxr-x 2 ubuntu ubuntu 4096 May 31 01:43 . drwxrwxrwt 3 root root 4096 May 31 01:43 .. This issue also happens on 5.4.0-1049-bluefield kernel. ** Affects: linux-bluefield (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux-bluefield in Ubuntu. https://bugs.launchpad.net/bugs/2021930 Title: kdump cannot generate coredump file on bluefield with 5.4 kernel Status in linux-bluefield package in Ubuntu: New Bug description: kdump cannot generate coredump file on bluefield with 5.4 kernel Bug description: Following the instruction in https://ubuntu.com/server/docs/kernel-crash-dump, the coredump file cannot be generated. Bluefield is running 5.4 kernel bf2:~$ uname -a Linux sw-mtx-008-bf2 5.4.0-1060-bluefield #66-Ubuntu SMP PREEMPT Mon Mar 27 15:52:50 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux crashkernel parameter is configured bf2:~$ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-5.4.0-1060-bluefield root=UUID=52ddbe2c-ee4f-48d4-b7d4-ab76e264e438 ro console=hvc0 console=ttyAMA0 earlycon=pl011,0x01000000 fixrtc net.ifnames=0 biosdevname=0 iommu.passthrough=1 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M bf2:~$ dmesg | grep -i crash [ 0.000000] crashkernel reserved: 0x00000000cfe00000 - 0x00000000efe00000 (512 MB) [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-1060-bluefield root=UUID=52ddbe2c-ee4f-48d4-b7d4-ab76e264e438 ro console=hvc0 console=ttyAMA0 earlycon=pl011,0x01000000 fixrtc net.ifnames=0 biosdevname=0 iommu.passthrough=1 crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M [ 8.070921] pstore: Using crash dump compression: deflate kdump-config is as below: bf2:~$ kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash crashkernel addr: 0x /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1060-bluefield kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-5.4.0-1060-bluefield current state: ready to kdump kexec command: /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1060-bluefield root=UUID=52ddbe2c-ee4f-48d4-b7d4-ab76e264e438 ro console=hvc0 console=ttyAMA0 earlycon=pl011,0x01000000 fixrtc net.ifnames=0 biosdevname=0 iommu.passthrough=1 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz sysrq: bf2:/# cat /proc/sys/kernel/sysrq 176 After trigged the crash manually with "echo c > /proc/sysrq-trigger", the system could not come up because of OOM. And after change the crashkernel with 1024M memory it still hangs. With default 512M, it hangs at "Killed process 674" [ 8.718188] systemd-journald[368]: File /var/log/journal/8244d38b2f804fc692f3f2dbf8206f57/system.journal corrupted or uncleanly shut down, renaming and re. [ 30.252513] Out of memory: Killed process 651 (systemd-resolve) total-vm:24380kB, anon-rss:3812kB, file-rss:1828kB, shmem-rss:0kB, UID:101 pgtables:80kB o0 ... [ 34.651927] Out of memory: Killed process 674 (dbus-daemon) total-vm:7884kB, anon-rss:552kB, file-rss:1380kB, shmem-rss:0kB, UID:103 pgtables:52kB oom_sco0 With 1024M, it hangs at following [ 8.733323] systemd-journald[369]: File /var/log/journal/8244d38b2f804fc692f3f2dbf8206f57/system.journal corrupted or uncleanly shut down, renaming and re. After soft reboot the Bluefield, there's no coredump file generated. bf2:~$ ls /var/crash/ -la total 52 drwxrwxrwt 3 root root 4096 May 31 01:43 . drwxr-xr-x 14 root root 4096 Apr 30 11:26 .. drwxrwxr-x 2 ubuntu ubuntu 4096 May 31 01:43 202305310143 -rw-r----- 1 root root 34307 May 31 01:18 _usr_share_netplan_netplan.script.0.crash -rw-r--r-- 1 root root 0 May 31 03:47 kdump_lock -rw-r--r-- 1 root root 358 May 31 03:48 kexec_cmd bf2:~$ ls /var/crash/202305310143/ -la total 8 drwxrwxr-x 2 ubuntu ubuntu 4096 May 31 01:43 . drwxrwxrwt 3 root root 4096 May 31 01:43 .. This issue also happens on 5.4.0-1049-bluefield kernel. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-bluefield/+bug/2021930/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
