[Kernel-packages] [Bug 2021930] Re: kdump cannot generate coredump file on bluefield with 5.4 kernel

2023-11-27 Thread William Tu
adding "-d" to kexec which shows debugging info. I did not see any error...

[  441.468322] kdump-tools[7548]: arch_process_options:178: command_line: 
BOOT_IMAGE=/boot/vmlinuz-5.15.0-1031-bluefield root=1
[  441.468586] kdump-tools[7548]: arch_process_options:180: initrd: 
/var/lib/kdump/initrd.img
[  441.468726] kdump-tools[7548]: arch_process_options:182: dtb: (null)
[  441.468827] kdump-tools[7548]: arch_process_options:185: console: (null)
[  441.468936] kdump-tools[7548]: Try gzip decompression.
[  441.734545] kdump-tools[7548]: kernel: 0xa6946010 kernel_size: 0x2c9c198
[  441.734767] kdump-tools[7548]: set_phys_offset: phys_offset : 
8000 (method : vmcoreinfo pt_note)
[  441.734967] kdump-tools[7548]: get_memory_ranges:+[0] 8200 - 
87ff
[  441.735103] kdump-tools[7548]: get_memory_ranges:+[1] 88002000 - 
fffd
[  441.735209] kdump-tools[7548]: get_memory_ranges:-  b900 - 
bcff
[  441.735308] kdump-tools[7548]: get_memory_ranges:-  fd00 - 
feff
[  441.735391] kdump-tools[7548]: get_memory_ranges:+[4] 0001 - 
00085a0b
[  441.735467] kdump-tools[7548]: get_memory_ranges:-  0001001ea000 - 
0001001eafff
[  441.735568] kdump-tools[7548]: get_memory_ranges:-  00010022 - 
00010022
[  441.735655] kdump-tools[7548]: get_memory_ranges:-  00010023 - 
00010023
[  441.735737] kdump-tools[7548]: get_memory_ranges:-  00010024 - 
00010024
[  441.735813] kdump-tools[7548]: get_memory_ranges:-  00010025 - 
00010025
[  441.735885] kdump-tools[7548]: get_memory_ranges:-  00010026 - 
00010026
[  441.735968] kdump-tools[7548]: get_memory_ranges:-  00010027 - 
00010027
[  441.736117] kdump-tools[7548]: get_memory_ranges:-  00010028 - 
00010028
[  441.736224] kdump-tools[7548]: get_memory_ranges:-  00010029 - 
00010029
[  441.736306] kdump-tools[7548]: get_memory_ranges:-  0001002a - 
0001002a
[  441.736380] kdump-tools[7548]: get_memory_ranges:-  0001002b - 
0001002b
[  441.736481] kdump-tools[7548]: get_memory_ranges:-  0001002c - 
0001002c
[  441.736570] kdump-tools[7548]: get_memory_ranges:-  0001002d - 
0001002d
[  441.736650] kdump-tools[7548]: get_memory_ranges:-  0001002e - 
0001002e
[  441.736737] kdump-tools[7548]: get_memory_ranges:-  0001002f - 
0001002f
[  441.736815] kdump-tools[7548]: get_memory_ranges:-  00010030 - 
00010030
[  441.736887] kdump-tools[7548]: get_memory_ranges:-  00010031 - 
00010031
[  441.736957] kdump-tools[7548]: get_memory_ranges:-  00010032 - 
00010032
[  441.737028] kdump-tools[7548]: get_memory_ranges:-  000276d5 - 
0002776a
[  441.737096] kdump-tools[7548]: get_memory_ranges:-  00082f40 - 
00084ebf
[  441.737240] kdump-tools[7548]: get_memory_ranges:-  00084ed5 - 
00084ed50fff
[  441.737392] kdump-tools[7548]: get_memory_ranges:-  000851d3 - 
000851d3
[  441.737504] kdump-tools[7548]: get_memory_ranges:-  00085214b000 - 
000854ee6fff
[  441.737588] kdump-tools[7548]: get_memory_ranges:-  0008570c - 
00085a0b
[  441.737665] kdump-tools[7548]: get_memory_ranges:+[12] 00085a0d - 
00085a1a
[  441.737740] kdump-tools[7548]: get_memory_ranges:+[13] 00085a2a - 
00085a49afff
[  441.737828] kdump-tools[7548]: get_memory_ranges:-  00085a2ac000 - 
00085a2acfff
[  441.737912] kdump-tools[7548]: get_memory_ranges:+[15] 00085a4f7000 - 
00085a4f
[  441.738007] kdump-tools[7548]: get_memory_ranges:+[16] 00085a64 - 
00085a79
[  441.738099] kdump-tools[7548]: get_memory_ranges:+[17] 00085a7c - 
00085a7c
[  441.738180] kdump-tools[7548]: get_memory_ranges:+[18] 00085a91 - 
00085a92
[  441.738253] kdump-tools[7548]: get_memory_ranges:+[19] 00085a9d - 
00085aa0
[  441.738328] kdump-tools[7548]: get_memory_ranges:-  00085a9d - 
00085a9d0fff
[  441.738399] kdump-tools[7548]: get_memory_ranges:+[20] 00085acc - 
00085ad9
[  441.738486] kdump-tools[7548]: get_memory_ranges:-  00085ad9 - 
00085ad9
[  441.738567] kdump-tools[7548]: get_memory_ranges:+[21] 00085af8 - 
00085ff9
[  441.738642] kdump-tools[7548]: get_memory_ranges:-  00085bde4000 - 
00085c083fff
[  441.738736] kdump-tools[7548]: get_memory_ranges:-  00085c086000 - 
00085c087fff
[  441.738822] kdump-tools[7548]: get_memory_ranges:-  00085c088000 - 
00085c088fff
[  441.738895] kdump-tools[7548]: get_memory_ranges:-  00085c089000 - 

[Kernel-packages] [Bug 2021930] Re: kdump cannot generate coredump file on bluefield with 5.4 kernel

2023-11-27 Thread William Tu
I also tested it on 5.15.0-1031-bluefield and it also fails.

Configurations:

root@bu-oob:~# kdump-config show
DUMP_MODE:  kdump
USE_KDUMP:  1
KDUMP_COREDIR:  /var/crash
crashkernel addr: 0xbd00
   /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.15.0-1031-bluefield
kdump initrd:
   /var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-5.15.0-1031-bluefield
current state:ready to kdump

kexec command:
  /sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.15.0-1031-bluefield 
root=UUID=8e8b38a6-7d3d-4a29-b7a0-99761624f941 ro console=hvc0 console=ttyAMA0 
earlycon=pl011,0x1301 fixrtc net.ifnames=0 biosdevname=0 
iommu.passthrough=1 console=tty1 console=ttyS0 reset_devices 
systemd.unit=kdump-tools-dump.service nr_cpus=1" 
--initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
root@bu-lab60v3-oob:~# 

###
root@bu-oob:~# dmesg |grep -i crash
[0.00] crashkernel reserved: 0xbd00 - 0xfd00 
(1024 MB)
[0.00] Kernel command line: 
BOOT_IMAGE=/boot/vmlinuz-5.15.0-1031-bluefield 
root=UUID=8e8b38a6-7d3d-4a29-b7a0-99761624f941 ro console=hvc0 console=ttyAMA0 
earlycon=pl011,0x1301 fixrtc net.ifnames=0 biosdevname=0 
iommu.passthrough=1 console=tty1 console=ttyS0 
crashkernel=2G-4G:320M,4G-32G:1024M,32G-64G:1536M,64G-128G:2048M,128G-:4096M
[5.230439] pstore: Using crash dump compression: deflate
root@bu-oob:~# 


root@bu-oob:~# cat /etc/default/grub.d/kdump-tools.cfg
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT 
crashkernel=2G-4G:320M,4G-32G:1024M,32G-64G:1536M,64G-128G:2048M,128G-:4096M"

root@bu-lab60v3-oob:~# grep -e "CRASH" -e "KEXEC" 
/boot/config-5.15.0-1031-bluefield
CONFIG_KEXEC=y
CONFIG_KEXEC_FILE=y
CONFIG_KEXEC_SIG=y
CONFIG_KEXEC_IMAGE_VERIFY_SIG=y
CONFIG_CRASH_DUMP=y
CONFIG_CRASH_CORE=y
CONFIG_KEXEC_CORE=y
CONFIG_HAVE_IMA_KEXEC=y
CONFIG_IMA_KEXEC=y

*** How to reproduce ***
When manually triggers the crash "echo c > /proc/sysrq-trigger" 
the system just hangs without showing any message/log.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-bluefield in Ubuntu.
https://bugs.launchpad.net/bugs/2021930

Title:
  kdump cannot generate coredump file on bluefield with 5.4  and 5.15
  kernel

Status in linux-bluefield package in Ubuntu:
  New

Bug description:
  kdump cannot generate coredump file on bluefield with 5.4 kernel

  Bug description:

  Following the instruction in 
https://ubuntu.com/server/docs/kernel-crash-dump, the coredump file cannot be 
generated.

  Bluefield is running 5.4 kernel
bf2:~$ uname -a
Linux sw-mtx-008-bf2 5.4.0-1060-bluefield #66-Ubuntu SMP PREEMPT Mon 
Mar 27 15:52:50 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

  crashkernel parameter is configured
bf2:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.4.0-1060-bluefield 
root=UUID=52ddbe2c-ee4f-48d4-b7d4-ab76e264e438 ro console=hvc0 console=ttyAMA0 
earlycon=pl011,0x0100 fixrtc net.ifnames=0 biosdevname=0 
iommu.passthrough=1 
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
bf2:~$ dmesg | grep -i crash
[0.00] crashkernel reserved: 0xcfe0 - 
0xefe0 (512 MB)
[0.00] Kernel command line: 
BOOT_IMAGE=/boot/vmlinuz-5.4.0-1060-bluefield 
root=UUID=52ddbe2c-ee4f-48d4-b7d4-ab76e264e438 ro console=hvc0 console=ttyAMA0 
earlycon=pl011,0x0100 fixrtc net.ifnames=0 biosdevname=0 
iommu.passthrough=1 
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M
[8.070921] pstore: Using crash dump compression: deflate

  kdump-config is as below:
bf2:~$ kdump-config show
DUMP_MODE:kdump
USE_KDUMP:1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR:/var/crash
crashkernel addr: 0x
/var/lib/kdump/vmlinuz: symbolic link to 
/boot/vmlinuz-5.4.0-1060-bluefield
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-5.4.0-1060-bluefield
current state:ready to kdump

kexec command:
/sbin/kexec -p 
--command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1060-bluefield 
root=UUID=52ddbe2c-ee4f-48d4-b7d4-ab76e264e438 ro console=hvc0 console=ttyAMA0 
earlycon=pl011,0x0100 fixrtc net.ifnames=0 biosdevname=0 
iommu.passthrough=1 reset_devices systemd.unit=kdump-tools-dump.service 
nr_cpus=1" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

  sysrq:
bf2:/# cat /proc/sys/kernel/sysrq
176

  After trigged the crash manually with "echo c > /proc/sysrq-trigger", the 
system could not come up because of OOM. And after change the crashkernel with 
1024M memory it still hangs.
With default 512M, it hangs at "Killed process 674"