[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-06-25 12:13 EDT--- Michael, actually this sounds like LP 1828597 It is worth trying with maxcpus=1 instead of nr_cpus=1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From mranw...@us.ibm.com 2019-06-25 10:14 EDT--- I tried this on a witherspoon with NVMe, but I believe I also hit LP 1778844: ubuntu@tempbmc1-p1:~$ [ 116.427517] sysrq: SysRq : Trigger a crash [ 116.427566] Unable to handle kernel paging request for data at address 0x [ 116.427619] Faulting instruction address: 0xc08205b8 [ 116.427668] Oops: Kernel access of bad area, sig: 11 [#1] [ 116.427762] LE SMP NR_CPUS=2048 NUMA PowerNV [ 116.427838] Modules linked in: vmx_crypto crct10dif_vpmsum nouveau ast ttm drm_kms_helper drm at24 ofpart drm_panel_orientation_quirks fb_sys_fops syscopyarea cmdlinepart sysfillrect ipmi_powernv sysimgblt powernv_flash ipmi_devintf i2c_algo_bit ibmpowernv mtd ipmi_msghandler opal_prd uio_pdrv_genirq uio sch_fq_codel ip_tables x_tables autofs4 nvme crc32c_vpmsum nvme_core ahci tg3 libahci [ 116.428241] CPU: 80 PID: 5322 Comm: bash Kdump: loaded Not tainted 4.18.0-24-generic #25-Ubuntu [ 116.428366] NIP: c08205b8 LR: c0821444 CTR: c0820590 [ 116.428464] REGS: c00020393dc5fa00 TRAP: 0300 Not tainted (4.18.0-24-generic) [ 116.428581] MSR: 90009033 CR: 2848 XER: 2004 [ 116.428675] CFAR: c0821440 DAR: DSISR: 4200 IRQMASK: 0 [ 116.428675] GPR00: c0821444 c00020393dc5fc80 c177ca00 0063 [ 116.428675] GPR04: 0001 04f1 90009033 35c80060 [ 116.428675] GPR08: 0007 0001 90001003 [ 116.428675] GPR12: c0820590 c000203fff652480 0dda71119760 [ 116.428675] GPR16: 0dda7bd4d910 0dda710b4a48 0dda71119760 0dda71038b00 [ 116.428675] GPR20: 0001 0dda710c70b8 7fffd6124ea4 [ 116.428675] GPR24: 7fffd6124ea0 0dda7111afe4 718ed556bfa8 c16736b0 [ 116.428675] GPR28: 0004 0002 0063 c15f2e38 [ 116.429616] NIP [c08205b8] sysrq_handle_crash+0x28/0x30 [ 116.429708] LR [c0821444] __handle_sysrq+0xf4/0x230 [ 116.429792] Call Trace: [ 116.429843] [c00020393dc5fc80] [c0821428] __handle_sysrq+0xd8/0x230 (unreliable) [ 116.429976] [c00020393dc5fd10] [c0821bc8] write_sysrq_trigger+0x68/0x90 [ 116.430099] [c00020393dc5fd40] [c04a99a4] proc_reg_write+0x84/0xc0 [ 116.430202] [c00020393dc5fd70] [c03f992c] __vfs_write+0x3c/0x70 [ 116.430305] [c00020393dc5fd90] [c03f9bc8] vfs_write+0xd8/0x220 [ 116.430407] [c00020393dc5fde0] [c03f9f44] ksys_write+0x64/0x110 [ 116.430512] [c00020393dc5fe30] [c000b288] system_call+0x5c/0x70 [ 116.430605] Instruction dump: [ 116.430649] 4bfff9e1 4bfffe44 3c4c00f6 3842c470 7c0802a6 6000 3921 3d42001d [ 116.430760] 394a90f0 912a 7c0004ac 3940 <992a> 4e800020 3c4c00f6 3842c440 [ 116.430867] ---[ end trace 4f8cfae97c893934 ]--- [ 116.534209] [ 116.534308] Sending IPI to othe Looks like that might be blocking this one, I will get that mirrored over, too. --- Comment From mranw...@us.ibm.com 2019-06-25 10:16 EDT--- Sorry, previous comment should have said hitting LP 1832388. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From indira.pr...@in.ibm.com 2019-04-12 05:41 EDT--- Hi , Any update on this bug, when this fix is expected in official build. Please update the approximate date, as issue exists with latest Ubuntu180401 kernel - 4.15.0-47-generic. Regards, Indira -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-02-12 06:15 EDT--- (In reply to comment #87) > Hi, Andrew and Hari. > > The original issue is only resolved by using an initramfs-tools from my ppa. > We still need to get this fix included in our releases. I will start working > to get this in disco. > > Cascardo. Right, Cascardo. Waiting for the fix to be released in an official build.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-02-12 01:11 EDT--- (In reply to comment #83) [...] > Will try to increase to some more above 4096M and retry it. The recommendations are based on a few assumptions that don't apply to all machines. So, hitting OOM in this case is not surprising in that sense. As increasing crashkernel value helped, we can consider the original issue reported here resolved.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From indira.pr...@in.ibm.com 2019-02-06 09:45 EDT--- Triggered crash with patch using crashkernel=8192M and did not see OOM traces and dump copied properly as below root@woo:/var/crash/201902060928# pwd /var/crash/201902060928 root@woo:/var/crash/201902060928# ls -l total 469212 -rw--- 1 root root114308 Feb 6 09:29 dmesg.201902060928 -rw--- 1 root root 480474149 Feb 6 09:29 dump.201902060928 root@woo:/var/crash/201902060928# date Wed Feb 6 09:44:42 EST 2019 root@woo:/var/crash/201902060928# Regards, Indira -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From indira.pr...@in.ibm.com 2019-02-06 09:27 EDT--- (In reply to comment #82) > The system has too many devices, so too many drivers are using too much > memory. So, the system goes out-of-memory before makedumpfile is complete. > Please, reserve more memory on crashkernel, reboot the system and repeat the > crash test. > > To reserve more memory on crashkernel, edit file > /etc/default/grub.d/kdump-tools.cfg and change crashkernel value to > crashkernel=384M, for example, then run update-grub. > > Thanks. > Cascardo. Hi Cascardo As per SUSE document recommendation used 4096M crash kernel and with this value i have seen OOM traces during kdump. root@woo:~# free -h totalusedfree shared buff/cache available Mem: 251G7.2G243G 16M567M243G Swap: 2.0G 0B2.0G root@woo:~# cat /proc/cmdline root=UUID=df4c0331-9e6c-4592-9e0c-84f431ecb1f7 ro splash quiet crashkernel=4096M Will try to increase to some more above 4096M and retry it. Regards, Indira -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From indira.pr...@in.ibm.com 2019-02-06 01:54 EDT--- Have installed below patch and retried kdump scenario with root disk as nvme disk. Now system is not dropping to initramfs state its moving further but not saving dump file (core) properly. initramfs-tools_0.130ubuntu3.6+nvme1_all.deb initramfs-tools-bin_0.130ubuntu3.6+nvme1_ppc64el.deb initramfs-tools-core_0.130ubuntu3.6+nvme1_all.deb root@woo:~# dpkg -l | grep initramfs ii busybox-initramfs 1:1.27.2-2ubuntu3 ppc64el Standalone shell setup for initramfs ii initramfs-tools 0.130ubuntu3.6+nvme1 all generic modular initramfs generator (automation) ii initramfs-tools-bin 0.130ubuntu3.6+nvme1 ppc64el binaries used by initramfs-tools ii initramfs-tools-core 0.130ubuntu3.6+nvme1 all generic modular initramfs generator (core tools) ii libklibc 2.0.4-9ubuntu2 ppc64el minimal libc subset for use with initramfs root@woo:~# rm /var/lib/kdump/initr* reboot After reboot triggered crash > Attached console logs Regards, Indira -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2018-09-27 22:06 EDT--- (In reply to comment #61) > As a workaround, I suggest setting nvme-core.ko multipath=0 parameter. Can > you try that, rebooting, recreating the kdump-tools initrd, then crashing? Hi Cascardo, Where/How to set this parameters? You mean, passing them as boot parameters? Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: nvme multipath does not report path relationships To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2018-09-27 02:42 EDT--- Cascardo/Canonical, Did you get a chance to look at the logs attached? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From mainam...@in.ibm.com 2018-09-25 04:34 EDT--- Hello Canonical Please download the files using Anonymous FTP and download the files from following path ftp://testcase.software.ibm.com/fromibm/linux/lp1778844_sys.tar.gz Note: Files will remain in /fromibm/ for 3 business days before being deleted. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From indira.pr...@in.ibm.com 2018-09-20 07:42 EDT--- Unable to attach "lp1778844_sys.tar.gz" file because of size, please refer to logs path .Also test machine is taken back for other release testing. banner.isst.aus.stglabs.ibm.com [banner/don2rry ] : /home/banner/169067_sep20 Regards, Indira -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From indira.pr...@in.ibm.com 2018-09-20 07:06 EDT--- Hi, We lost the old environment as jmet got returned. Got a new Witherspoon jmet and retried the kdump scenario & issue recreated on latest ubuntu180401 kernel (4.15.0-34-generic) kernel. > Triggered kdump with default crashkernel parameter and it went to initramfs > state root@wax:~# cat /proc/cmdline root=UUID=38d0124f-14d8-4098-a746-03f9bda5c22e ro crashkernel=8192M splash quiet crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M > Collected console & dmesg logs from same state > Rebooted system & collected sosreport, tar czf lp1778844_sys.tar.gz /sys/ > Also tried comment#23 steps & collected /tmp/initrd.img root@wax:~# cat /etc/initramfs-tools/initramfs.conf | grep MODULES # MODULES: [ most | netboot | dep | list ] MODULES=dep root@wax:~# mkinitramfs -o /tmp/initrd.img W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast root@wax:~# uname -r 4.15.0-34-generic root@wax:~# ls -l /tmp/initrd.img -rw-r--r-- 1 root root 24128581 Sep 20 04:59 /tmp/initrd.img > Attaching all above logs & if i am unable to attach any of the logs because of size issue. Logs are placed in the path banner.isst.aus.stglabs.ibm.com [banner/don2rry ] : /home/banner/169067_sep20 > Attaching console logs from first step till last step ( starting from system booting with latest kernel level, trigger crash , initramfs state etc...) Thanks & Regards, Indira -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2018-07-25 01:28 EDT--- (In reply to comment #34) > Created attachment 128858 [details] > sosreport_moduels=set > > Attached sosreport when MODULES=set used Cascardo, is the sosreport no good? Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From indira.pr...@in.ibm.com 2018-07-24 06:36 EDT--- (In reply to comment #36) > Can you run apport-collect -p linux 1778844 ? > > Thank you. > Cascardo. HI Cascardo, I have tried to collect , but it fails as below root@woo:~# apport-collect -p linux 1778844 ERROR: connecting to Launchpad failed: [Errno 110] Connection timed out You can reset the credentials by removing the file "/root/.cache/apport/launchpad.credentials" root@woo:~# export http_proxy="http://10.33.11.31:3128; root@woo:~# apport-collect -p linux 1778844 ERROR: connecting to Launchpad failed: [Errno 110] Connection timed out You can reset the credentials by removing the file "/root/.cache/apport/launchpad.credentials" root@woo:~# Regards, Indira -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From indira.pr...@in.ibm.com 2018-07-23 08:14 EDT--- Also both logs were placed in banner@9.3.117.11 (don2rry) : /home/banner banner@banner:~> ls -l sosreport-woo-20180723065534.tar.xz -rw--- 1 banner banner 7171548 Jul 23 07:05 sosreport-woo-20180723065534.tar.xz banner@banner:~> ls -l initrd.img -rw-r--r-- 1 banner banner 24119847 Jul 23 07:04 initrd.img banner@banner:~> -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2018-07-23 07:30 EDT--- Indira, I think Cascardo is looking for /tmp/initrd.img-`uname -r` file whne MODULES=dep is used. Also, sosreport? (In reply to comment #26) > Created attachment 128855 [details] > woo_kdump_nvmedisk_console logs btw, looks like PHB initialization is failing. So, whether nvme driver is present or not in the initrd is secondary here, the device initialization itself could be at fault. What is the f/w level. Is it latest? Thanks Hari --- Comment From hbath...@in.ibm.com 2018-07-23 07:35 EDT--- (In reply to comment #28) > Indira, I think Cascardo is looking for /tmp/initrd.img-`uname -r` file > whne MODULES=dep is used. Also, sosreport? > > (In reply to comment #26) > > Created attachment 128855 [details] > > woo_kdump_nvmedisk_console logs > > btw, looks like PHB initialization is failing. So, whether nvme driver > is present or not in the initrd is secondary here, the device initialization > itself could be at fault. What is the f/w level. Is it latest? also, the initial console log is different from this.. Was the f/w on the system updated as well.. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1778844] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2018-07-09 06:18 EDT--- Canonical, any update on this..? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1778844 Title: ISST-LTE:PNV:Ubuntu180401:Witherspoon:woo: After triggering crash,kdump is not working and system enters into initramfs state To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1778844/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
