[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-05-20 13:34 EDT--- (In reply to comment #61) > Issue is now "Fix Released" everywhere except Xenial. Should this now be > backported to Xenial or would that incur too much regression risk? No regressions expected but would be better off taking this along with the fix for follow-up LP bug 1828596 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-05-10 11:40 EDT--- (In reply to comment #53) [...] > bionic? I know newer kernels may have regressed when crashing from the > non-boot CPU, which should affect 4.18 kernels and later, but not 4.15. Raised launchpad bug 1828597 to follow-up on this.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-05-10 11:32 EDT--- (In reply to comment #55) > (In reply to comment #53) > [...] > > could proceed with the SRU, and open a new bug to fix the CPU hotplug case, > > if there is one. > > Raised bug 177551 to follow-up on the problem in CPU hot-add case which Bug 1828596 is the corresponding launchpad bug.. > is currently being screened internally. Will update the launchpad bug number > once we have it.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-05-10 07:10 EDT--- (In reply to comment #53) [...] > could proceed with the SRU, and open a new bug to fix the CPU hotplug case, > if there is one. Raised bug 177551 to follow-up on the problem in CPU hot-add case which is currently being screened internally. Will update the launchpad bug number once we have it.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-05-07 15:03 EDT--- While I initially suggested the udev below rules: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" request you use the below rules instead: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" because ppc64 does not follow the standard CPU hot add framework where add/remove event is ejected on CPU hot add/remove operation. It only ejects online/offline event to user space when CPU is hot added/removed. This is because /sys/devices/system/cpu/cpuX nodes are present for all "possible", irrespective of whether a CPU is hot-added/removed. So, a udev event only for CPU hot add (online) is sufficient to pass on an updated DT blob for kdump kernel to scan for CPUs... Also, ran into a couple of other bugs (IBM bug 177451 & IBM bug 177452) while validating this fix on bionic 4.15.0-48-generic and 4.18.0-18-generic. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2018-08-11 08:48 EDT--- After upgrading to kdump-tools_1.6.4-1~16.04.0cascardo2_ppc64el.deb package, dump capture succeeds without any complaints: The udev rules trigger kdump-tools service reload as can be seen below: -- root@ubuntu:~# dpkg -l | grep kdump-tools ii kdump-tools 1:1.6.4-1~16.04.0cascardo2 ppc64el scripts and tools for automating kdump (Linux crash dumps) root@ubuntu:~# root@ubuntu:~# root@ubuntu:~# root@ubuntu:~# systemctl status kdump-tools.service -l ? kdump-tools.service - Kernel crash dump capture service Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; vendor pres Active: active (exited) since Sat 2018-08-11 08:33:27 EDT; 2min 8s ago Process: 25166 ExecStop=/etc/init.d/kdump-tools stop (code=exited, status=0/SU Process: 25200 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/ Main PID: 25200 (code=exited, status=0/SUCCESS) Aug 11 08:33:26 ubuntu systemd[1]: Starting Kernel crash dump capture service... Aug 11 08:33:26 ubuntu kdump-tools[25200]: Starting kdump-tools: * Creating sym Aug 11 08:33:26 ubuntu kdump-tools[25200]: * Creating symlink /var/lib/kdump/in Aug 11 08:33:26 ubuntu kdump-tools[25200]: Modified cmdline:BOOT_IMAGE=/boot/vml Aug 11 08:33:26 ubuntu kdump-tools[25200]: * loaded kdump kernel Aug 11 08:33:26 ubuntu kdump-tools[25248]: /sbin/kexec -p --command-line="BOOT_I Aug 11 08:33:27 ubuntu kdump-tools[25249]: loaded kdump kernel Aug 11 08:33:27 ubuntu systemd[1]: Started Kernel crash dump capture service. root@ubuntu:~# root@ubuntu:~# root@ubuntu:~# root@ubuntu:~# root@ubuntu:~# drmgr -c mem -r -q 1 Validating Memory DLPAR capability...yes. [1459216.723209] pseries-hotplug-mem: Attempting to hot-remove 1 LMB(s) [1459216.749014] Offlined Pages 4096 [1459193.925176] kdump-tools[25419]: Stopping kdump-tools: * unloaded kdump kernel [1459193.985143] kdump-tools[25453]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz [1459193.986120] kdump-tools[25453]: * Creating symlink /var/lib/kdump/initrd.img [1459194.024961] kdump-tools[25453]: Modified cmdline:BOOT_IMAGE=/boot/vmlinux-4.15.0-24-generic root=UUID=1aa9458c-3974-4cb4-9ab3-9ee03c0f4e5e ro xmon=on nr_cpus=1 systemd.unit=kdump-tools.service irqpoll noirqdistrib nousb elfcorehdr=158144K [1459217.005820] pseries-hotplug-mem: Memory at 4000 was hot-removed DR_TOTAL_RESOURCES=1 root@ubuntu:~# [1459194.297073] kdump-tools[25453]: * loaded kdump kernel root@ubuntu:~# root@ubuntu:~# root@ubuntu:~# root@ubuntu:~# systemctl status kdump-tools.service -l ? kdump-tools.service - Kernel crash dump capture service Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; vendor pres Active: active (exited) since Sat 2018-08-11 08:35:47 EDT; 12s ago Process: 25419 ExecStop=/etc/init.d/kdump-tools stop (code=exited, status=0/SU Process: 25453 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/ Main PID: 25453 (code=exited, status=0/SUCCESS) Aug 11 08:35:47 ubuntu systemd[1]: Starting Kernel crash dump capture service... Aug 11 08:35:47 ubuntu kdump-tools[25453]: Starting kdump-tools: * Creating sym Aug 11 08:35:47 ubuntu kdump-tools[25453]: * Creating symlink /var/lib/kdump/in Aug 11 08:35:47 ubuntu kdump-tools[25453]: Modified cmdline:BOOT_IMAGE=/boot/vml Aug 11 08:35:47 ubuntu kdump-tools[25453]: * loaded kdump kernel Aug 11 08:35:47 ubuntu kdump-tools[25505]: /sbin/kexec -p --command-line="BOOT_I Aug 11 08:35:47 ubuntu kdump-tools[25506]: loaded kdump kernel Aug 11 08:35:47 ubuntu systemd[1]: Started Kernel crash dump capture service. root@ubuntu:~# -- and dump captured successfully after memory dlpar operation: -- [ 139.944700] kdump-tools[1278]: Starting kdump-tools: * running makedumpfile -c -d 31 /proc/vmcore /var/crash/201808110838/dump-incomplete Copying data : [100.0 %] - eta: 0s [ 147.858420] kdump-tools[1278]: The kernel version is not supported. [ 147.858718] kdump-tools[1278]: The makedumpfile operation may be incomplete. [ 147.859002] kdump-tools[1278]: The dumpfile is saved to /var/crash/201808110838/dump-incomplete. [ 147.859260] kdump-tools[1278]: makedumpfile Completed. [ 147.899799] kdump-tools[1278]: * kdump-tools: saved vmcore in /var/crash/201808110838 [ 147.998442] kdump-tools[1278]: * running makedumpfile --dump-dmesg /proc/vmcore /var/crash/201808110838/dmesg.201808110838 [ 148.008063] kdump-tools[1278]: The kernel version is not supported. [ 148.008436] kdump-tools[1278]: The makedumpfile operation may be incomplete. [ 148.008672] kdump-tools[1278]: The dmesg log is saved to /var/crash/201808110838/dmesg.201808110838. [ 148.008913] kdump-tools[1278]: makedumpfile Completed. [ 148.009135] kdump-tools[1278]: * kdump-tools: saved dmesg content in /var/crash/201808110838 [ 148.033105] kdump-tools[1278]: Sat, 11 Aug 2018 08:38:51 -0400 [ 148.069165]
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2018-03-22 13:18 EDT--- (In reply to comment #32) > (In reply to comment #31) > > Could you please comment as to whether the workaround resolved the issue? > > Yes, the workaround resolves the issue. But could we have a udev rule added > in kdump-tools > to trigger kdump-tools.service restart after a DLPAR memory/CPU add/remove > operation > instead of manually restarting kdump-tools service after a DLPAR operation.. Basically, a udev rule file that looks like this: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" put in "/lib/udev/rules.d" dir as part of the kdump-tools package to avoid the need to manually reload kdump.. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2018-03-21 09:04 EDT--- (In reply to comment #31) > Could you please comment as to whether the workaround resolved the issue? Yes, the workaround resolves the issue. But could we have a udev rule added in kdump-tools to trigger kdump-tools.service restart after a DLPAR memory/CPU add/remove operation instead of manually restarting kdump-tools service after a DLPAR operation.. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2018-03-08 14:01 EDT--- (In reply to comment #29) > So, the workaround would be reloading the dump after DLPAR. In order to do > that, run: > > kdump-config unload ; kdump-config load > The "console log" attachment is basically the error seen when the above workaround is not used. > > It's not clear what the logs in comments #16 and #17 refer to. Could you > please clarify? The other attachment is the sosreport on the failed system.. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2017-08-31 04:54 EDT--- (In reply to comment #22) > Hi, Hari. Hello Cascardo, > > My understanding is that the kernel should handle that. Otherwise, whenever Sadly, it can't as kexec-tools is the one that creates elf headers for /proc/vmcore. So, with a change in available CPUs/memory, kdump kernel needs to be reloaded. Probably, with a udev event that does a try-restart of kdump-tool service on CPU/Memory hot add/remove operation. > the user manually loads a different kdump kernel, a memory hotplug would > cause the default kdump kernel to be loaded, not what the user has loaded. Can't do much about it. If need be, a user has the option to workaround this problem by adjusting the settings in /etc/default/kdump-tools file to load a different kdump kernel.. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04.2: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2017-08-22 10:32 EDT--- (In reply to comment #20) > Hi, hbathini. > Hello Cascardo, > Why is it needed to restart kdump-tools after a DLPAR? Does it refer to a Yes. It refers to either of memory or CPU hot add/remove operations, when kdump kernel needs to be reloaded - to rebuild the vmcore elf notes. > memory hotplug operation? What is LTC bug #150353? Is it mirrored to > launchpad already? Yes, it is mirrored. LTC Bug 150353 refers to Launchpad bug 1658968. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04.2: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From pavra...@in.ibm.com 2017-07-14 05:02 EDT--- Issue is observed even on 16.04.03. When crash is triggered after memory remove operation. Thanks, Pavithra -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04.2: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2017-01-23 01:45 EDT--- Hi Canonical, After a DLPAR operation, kdump-tools.service must be restarted for kdump/fadump to work properly.. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04.2: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1655280] Comment bridged from LTC Bugzilla
--- Comment From pt...@cn.ibm.com 2017-01-10 20:20 EDT--- (In reply to comment #7) > Assigning to Hari for some assistance but would like to know if this issue > can be recreated consistently and if so how and does this occur on other > LPARs running the same level? Looks like this bug only can be reproduced on roselp4 when triggering bug 150353. But looks like it is quite easy to trigger it when testing dlpar. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1655280 Title: ISST-LTE:pVM:roselp4:ubuntu 16.04.2: cp: error reading '/proc/vmcore': Bad address when trying to dump vmcore To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655280/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
