[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2020-01-16 11:02 EDT--- (In reply to comment #78) > Hello Hari, > > Glad to see you around on bugs again. :) Thanks, Mauricio :) > Could you please confirm whether the fix for this bug also addresses LP bug > 1655280? > (i.e., the verification for xenial would be equivalent/also validate that > other bug.) Yes, it does. Updated that bug.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-10-25 06:29 EDT---
(In reply to comment #49)
> New version on ppa:cascardo/ppa for bionic. That should handle fadump and
> lock in the case of try-reload/condreload.
>
> Hari, can you give it a try?
Thanks, Cascardo. That works well for FADump case.
But is failing for KDump case though. With the below change
on top of the kdump-tools package you shared, things work
as expected for KDump case too:
---
diff --git a/usr/sbin/kdump-config.orig b/usr/sbin/kdump-config
index 08fe301..fd5e469 100755
--- a/usr/sbin/kdump-config.orig
+++ b/usr/sbin/kdump-config
@@ -923,7 +923,7 @@ reload()
condreload()
{
- local $sys_loaded="$sys_kexec_crash"
+ local sys_loaded="$sys_kexec_crash"
if [ "$DUMP_MODE" == "fadump" ] ; then
check_fadump_support
sys_loaded="$sys_fadump_registered"
---
Thanks
Hari
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1828596
Title:
kdump fails when crash is triggered after DLPAR cpu add operation
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-09-25 02:41 EDT--- (In reply to comment #47) > Hi, Hari. > > makedumpfile 1:1.6.5-1ubuntu1~18.04.2+cascardo2 on ppa:cascardo/ppa uses a > try-reload instead. Can you test it, please? Cascardo, try-reload is not considering fadump case (supported on powerpc). For fadump case, need to check whether "/sys/kernel/fadump_registered" is `1` before proceeding with unload/load.. A suggestion I have is to check for "systemctl is-active kdump-tools" and run "kdump-config reload" if it returns true, instead of "kdump-config try-reload" as that should cover for both kdump and fadump cases. Also, shouldn't we account for races when multiple udev events are triggered simultaneously by using locks or such? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-09-23 13:28 EDT--- Sorry about the delay. Observed that kdump/fadump is loaded even when kdump-tools service is disabled. Not desirable, I guess. Probably need to check if kdump-tools service is active before trying a reload? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-07-26 06:48 EDT--- Guilherme, the initial fix (udev rule) is still available. But while testing I observed failure due to systemd ratelimiting. I proposed to disable ratelimit but IIUC, Cascardo preferred a different approach that does not involve disabling systemd ratelimit and provided an updated package with a different approach to solve ratelimiting. My recent comment is that there is no updated package but just the initial fix. Hope that clears it up.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-07-25 05:43 EDT--- (In reply to comment #27) > Disabling the ratelimit in general would break other failure modes, so I > would rather just reset-failed when calling try-restart because of the > hotplug events. > > Can you try the package in ppa:cascardo/kdump2? Packages for eoan, disco and > bionic available. Cascardo, is the fix package you are proposing still here? I see the below package version: ii kdump-tools 1:1.6.5-1ubuntu2~18.04.1 which doesn't seem to have "systemctl reset-failed kdump-tools" invoked anywhere. I was trying this out on bionic with 5.0.0-17-generic kernel and the issue is reproducible.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-07-15 06:36 EDT--- Cascardo, I did not tinker with other options but disabling ratelimit helped: "StartLimitInterval=0" "systemctl reset-failed kdump-tools.service" seems like a good option but may not be needed if ratelimit is disabled.. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1828596] Comment bridged from LTC Bugzilla
On Mon, Jun 24, 2019 at 11:59:48AM -, bugproxy wrote: > --- Comment From hbath...@in.ibm.com 2019-06-24 07:49 EDT--- > Thanks for the change. With it, try-restart is being triggered for > kdump-tools service after CPU add operation but systemd reported > failure with below logs: > > Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service. > Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture > service... > Jun 24 06:47:06 ubuntu kdump-tools[2023]: Starting kdump-tools: * Creating > symlink /var/lib/kdump/vmlinuz > Jun 24 06:47:06 ubuntu kdump-tools[2023]: * Creating symlink > /var/lib/kdump/initrd.img > Jun 24 06:47:06 ubuntu kdump-tools[2023]: Modified > cmdline:BOOT_IMAGE=/vmlinux-5.0.0-17-generic root=/dev/mapper/ubuntu--vg-root > ro systemd.unit=kdump-tools-dump.service maxcpus=1 irqpo > Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Main process exited, > code=killed, status=15/TERM > Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result > 'signal'. > Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service. > Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture > service... > Jun 24 06:47:06 ubuntu kdump-tools[2071]: Starting kdump-tools: * Creating > symlink /var/lib/kdump/vmlinuz > Jun 24 06:47:06 ubuntu kdump-tools[2071]: * Creating symlink > /var/lib/kdump/initrd.img > Jun 24 06:47:06 ubuntu kdump-tools[2071]: Modified > cmdline:BOOT_IMAGE=/vmlinux-5.0.0-17-generic root=/dev/mapper/ubuntu--vg-root > ro systemd.unit=kdump-tools-dump.service maxcpus=1 irqpo > Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Main process exited, > code=killed, status=15/TERM > Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result > 'signal'. > Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service. > Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Start request > repeated too quickly. > Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result > 'signal'. > Jun 24 06:47:06 ubuntu systemd[1]: Failed to start Kernel crash dump capture > service. > > --- > Looks like a ratelimit issue with systemd. Is there some systemd option to > workaround it? > > I am running the below command on a PowerVM machine: > > # drmgr -c cpu -r -q 1 (to remove a core) > # drmgr -c cpu -a -q 1 (to add it back -> this triggers 8 CPU online udev > events as SMT is 8) > > To conclude, udev rule alone is not sufficient. Need a way to address the > multiple > requests at once.. There are these systemd options, which default to a burst limit of 5 restart in the interval of 10s. StartLimitIntervalSec=interval, StartLimitBurst=burst One other option that I prefer, howerver, is resetting the start rate limit counter by using systemctl reset-failed kdump-tools.service on the udev rule. Can you try that? Thanks. Cascardo. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-06-24 07:49 EDT--- Thanks for the change. With it, try-restart is being triggered for kdump-tools service after CPU add operation but systemd reported failure with below logs: Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service. Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture service... Jun 24 06:47:06 ubuntu kdump-tools[2023]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz Jun 24 06:47:06 ubuntu kdump-tools[2023]: * Creating symlink /var/lib/kdump/initrd.img Jun 24 06:47:06 ubuntu kdump-tools[2023]: Modified cmdline:BOOT_IMAGE=/vmlinux-5.0.0-17-generic root=/dev/mapper/ubuntu--vg-root ro systemd.unit=kdump-tools-dump.service maxcpus=1 irqpo Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Main process exited, code=killed, status=15/TERM Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result 'signal'. Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service. Jun 24 06:47:06 ubuntu systemd[1]: Starting Kernel crash dump capture service... Jun 24 06:47:06 ubuntu kdump-tools[2071]: Starting kdump-tools: * Creating symlink /var/lib/kdump/vmlinuz Jun 24 06:47:06 ubuntu kdump-tools[2071]: * Creating symlink /var/lib/kdump/initrd.img Jun 24 06:47:06 ubuntu kdump-tools[2071]: Modified cmdline:BOOT_IMAGE=/vmlinux-5.0.0-17-generic root=/dev/mapper/ubuntu--vg-root ro systemd.unit=kdump-tools-dump.service maxcpus=1 irqpo Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Main process exited, code=killed, status=15/TERM Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result 'signal'. Jun 24 06:47:06 ubuntu systemd[1]: Stopped Kernel crash dump capture service. Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Start request repeated too quickly. Jun 24 06:47:06 ubuntu systemd[1]: kdump-tools.service: Failed with result 'signal'. Jun 24 06:47:06 ubuntu systemd[1]: Failed to start Kernel crash dump capture service. --- Looks like a ratelimit issue with systemd. Is there some systemd option to workaround it? I am running the below command on a PowerVM machine: # drmgr -c cpu -r -q 1 (to remove a core) # drmgr -c cpu -a -q 1 (to add it back -> this triggers 8 CPU online udev events as SMT is 8) To conclude, udev rule alone is not sufficient. Need a way to address the multiple requests at once.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-05-22 07:16 EDT--- (In reply to comment #12) [...] > But I thought this rules were there by accident and the entries would be put > under arch flags to avoid them for ppc64.. If that is too much to ask, I am fine with the current change. The change works as expected.. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-05-22 02:33 EDT--- (In reply to comment #11) > Hi, Hari. > > So, as you said, other architectures will use add/remove instead of online, > and we want to support them too. Any reason not to do it that you are > thinking of? No action with these rules on ppc64 as ADD/REMOVE events are not ejected for CPU subsystem as of today. So, they don't have any impact and can be ignored. But I thought this rules were there by accident and the entries would be put under arch flags to avoid them for ppc64.. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1828596] Comment bridged from LTC Bugzilla
--- Comment From hbath...@in.ibm.com 2019-05-21 06:16 EDT--- Cascardo, the udev rules (/lib/udev/rules.d/50-kdump-tools.rules) should have been: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" but the package has: SUBSYSTEM=="memory", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="memory", ACTION=="offline", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="add", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="remove", PROGRAM="/bin/systemctl try-restart kdump-tools.service" SUBSYSTEM=="cpu", ACTION=="online", PROGRAM="/bin/systemctl try-restart kdump-tools.service" Can we get that sorted.. Thanks Hari -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1828596 Title: kdump fails when crash is triggered after DLPAR cpu add operation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1828596/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
