On Wed, Mar 15, 2023 at 01:12:05PM +0100, Christian Theune via
Openipmi-developer wrote:
> Ah, fantastic! That explains it of course … :)
>
> From my side I guess this works and I don’t have to retry with that, but I’d
> be happy to just wait for 5.10.175 … or would you prefer me explicitly
> testing your original?
We can just wait. The problem is obvious now, and the backports are in
progress.
Thanks for helping me with this.
-corey
>
> Christian
>
> > On 15. Mar 2023, at 13:07, Corey Minyard <miny...@acm.org> wrote:
> >
> > On Wed, Mar 15, 2023 at 07:32:41AM +0100, Christian Theune via
> > Openipmi-developer wrote:
> >> Hi,
> >>
> >> that didn’t apply on 5.10. Here’s what I’m currently trying to build after
> >> manually inspecting the rejected patch:
> >>
> >
> > Well, I guess I should have sent the prerequisite patch, too. Her it
> > is:
> >
> > a01a89b1db ("ipmi/watchdog: replace atomic_add() and atomic_sub()")
> >
> > Also attached.
> >
> > -corey
> >
> >>
> >>
> >>> On 14. Mar 2023, at 18:29, Corey Minyard <miny...@acm.org> wrote:
> >>>
> >>> Well, dang, I had already fixed this a year and a half ago. I wish I
> >>> had a better memory.
> >>>
> >>> Anyway, the fix is commit db05ddf7f321634c5659a0cf7ea56594e22365f7
> >>> ("ipmi:watchdog: Set panic count to proper value on a panic") in
> >>> mainstream 5.16. I'm attaching that patch.
> >>>
> >>> -corey
> >>>
> >>> On Tue, Mar 14, 2023 at 03:58:26PM +0100, Christian Theune via
> >>> Openipmi-developer wrote:
> >>>> Awesome!
> >>>>
> >>>>> On 14. Mar 2023, at 15:54, Corey Minyard <miny...@acm.org> wrote:
> >>>>>
> >>>>> On Tue, Mar 14, 2023 at 03:22:39PM +0100, Christian Theune via
> >>>>> Openipmi-developer wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> sorry, I didn’t expect you to make me a branch. I had already taken
> >>>>>> your diff over to 5.10 as it applied cleanly … sorry for the
> >>>>>> additional work and thanks anyways.
> >>>>>
> >>>>> Ok, that's great. It's something about the IPMI watchdog panic
> >>>>> routines, and I can reproduce. I should be able to fix this pretty
> >>>>> quickly. I'll send a patch when I get this fixed.
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> -corey
> >>>>>
> >>>>>>
> >>>>>> Here’s the output:
> >>>>>>
> >>>>>> [ 6521.905890] sysrq: Trigger a crash
> >>>>>> [ 6521.909294] Kernel panic - not syncing: sysrq triggered crash
> >>>>>> [ 6521.915026] CPU: 1 PID: 43785 Comm: bash Tainted: G I
> >>>>>> 5.10.159 #1-NixOS
> >>>>>> [ 6521.922925] Hardware name: Dell Inc. PowerEdge R510/00HDP0, BIOS
> >>>>>> 1.11.0 07/23/2012
> >>>>>> [ 6521.930475] Call Trace:
> >>>>>> [ 6521.932923] dump_stack+0x6b/0x83
> >>>>>> [ 6521.936230] panic+0x101/0x2c8
> >>>>>> [ 6521.939276] ? printk+0x58/0x73
> >>>>>> [ 6521.942408] sysrq_handle_crash+0x16/0x20
> >>>>>> [ 6521.946407] __handle_sysrq.cold+0x43/0x11a
> >>>>>> [ 6521.950580] write_sysrq_trigger+0x24/0x40
> >>>>>> [ 6521.954668] proc_reg_write+0x51/0x90
> >>>>>> [ 6521.958322] vfs_write+0xc3/0x280
> >>>>>> [ 6521.961627] ksys_write+0x5f/0xe0
> >>>>>> [ 6521.964935] do_syscall_64+0x33/0x40
> >>>>>> [ 6521.968502] entry_SYSCALL_64_after_hwframe+0x61/0xc6
> >>>>>> [ 6521.973540] RIP: 0033:0x7f2c6b91a133
> >>>>>> [ 6521.977106] Code: 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b3
> >>>>>> 0f 1f 80 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75 14 b8 01 00 00
> >>>>>> 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40 00 41 54 49 89 d4 55 48
> >>>>>> 89 f5
> >>>>>> [ 6521.995836] RSP: 002b:00007ffc4cf11088 EFLAGS: 00000246 ORIG_RAX:
> >>>>>> 0000000000000001
> >>>>>> [ 6522.003387] RAX: ffffffffffffffda RBX: 0000000000000002 RCX:
> >>>>>> 00007f2c6b91a133
> >>>>>> [ 6522.010505] RDX: 0000000000000002 RSI: 0000000001555c08 RDI:
> >>>>>> 0000000000000001
> >>>>>> [ 6522.017623] RBP: 0000000001555c08 R08: 000000000000000a R09:
> >>>>>> 00007f2c6b9aaf40
> >>>>>> [ 6522.024743] R10: 00000000016e4218 R11: 0000000000000246 R12:
> >>>>>> 0000000000000002
> >>>>>> [ 6522.031864] R13: 00007f2c6b9e8520 R14: 00007f2c6b9e8720 R15:
> >>>>>> 0000000000000002
> >>>>>> [ 6522.039085] Calling notifier panic_event+0x0/0x410
> >>>>>> [ipmi_msghandler] (000000008eb8cb44)
> >>>>>> [ 6522.047071] IPMI message handler: IPMI: panic event handler
> >>>>>> [ 6522.052628] IPMI message handler: IPMI: handling panic event for
> >>>>>> intf 0: 00000000443777b3 0000000067d05ff8
> >>>>>> …
> >>>>>> and then it reboots after the 255 seconds from the watchdog timer are
> >>>>>> passed.
> >>>>>>
> >>>>>> Christian
> >>>>>>
> >>>>>>> On 13. Mar 2023, at 18:13, Corey Minyard <miny...@acm.org> wrote:
> >>>>>>>
> >>>>>>> On Mon, Mar 13, 2023 at 05:42:39PM +0100, Christian Theune wrote:
> >>>>>>>> Hrghs. I’m applying your patch to 5.10 as my distro build
> >>>>>>>> infrastructure has some patches that don’t apply to 6.2 and that I
> >>>>>>>> don’t know how to circumvent quickly enough… :)
> >>>>>>>
> >>>>>>> Ok, there's a
> >>>>>>>
> >>>>>>> https://github.com/cminyard/linux-ipmi.git:debug-panic-oem-events-5.10
> >>>>>>>
> >>>>>>> branch available for you to pull. It's on top of latest 5.10.
> >>>>>>>
> >>>>>>> -corey
> >>>>>>>
> >>>>>>>>
> >>>>>>>>> On 13. Mar 2023, at 16:59, Christian Theune <c...@flyingcircus.io>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> I should be easily able to run 6.2, no worries.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On 13. Mar 2023, at 16:33, Corey Minyard <miny...@acm.org> wrote:
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Mar 13, 2023 at 02:07:01PM +0100, Christian Theune wrote:
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> yeah, the IPMI log is fine. This is a 10 minute interval job in
> >>>>>>>>>>> our system that exports the log and clears it:
> >>>>>>>>>>>
> >>>>>>>>>>> The job looks like this:
> >>>>>>>>>>>
> >>>>>>>>>>> /nix/store/m7lb36dr93qj27r9vskmjihz8imywy86-ipmitool-1.8.18/bin/ipmitool
> >>>>>>>>>>> sel elist
> >>>>>>>>>>> /nix/store/m7lb36dr93qj27r9vskmjihz8imywy86-ipmitool-1.8.18/bin/ipmitool
> >>>>>>>>>>> sel clear
> >>>>>>>>>>>
> >>>>>>>>>>> So it’s not atomic but it runs after the boot and the elist
> >>>>>>>>>>> should output it properly … at least it did in the past. ;)
> >>>>>>>>>>>
> >>>>>>>>>>> As I said - I’m happy to run any patches you have. If you point
> >>>>>>>>>>> me to a git branch somewhere I can switch that system easily.
> >>>>>>>>>>
> >>>>>>>>>> Ok, I have a branch at
> >>>>>>>>>>
> >>>>>>>>>> https://github.com/cminyard/linux-ipmi.git:debug-panic-oem-events
> >>>>>>>>>>
> >>>>>>>>>> that has debug tracing. It will print the function for all panic
> >>>>>>>>>> event
> >>>>>>>>>> handlers, their return values, and adds traces in the IPMI panic
> >>>>>>>>>> event
> >>>>>>>>>> handlers.
> >>>>>>>>>>
> >>>>>>>>>> It's a single patch right on top of 6.2; I'm not sure how portable
> >>>>>>>>>> it is
> >>>>>>>>>> to other kernel versions. I can port if you like.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> -corey
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>> Christian
> >>>>>>>>>>>
> >>>>>>>>>>>>> On 13. Mar 2023, at 13:58, Corey Minyard <miny...@acm.org>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Mar 13, 2023 at 10:27:51AM +0100, Christian Theune wrote:
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> alright, so here’s the output from the NixOS machine:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> root@xxx ~ # echo c >/proc/sysrq-trigger
> >>>>>>>>>>>>> client_loop: send disconnect: Broken pipe
> >>>>>>>>>>>>> …
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> root@xxx ~ # journalctl -u ipmi-log.service
> >>>>>>>>>>>>> -- Journal begins at Sun 2023-02-26 14:25:36 CET, ends at Mon
> >>>>>>>>>>>>> 2023-03-13 10:25:27 CET. --
> >>>>>>>>>>>>> Mar 13 10:12:38 xxx ipmi-log-start[520973]: Clearing SEL.
> >>>>>>>>>>>>> Please allow a few seconds to erase.
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>> -- Boot fdef496e784e4541abd9ae40df472a0b --
> >>>>>>>>>>>>> Mar 13 10:25:07 xxx ipmi-log-start[1973]: 1 | 03/13/2023 |
> >>>>>>>>>>>>> 09:12:49 | Event Logging Disabled SEL | Log area reset/cleared
> >>>>>>>>>>>>> | Asserted
> >>>>>>>>>>>>> Mar 13 10:25:07 xxx ipmi-log-start[1973]: 2 | 03/13/2023 |
> >>>>>>>>>>>>> 09:21:06 | Watchdog2 OS Watchdog | Hard reset | Asserted
> >>>>>>>>>>>>> Mar 13 10:25:07 xxx ipmi-log-start[1977]: Clearing SEL. Please
> >>>>>>>>>>>>> allow a few seconds to erase.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hmm, the SEL got cleared. That would clear out any of the logs
> >>>>>>>>>>>> that
> >>>>>>>>>>>> were issued before that time. I'm not sure when the above
> >>>>>>>>>>>> happened
> >>>>>>>>>>>> verses the crash, though. It looks like it occurred as part of
> >>>>>>>>>>>> the
> >>>>>>>>>>>> reboot, but I'm not sure what I'm seeing. Maybe you have a
> >>>>>>>>>>>> startup
> >>>>>>>>>>>> process that clears the SEL?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Assuming that's not the issue, what you have looks ok. I'd need
> >>>>>>>>>>>> to add
> >>>>>>>>>>>> some logs to the kernel to see if the log operation ever happens.
> >>>>>>>>>>>>
> >>>>>>>>>>>> -corey
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The SOL log looks like this:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [1107585.917689] sysrq: Trigger a crash
> >>>>>>>>>>>>> [1107585.921272] Kernel panic - not syncing: sysrq triggered
> >>>>>>>>>>>>> crash
> >>>>>>>>>>>>> [1107585.927178] CPU: 1 PID: 521033 Comm: bash Tainted: G
> >>>>>>>>>>>>> I 5.10.159 #1-NixOS
> >>>>>>>>>>>>> [1107585.935335] Hardware name: Dell Inc. PowerEdge
> >>>>>>>>>>>>> R510/00HDP0, BIOS 1.11.0 07/23/2012
> >>>>>>>>>>>>> [1107585.943058] Call Trace:
> >>>>>>>>>>>>> [1107585.945680] dump_stack+0x6b/0x83
> >>>>>>>>>>>>> [1107585.949158] panic+0x101/0x2c8
> >>>>>>>>>>>>> [1107585.952379] ? printk+0x58/0x73
> >>>>>>>>>>>>> [1107585.955687] sysrq_handle_crash+0x16/0x20
> >>>>>>>>>>>>> [1107585.959859] __handle_sysrq.cold+0x43/0x11a
> >>>>>>>>>>>>> [1107585.964203] write_sysrq_trigger+0x24/0x40
> >>>>>>>>>>>>> [1107585.968463] proc_reg_write+0x51/0x90
> >>>>>>>>>>>>> [1107585.972290] vfs_write+0xc3/0x280
> >>>>>>>>>>>>> [1107585.975768] ksys_write+0x5f/0xe0
> >>>>>>>>>>>>> [1107585.979248] do_syscall_64+0x33/0x40
> >>>>>>>>>>>>> [1107585.982987] entry_SYSCALL_64_after_hwframe+0x61/0xc6
> >>>>>>>>>>>>> [1107585.988199] RIP: 0033:0x7f5873932133
> >>>>>>>>>>>>> [1107585.991938] Code: 0c 00 f7 d8 64 89 02 48 c7 c0 ff ff ff
> >>>>>>>>>>>>> ff eb b3 0f 1f 80 00 00 00 00 64 8b 04 25 18 00 00 00 85 c0 75
> >>>>>>>>>>>>> 14 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 55 c3 0f 1f 40
> >>>>>>>>>>>>> 00 41 54 49 89 d4 55 48 89 f5
> >>>>>>>>>>>>> [1107586.010842] RSP: 002b:00007ffcc13808c8 EFLAGS: 00000246
> >>>>>>>>>>>>> ORIG_RAX: 0000000000000001
> >>>>>>>>>>>>> [1107586.018566] RAX: ffffffffffffffda RBX: 0000000000000002
> >>>>>>>>>>>>> RCX: 00007f5873932133
> >>>>>>>>>>>>> [1107586.025923] RDX: 0000000000000002 RSI: 00000000005c1c08
> >>>>>>>>>>>>> RDI: 0000000000000001
> >>>>>>>>>>>>> [1107586.033213] RBP: 00000000005c1c08 R08: 000000000000000a
> >>>>>>>>>>>>> R09: 00007f58739c2f40
> >>>>>>>>>>>>> [1107586.040504] R10: 00000000005cc348 R11: 0000000000000246
> >>>>>>>>>>>>> R12: 0000000000000002
> >>>>>>>>>>>>> [1107586.047794] R13: 00007f5873a00520 R14: 00007f5873a00720
> >>>>>>>>>>>>> R15: 0000000000000002
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Nothing obvious to me here … if you have any further ideas what
> >>>>>>>>>>>>> to test, let me know. I should be more responsive again now.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks and kind regards,
> >>>>>>>>>>>>> Christian
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 5. Mar 2023, at 23:53, Corey Minyard <miny...@acm.org>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Mar 01, 2023 at 06:00:07PM +0100, Christian Theune
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> I’m going to actually attach a serial console to watch the
> >>>>>>>>>>>>>>> “echo c” panic, maybe that gives _some_ indication.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Otherwise: I can quickly run patches on the kernel there to
> >>>>>>>>>>>>>>> try out things. (And the funding offer still stands.)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Any news on this? I'm curious what this could be.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -corey
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Christian
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 1. Mar 2023, at 17:58, Corey Minyard <miny...@acm.org>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Tue, Feb 28, 2023 at 06:36:17PM +0100, Christian Theune
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>> Thanks, both machines report:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> # cat /sys/module/ipmi_msghandler/parameters/panic_op
> >>>>>>>>>>>>>>>>> string
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> At this point, I have no idea. I'd have to start adding
> >>>>>>>>>>>>>>>> printks into
> >>>>>>>>>>>>>>>> the code and cause crashes to see what is happing.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Maybe something is getting in the way of the panic notifiers
> >>>>>>>>>>>>>>>> and doing
> >>>>>>>>>>>>>>>> something to prevent the IPMI driver from working.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -corey
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On 28. Feb 2023, at 18:04, Corey Minyard <miny...@acm.org>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Oh, I forgot. You can look at panic_op in
> >>>>>>>>>>>>>>>>>> /sys/module/ipmi_msghandler/parameters/panic_op
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> -corey
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Feb 28, 2023 at 05:48:07PM +0100, Christian Theune
> >>>>>>>>>>>>>>>>>> via Openipmi-developer wrote:
> >>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On 28. Feb 2023, at 17:36, Corey Minyard
> >>>>>>>>>>>>>>>>>>>> <miny...@acm.org> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Tue, Feb 28, 2023 at 02:53:12PM +0100, Christian
> >>>>>>>>>>>>>>>>>>>> Theune via Openipmi-developer wrote:
> >>>>>>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I’ve been trying to debug the PANIC and OEM string
> >>>>>>>>>>>>>>>>>>>>> handling and am running out of ideas whether this is a
> >>>>>>>>>>>>>>>>>>>>> bug or whether something so subtle has changed in my
> >>>>>>>>>>>>>>>>>>>>> config that I’m just not seeing it.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> (Note: I’m willing to pay for consulting.)
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Probably not necessary.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks! The offer always stands. If we should ever meet
> >>>>>>>>>>>>>>>>>>> I’m also able to pay in beverages. ;)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I have machines that we’ve moved from an older setup
> >>>>>>>>>>>>>>>>>>>>> (Gentoo, (mostly) vanilla kernel 4.19.157) to a newer
> >>>>>>>>>>>>>>>>>>>>> setup (NixOS, (mostly) vanilla kernel 5.10.159) and I’m
> >>>>>>>>>>>>>>>>>>>>> now experiencing crashes that seem to be kernel panics
> >>>>>>>>>>>>>>>>>>>>> but do not get the usual messages in the IPMI SEL.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> I just tested on stock 5.10.159 and it worked without
> >>>>>>>>>>>>>>>>>>>> issue. Everything
> >>>>>>>>>>>>>>>>>>>> you have below looks ok.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Can you test by causing a crash with:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> echo c >/proc/sysrq-trigger
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> and see if it works?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yeah, already tried that and unfortunately that _doesn’t_
> >>>>>>>>>>>>>>>>>>> work.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> It sounds like you are having some type of crash that
> >>>>>>>>>>>>>>>>>>>> you would normally
> >>>>>>>>>>>>>>>>>>>> use the IPMI logs to debug. However, they aren't
> >>>>>>>>>>>>>>>>>>>> perfect, the system
> >>>>>>>>>>>>>>>>>>>> has to stay up long enough to get them into the event
> >>>>>>>>>>>>>>>>>>>> log.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I think they are staying up long enough because a panic
> >>>>>>>>>>>>>>>>>>> triggers the 255 second bump in the watchdog and only
> >>>>>>>>>>>>>>>>>>> then pass on. However, i’ve also noticed that the kernel
> >>>>>>>>>>>>>>>>>>> _should_ be rebooting after a panic much faster (and not
> >>>>>>>>>>>>>>>>>>> rely on the watchdog) and that doesn’t happen either.
> >>>>>>>>>>>>>>>>>>> (Sorry this just popped from the back of my head).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> In this situation, getting a serial console (probably
> >>>>>>>>>>>>>>>>>>>> through IPMI
> >>>>>>>>>>>>>>>>>>>> Serial over LAN) and getting the console output on a
> >>>>>>>>>>>>>>>>>>>> crash is probably
> >>>>>>>>>>>>>>>>>>>> your best option. You can use ipmitool for this, or I
> >>>>>>>>>>>>>>>>>>>> have a library
> >>>>>>>>>>>>>>>>>>>> that is able to make connections to serial ports,
> >>>>>>>>>>>>>>>>>>>> including through IPMI
> >>>>>>>>>>>>>>>>>>>> SoL.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yup. Been there, too. :)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Unfortunately we’re currently chasing something that pops
> >>>>>>>>>>>>>>>>>>> up very randomly on somewhat odd machines and I also have
> >>>>>>>>>>>>>>>>>>> the feeling that it’s systematically broken right now (as
> >>>>>>>>>>>>>>>>>>> the “echo c” doesn’t work).
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks a lot,
> >>>>>>>>>>>>>>>>>>> Christian
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> >>>>>>>>>>>>>>>>>>> Flying Circus Internet Operations GmbH ·
> >>>>>>>>>>>>>>>>>>> https://flyingcircus.io
> >>>>>>>>>>>>>>>>>>> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> >>>>>>>>>>>>>>>>>>> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune,
> >>>>>>>>>>>>>>>>>>> Christian Zagrodnick
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>>>>>> Openipmi-developer mailing list
> >>>>>>>>>>>>>>>>>>> Openipmi-developer@lists.sourceforge.net
> >>>>>>>>>>>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/openipmi-developer
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Liebe Grüße,
> >>>>>>>>>>>>>>>>> Christian Theune
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> >>>>>>>>>>>>>>>>> Flying Circus Internet Operations GmbH ·
> >>>>>>>>>>>>>>>>> https://flyingcircus.io
> >>>>>>>>>>>>>>>>> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> >>>>>>>>>>>>>>>>> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune,
> >>>>>>>>>>>>>>>>> Christian Zagrodnick
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Liebe Grüße,
> >>>>>>>>>>>>>>> Christian Theune
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> >>>>>>>>>>>>>>> Flying Circus Internet Operations GmbH ·
> >>>>>>>>>>>>>>> https://flyingcircus.io
> >>>>>>>>>>>>>>> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> >>>>>>>>>>>>>>> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune,
> >>>>>>>>>>>>>>> Christian Zagrodnick
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Liebe Grüße,
> >>>>>>>>>>>>> Christian Theune
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> >>>>>>>>>>>>> Flying Circus Internet Operations GmbH · https://flyingcircus.io
> >>>>>>>>>>>>> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> >>>>>>>>>>>>> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune,
> >>>>>>>>>>>>> Christian Zagrodnick
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Liebe Grüße,
> >>>>>>>>>>> Christian Theune
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> >>>>>>>>>>> Flying Circus Internet Operations GmbH · https://flyingcircus.io
> >>>>>>>>>>> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> >>>>>>>>>>> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune,
> >>>>>>>>>>> Christian Zagrodnick
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>>> Liebe Grüße,
> >>>>>>>> Christian Theune
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> >>>>>>>> Flying Circus Internet Operations GmbH · https://flyingcircus.io
> >>>>>>>> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> >>>>>>>> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian
> >>>>>>>> Zagrodnick
> >>>>>>>>
> >>>>>>
> >>>>>> Liebe Grüße,
> >>>>>> Christian Theune
> >>>>>>
> >>>>>> --
> >>>>>> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> >>>>>> Flying Circus Internet Operations GmbH · https://flyingcircus.io
> >>>>>> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> >>>>>> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian
> >>>>>> Zagrodnick
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> Openipmi-developer mailing list
> >>>>>> Openipmi-developer@lists.sourceforge.net
> >>>>>> https://lists.sourceforge.net/lists/listinfo/openipmi-developer
> >>>>
> >>>> Liebe Grüße,
> >>>> Christian Theune
> >>>>
> >>>> --
> >>>> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> >>>> Flying Circus Internet Operations GmbH · https://flyingcircus.io
> >>>> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> >>>> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian
> >>>> Zagrodnick
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Openipmi-developer mailing list
> >>>> Openipmi-developer@lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/openipmi-developer
> >>> <0001-ipmi-watchdog-Set-panic-count-to-proper-value-on-a-p.patch>
> >>
> >> Liebe Grüße,
> >> Christian Theune
> >>
> >> --
> >> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> >> Flying Circus Internet Operations GmbH · https://flyingcircus.io
> >> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> >> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian
> >> Zagrodnick
> >>
> >
> >
> >> _______________________________________________
> >> Openipmi-developer mailing list
> >> Openipmi-developer@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/openipmi-developer
> >
> > <0001-ipmi-watchdog-replace-atomic_add-and-atomic_sub.patch>
>
> Liebe Grüße,
> Christian Theune
>
> --
> Christian Theune · c...@flyingcircus.io · +49 345 219401 0
> Flying Circus Internet Operations GmbH · https://flyingcircus.io
> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
>
>
>
> _______________________________________________
> Openipmi-developer mailing list
> Openipmi-developer@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/openipmi-developer
_______________________________________________
Openipmi-developer mailing list
Openipmi-developer@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/openipmi-developer
