Hi,
> On 1. Oct 2023, at 03:49, Corey Minyard <[email protected]> wrote:
>
> On Sat, Sep 30, 2023 at 11:14:01PM +0200, Christian Theune via Openipmi-developer wrote:
>> Hi,
>>
>> sorry if this isn’t directly a developers question, but I’ve run out of avenues after googling and looking around…
>>
>> We’re experiencing weird system stability issue where the “log to SEL” doesn’t cut it: we see watchdog reboots but no kernel output whatsoever ending up in the SEL. (I’ve debugged this with Corey before and we found something to fix but the watchdog events we’re experiencing still don’t get logged in more detail.)
>
> Can you not get kernel coredumps?Unfortunately no and I still have absolutely now idea why the watchdog triggers…
I have currently attached dozens of servers that are part of a mysterious series of crashes but they didn’t crash after I attached the SOL continuously. Just my kind of luck I guess … ;)
As we’re continuously updating our environment it might also be that we’ve successfully evaded a kernel bug that was haunting us … maybe … ;)
>>
>> I’m wondering: does anyone know of a “push” solution to instruct the BMC (mostly Supermicro in our case) to push SOL output proactively through some protocol like syslog?
>
> The SEL probably isn't big or fast enough to take system logs. You
> could create something like this as part of printk, but I suspect that
> it would quickly overflow the SEL.Yeah, I wasn’t thinking about the SEL but wondering whether serial output could be shipped in a push-manner from the BMC without having to attach and authenticate.
>> Otherwise we’d need to set up a central host with passwords for dozens of hosts to pull the SOL for logging and that doesn’t feel right either … -__
>
> I know people that do this; it's not terrible. You do have all of your
> IPMI passwords in one place, that's the biggest issue, but IMHO you
> should be monitoring the output of your consoles, anyway.Yeah, that’s what I’m pondering, too. IMHO it’s quite a bit terrible and thus I was wondering whether the BMC might have a built-in solution that would turn this upside down … but I gess not
> I support a program called ser2net that is capable of making SOL
> connections, logging the output, and allowing connections to the
> console. That would be a pretty complicated setup, but I can help you
> with it, if you like.The multiplexing sounds great. I’ve built a small shell wrapper to manage SOL connections and their logging (and reconnecting if the BMC acts up) which works for now.
From a design perspective I’d really love this to be push-based. I researched the dmtf site, but didn’t find anything there either … I guess I’m the odd-one out then …
Christian
--
Christian Theune · [email protected] · +49 345 219401 0
Flying Circus Internet Operations GmbH · https://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick_______________________________________________
Openipmi-developer mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/openipmi-developer
_______________________________________________ Openipmi-developer mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/openipmi-developer
