Re: [systemd-devel] Detecting Systemd crash

[Mantas Mikulėnas]

On Mon, Feb 5, 2024, 14:54 Lennart Poettering 
wrote:

> On So, 04.02.24 00:06, David Timber (d...@dev.snart.me) wrote:
>
> > 2: How do I get Systemd to freeze to test such program? I mean, if I kill
> > Systemd, the kernel would crash so I have to somehow tell Systemd to
> freeze?
>
> Not really, the kernel blocks SIGSTOP for PID1.
>

Attaching gdb to pid1 should do the job.

Re: [systemd-devel] Detecting Systemd crash

[Lennart Poettering]

On Sa, 03.02.24 16:55, Álvaro Cebrián Juan (acebrianj...@gmail.com) wrote:

> Great question!
>
> I am very interested in detecting systemd crashes too since I have
> experienced them recently and have been asked to come up with a solution to
> react when a PID1 crash happens.
> In fact, in my recent experiences, a journald crash was enough to render
> the system into an unreliable/degraded state in which some top-level
> applications worked while others didn't.
>
> So adding to David's 1st question, I need to detect systemd and journald
> crashes and then trigger a `systemctl reboot --force --force`
> command

As mentioned elsewhere in this thread just use RuntimeWatchdogSec= in
systemd-system.conf(5)

Lennart

--
Lennart Poettering, Berlin

Re: [systemd-devel] Detecting Systemd crash

[Lennart Poettering]

On Mo, 05.02.24 13:54, Lennart Poettering (lenn...@poettering.net) wrote:

> you can just use the usual hw watchdog. If pid1 dies it will not ping
> the hw watchdog, and thus a reset is triggered automatically. In fact
> we actually configure the hw watchdog by default these days on hw that
> has it (which are most PCs).

Actually, we don't really, I need to correct myself. We probably
should though, dunno.

See RuntimeWatchdogSec= in systemd-system.conf(5)

>
> > 2: How do I get Systemd to freeze to test such program? I mean, if I kill
> > Systemd, the kernel would crash so I have to somehow tell Systemd to freeze?
>
> Not really, the kernel blocks SIGSTOP for PID1.
>
> Lennart
>
> --
> Lennart Poettering, Berlin

Lennart

--
Lennart Poettering, Berlin

Re: [systemd-devel] Detecting Systemd crash

[Lennart Poettering]

On So, 04.02.24 00:06, David Timber (d...@dev.snart.me) wrote:

> Systemd crashed on me the other day. I was writing up some Systemd units and
> testing them out by daemon-reload every time I wanted to test them out. Not
> the best way to go on about, I know. My bad abusing Systemd to the point of
> crashing. Perhaps it was just a bit flip that caused this.
>
>systemd[2368]: Assertion 'path_is_absolute(p)' failed at
>src/basic/chase.c:628, function chase(). Aborting.
>systemd[1]: Assertion 'path_is_absolute(p)' failed at
>src/basic/chase.c:628, function chase(). Aborting.
>systemd[1]: Caught  from our own process.
>systemd-coredump[32497]: Due to PID 1 having crashed coredump
>collection will now be turned off.
>systemd-coredump[32497]: [] Process 32496 (systemd) of user 0
>dumped core.
>systemd[1]: Caught , dumped core as pid 32496.
>systemd[1]: Freezing execution.
>
>...
>
>systemd-journald[871]: Failed to send stream file descriptor to
>service manager: Transport endpoint is not connected
>
> I didn't even bother trying producing stack trace. I can get on that if
> anyone wants it. My machine started doing some weird things like
> Firefox not

If this is a current systemd version (v255), please generate a stack trace
and submit it as github issue to us, we'll look into it. If it's
older, please report to your distro first.

> being able to do Ajax properly whilst being able to go to a new page,
> Chromium not being able to create a new tab whilst all the text editors
> worked just fine, all the systemctl commands timing out. So basically, I was
> using Linux without fork(). Anyway.
> Well, I think any software can crash for any reason whatsoever. The
> problem

Yeah, an assert like the above is an error we need to fix in systemd.

> with Systemd I realised from this incident is that I had no way of knowing
> that Systemd had crashed until I opened up the journal and kernel logs and
> saw that Systemd had crashed some time ago. In this particular incident,
> Systemd caught the signal and decided to just freeze. No idea why you'd want
> that because if it had just crashed, the kernel would have just panicked and
> I would have realised something went wrong.
>
> 1: So I decided that I need a some sort of "watchdog" that warns me when
> something like this happens. Using dbus to poll the status of the Systemd
> process, it could be a GUI app running under a seat, just a daemon that
> writes a warning message using `wall` or just send mail using a primed up
> MUA process. I wonder if someone already had the same idea and went on to
> make one.

you can just use the usual hw watchdog. If pid1 dies it will not ping
the hw watchdog, and thus a reset is triggered automatically. In fact
we actually configure the hw watchdog by default these days on hw that
has it (which are most PCs).

> 2: How do I get Systemd to freeze to test such program? I mean, if I kill
> Systemd, the kernel would crash so I have to somehow tell Systemd to freeze?

Not really, the kernel blocks SIGSTOP for PID1.

Lennart

--
Lennart Poettering, Berlin

Re: [systemd-devel] Detecting Systemd crash

[František Šumšal]

On 2/3/24 16:06, David Timber wrote:

Systemd crashed on me the other day. I was writing up some Systemd units and 
testing them out by daemon-reload every time I wanted to test them out. Not the 
best way to go on about, I know. My bad abusing Systemd to the point of 
crashing. Perhaps it was just a bit flip that caused this.

    systemd[2368]: Assertion 'path_is_absolute(p)' failed at
    src/basic/chase.c:628, function chase(). Aborting.
    systemd[1]: Assertion 'path_is_absolute(p)' failed at
    src/basic/chase.c:628, function chase(). Aborting.
    systemd[1]: Caught  from our own process.
    systemd-coredump[32497]: Due to PID 1 having crashed coredump
    collection will now be turned off.
    systemd-coredump[32497]: [] Process 32496 (systemd) of user 0
    dumped core.
    systemd[1]: Caught , dumped core as pid 32496.
    systemd[1]: Freezing execution.

    ...

    systemd-journald[871]: Failed to send stream file descriptor to
    service manager: Transport endpoint is not connected

I didn't even bother trying producing stack trace. I can get on that if anyone 
wants it.


What you did was perfectly reasonable, systemd shouldn't just crash in that 
case. If you run the recent-ish systemd, a stack trace would be very welcome.


My machine started doing some weird things like Firefox not being able to do 
Ajax properly whilst being able to go to a new page, Chromium not being able to 
create a new tab whilst all the text editors worked just fine, all the 
systemctl commands timing out. So basically, I was using Linux without fork(). 
Anyway.
Well, I think any software can crash for any reason whatsoever. The problem 
with Systemd I realised from this incident is that I had no way of knowing that 
Systemd had crashed until I opened up the journal and kernel logs and saw that 
Systemd had crashed some time ago. In this particular incident, Systemd caught 
the signal and decided to just freeze. No idea why you'd want that because if 
it had just crashed, the kernel would have just panicked and I would have 
realised something went wrong.

1: So I decided that I need a some sort of "watchdog" that warns me when 
something like this happens. Using dbus to poll the status of the Systemd process, it 
could be a GUI app running under a seat, just a daemon that writes a warning message 
using `wall` or just send mail using a primed up MUA process. I wonder if someone already 
had the same idea and went on to make one.

2: How do I get Systemd to freeze to test such program? I mean, if I kill 
Systemd, the kernel would crash so I have to somehow tell Systemd to freeze?


Just trigger systemd's crash handler by sending it a SIGSEGV (kill -SEGV 1).

Re: [systemd-devel] Detecting Systemd crash

[František Šumšal]

On 2/3/24 16:55, Álvaro Cebrián Juan wrote:

Great question!

I am very interested in detecting systemd crashes too since I have experienced 
them recently and have been asked to come up with a solution to react when a 
PID1 crash happens.
In fact, in my recent experiences, a journald crash was enough to render the 
system into an unreliable/degraded state in which some top-level applications 
worked while others didn't.

So adding to David's 1st question, I need to detect systemd and journald 
crashes and then trigger a `systemctl reboot --force --force` command


You can tell systemd to do just that, by setting CrashReboot=yes in system.conf 
[0][1]. It defaults to 'no' to avoid reboot loops.

[0] 
https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html#LogColor=
[1] 
https://www.freedesktop.org/software/systemd/man/latest/systemd.html#systemd.crash_reboot



I have also read that Linux Magic System Request Key (SysRq) can help in such 
scenarios but I don't know how they work.

Any help would be very appreciated.
Thank you.

Some related links:
https://news.ycombinator.com/item?id=19023695 

https://news.ycombinator.com/item?id=36873927 

https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html 



El sáb, 3 feb 2024 a las 16:14, David Timber (mailto:d...@dev.snart.me>>) escribió:

Systemd crashed on me the other day. I was writing up some Systemd units
and testing them out by daemon-reload every time I wanted to test them
out. Not the best way to go on about, I know. My bad abusing Systemd to
the point of crashing. Perhaps it was just a bit flip that caused this.

     systemd[2368]: Assertion 'path_is_absolute(p)' failed at
     src/basic/chase.c:628, function chase(). Aborting.
     systemd[1]: Assertion 'path_is_absolute(p)' failed at
     src/basic/chase.c:628, function chase(). Aborting.
     systemd[1]: Caught  from our own process.
     systemd-coredump[32497]: Due to PID 1 having crashed coredump
     collection will now be turned off.
     systemd-coredump[32497]: [] Process 32496 (systemd) of user 0
     dumped core.
     systemd[1]: Caught , dumped core as pid 32496.
     systemd[1]: Freezing execution.

     ...

     systemd-journald[871]: Failed to send stream file descriptor to
     service manager: Transport endpoint is not connected

I didn't even bother trying producing stack trace. I can get on that if
anyone wants it. My machine started doing some weird things like Firefox
not being able to do Ajax properly whilst being able to go to a new
page, Chromium not being able to create a new tab whilst all the text
editors worked just fine, all the systemctl commands timing out. So
basically, I was using Linux without fork(). Anyway.
Well, I think any software can crash for any reason whatsoever. The
problem with Systemd I realised from this incident is that I had no way
of knowing that Systemd had crashed until I opened up the journal and
kernel logs and saw that Systemd had crashed some time ago. In this
particular incident, Systemd caught the signal and decided to just
freeze. No idea why you'd want that because if it had just crashed, the
kernel would have just panicked and I would have realised something went
wrong.

1: So I decided that I need a some sort of "watchdog" that warns me when
something like this happens. Using dbus to poll the status of the
Systemd process, it could be a GUI app running under a seat, just a
daemon that writes a warning message using `wall` or just send mail
using a primed up MUA process. I wonder if someone already had the same
idea and went on to make one.

2: How do I get Systemd to freeze to test such program? I mean, if I
kill Systemd, the kernel would crash so I have to somehow tell Systemd
to freeze?

Re: [systemd-devel] Detecting Systemd crash

[Álvaro Cebrián Juan]

Great question!

I am very interested in detecting systemd crashes too since I have
experienced them recently and have been asked to come up with a solution to
react when a PID1 crash happens.
In fact, in my recent experiences, a journald crash was enough to render
the system into an unreliable/degraded state in which some top-level
applications worked while others didn't.

So adding to David's 1st question, I need to detect systemd and journald
crashes and then trigger a `systemctl reboot --force --force` command

I have also read that Linux Magic System Request Key (SysRq) can help in
such scenarios but I don't know how they work.

Any help would be very appreciated.
Thank you.

Some related links:
https://news.ycombinator.com/item?id=19023695
https://news.ycombinator.com/item?id=36873927
https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html


El sáb, 3 feb 2024 a las 16:14, David Timber () escribió:

> Systemd crashed on me the other day. I was writing up some Systemd units
> and testing them out by daemon-reload every time I wanted to test them
> out. Not the best way to go on about, I know. My bad abusing Systemd to
> the point of crashing. Perhaps it was just a bit flip that caused this.
>
> systemd[2368]: Assertion 'path_is_absolute(p)' failed at
> src/basic/chase.c:628, function chase(). Aborting.
> systemd[1]: Assertion 'path_is_absolute(p)' failed at
> src/basic/chase.c:628, function chase(). Aborting.
> systemd[1]: Caught  from our own process.
> systemd-coredump[32497]: Due to PID 1 having crashed coredump
> collection will now be turned off.
> systemd-coredump[32497]: [] Process 32496 (systemd) of user 0
> dumped core.
> systemd[1]: Caught , dumped core as pid 32496.
> systemd[1]: Freezing execution.
>
> ...
>
> systemd-journald[871]: Failed to send stream file descriptor to
> service manager: Transport endpoint is not connected
>
> I didn't even bother trying producing stack trace. I can get on that if
> anyone wants it. My machine started doing some weird things like Firefox
> not being able to do Ajax properly whilst being able to go to a new
> page, Chromium not being able to create a new tab whilst all the text
> editors worked just fine, all the systemctl commands timing out. So
> basically, I was using Linux without fork(). Anyway.
> Well, I think any software can crash for any reason whatsoever. The
> problem with Systemd I realised from this incident is that I had no way
> of knowing that Systemd had crashed until I opened up the journal and
> kernel logs and saw that Systemd had crashed some time ago. In this
> particular incident, Systemd caught the signal and decided to just
> freeze. No idea why you'd want that because if it had just crashed, the
> kernel would have just panicked and I would have realised something went
> wrong.
>
> 1: So I decided that I need a some sort of "watchdog" that warns me when
> something like this happens. Using dbus to poll the status of the
> Systemd process, it could be a GUI app running under a seat, just a
> daemon that writes a warning message using `wall` or just send mail
> using a primed up MUA process. I wonder if someone already had the same
> idea and went on to make one.
>
> 2: How do I get Systemd to freeze to test such program? I mean, if I
> kill Systemd, the kernel would crash so I have to somehow tell Systemd
> to freeze?
>
>

[systemd-devel] Detecting Systemd crash

[David Timber]

Systemd crashed on me the other day. I was writing up some Systemd units 
and testing them out by daemon-reload every time I wanted to test them 
out. Not the best way to go on about, I know. My bad abusing Systemd to 
the point of crashing. Perhaps it was just a bit flip that caused this.


   systemd[2368]: Assertion 'path_is_absolute(p)' failed at
   src/basic/chase.c:628, function chase(). Aborting.
   systemd[1]: Assertion 'path_is_absolute(p)' failed at
   src/basic/chase.c:628, function chase(). Aborting.
   systemd[1]: Caught  from our own process.
   systemd-coredump[32497]: Due to PID 1 having crashed coredump
   collection will now be turned off.
   systemd-coredump[32497]: [] Process 32496 (systemd) of user 0
   dumped core.
   systemd[1]: Caught , dumped core as pid 32496.
   systemd[1]: Freezing execution.

   ...

   systemd-journald[871]: Failed to send stream file descriptor to
   service manager: Transport endpoint is not connected

I didn't even bother trying producing stack trace. I can get on that if 
anyone wants it. My machine started doing some weird things like Firefox 
not being able to do Ajax properly whilst being able to go to a new 
page, Chromium not being able to create a new tab whilst all the text 
editors worked just fine, all the systemctl commands timing out. So 
basically, I was using Linux without fork(). Anyway.
Well, I think any software can crash for any reason whatsoever. The 
problem with Systemd I realised from this incident is that I had no way 
of knowing that Systemd had crashed until I opened up the journal and 
kernel logs and saw that Systemd had crashed some time ago. In this 
particular incident, Systemd caught the signal and decided to just 
freeze. No idea why you'd want that because if it had just crashed, the 
kernel would have just panicked and I would have realised something went 
wrong.


1: So I decided that I need a some sort of "watchdog" that warns me when 
something like this happens. Using dbus to poll the status of the 
Systemd process, it could be a GUI app running under a seat, just a 
daemon that writes a warning message using `wall` or just send mail 
using a primed up MUA process. I wonder if someone already had the same 
idea and went on to make one.


2: How do I get Systemd to freeze to test such program? I mean, if I 
kill Systemd, the kernel would crash so I have to somehow tell Systemd 
to freeze?