Re: [systemd-devel] switch-root, init, SIGHUP
> > I think the SIGHUP must come from the new PID 1. > > I don't know if this will help you, however here is a program that I > wrote last year that starts a service in the initramfs, it survives > the systemd killing spree during the transition to the root > filesystem, a new copy of the program is started from the root > filesystem, the previous running state is passed from the initramfs to > the version started from the root filesystem, and the version started > from the initramfs is gracefully shut down. The program does not have > the SIGHUP issue that you describe, so maybe there's something here > that may help you. > > https://github.com/masneyb/early-service-example Thanks very much for this pointer! I will take a closer look at your code since I would indeed prefer to start my process (and systemd-bootchart) in the initrd. I had tried doing that using "rdinit=" and did observe that the SIGHUP problem does not occur in that case (btw, it is SIGHUP followed by SIGCONT). For systemd-bootchart, after it does some initial set-up, it only reads files under /proc (and it keeps a directory stream to /proc open). Unfortunately, the switch-root operation somehow causes /proc/schedstat to become inaccessible, so systemd-bootchart is broken with "rdinit=" as well. -James M
Re: [systemd-devel] switch-root, init, SIGHUP
Hey James, On Mon, Jan 20, 2025 at 1:21 PM James Muir (jamesmui) wrote: > I think the SIGHUP must come from the new PID 1. I don't know if this will help you, however here is a program that I wrote last year that starts a service in the initramfs, it survives the systemd killing spree during the transition to the root filesystem, a new copy of the program is started from the root filesystem, the previous running state is passed from the initramfs to the version started from the root filesystem, and the version started from the initramfs is gracefully shut down. The program does not have the SIGHUP issue that you describe, so maybe there's something here that may help you. https://github.com/masneyb/early-service-example Brian
Re: [systemd-devel] switch-root, init, SIGHUP
> > It is not clear to me if the '@' is necessary because the process is > > invoked using the "init" kernel parameter. > > That detail doesn't matter. It's irrelevant how the process gets > started. In particular as I understand you you fork()ed once, > i.e. init= starts PID 1, but this is not PID 1 we are talking about > here... I did not mean to imply that a process forked from "init=" gets special treatment. I only mentioned "init=" because I think it implies an ordering wrt the switch-root killing spree. > > Doesn't that mean it gets executed after the switch-root killing > > spree? (i.e. it gets executed when PID 1 from the initial ramdisk > > re-executes to run the new init). > > do you actually invoke the systemd binary in the initrd already, and > the ask it to switch root, or how does that work? I am working with a vanilla Fedora 41 image. systemd runs as PID 1 in the initrd. The process I specify using "init=" is invoked as part of the normal switch-root flow (when systemd from the initrd calls execve() ). > > I think the SIGHUP must come from the new PID 1. > > sigaction() actually tells you exactly where a signal comes from, if > you let it via the siginfo_t structure optionally delivered to your > signal handler. hence you can relatively easily figure this out. The > si_pid specifies the source pid. And si_code tells you if this was > sent by the kernel or by userspace, i.e. SI_KERNEL, SI_USER and so on. The strace log I pointed to shows PID 1 sent the signal. However, there is PID 1 from the initrd and the new PID 1 from the real root file system. It must be the new PID 1 that sends the SIGHUP. > > > Note that we'll also possibly reinitialize the tty on switch root, > > > maybe your tool has the tty open and gets a SIGHUP because of that. > > > > Brian M also suggested that might be the cause, but I don't see any > > ttys when I do "ls -l /proc//fd". > > it's not so much about that, but about which ctty your process has. There is no controlling tty listed in the output of "ps". -James M
Re: [systemd-devel] switch-root, init, SIGHUP
On Mo, 20.01.25 18:21, James Muir (jamesmui) ([email protected]) wrote: > > Are you sure you are setting argv[0][0] properly? the killing spree we > > do on switch root should exclude processes marked like that. > > When I check /proc//cmdline, I see the '@' character. > > It is not clear to me if the '@' is necessary because the process is > invoked using the "init" kernel parameter. That detail doesn't matter. It's irrelevant how the process gets started. In particular as I understand you you fork()ed once, i.e. init= starts PID 1, but this is not PID 1 we are talking about here... > Doesn't that mean it gets executed after the switch-root killing > spree? (i.e. it gets executed when PID 1 from the initial ramdisk > re-executes to run the new init). do you actually invoke the systemd binary in the initrd already, and the ask it to switch root, or how does that work? > I think the SIGHUP must come from the new PID 1. sigaction() actually tells you exactly where a signal comes from, if you let it via the siginfo_t structure optionally delivered to your signal handler. hence you can relatively easily figure this out. The si_pid specifies the source pid. And si_code tells you if this was sent by the kernel or by userspace, i.e. SI_KERNEL, SI_USER and so on. > > Note that we'll also possibly reinitialize the tty on switch root, > > maybe your tool has the tty open and gets a SIGHUP because of that. > > Brian M also suggested that might be the cause, but I don't see any > ttys when I do "ls -l /proc//fd". it's not so much about that, but about which ctty your process has. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] switch-root, init, SIGHUP
> > I have a program, init-jm, that forks and executes > > /usr/lib/systemd/systemd in the parent (using execl() ) while the > > child collects some stats in a loop. > > Uh, you can do this, but it's not trivial to get right: you need to > move yourself down the cgroup tree, because otherwise you'll collide > with cgroup's no-processes-in-inner-cgroup-nodes rule. Create two > cgroups in the root of the cgroup tree, then put your stuff in one, > and invoke systemd in the other. That's the only "correct" way to do > this, so that your stuff runs reasonably isolated from systemd. Thanks very much for your reply. I was not aware of this cgroup rule. I will read more about that. > > The child sets its argv[0][0] to '@'. > > > > init-jm is invoked using the "init" kernel parameter on a switch-root > > system (it's Fedora 41). > > > > Can someone explain why systemd (PID 1) sends SIGHUP to the child? > > Are you sure you are setting argv[0][0] properly? the killing spree we > do on switch root should exclude processes marked like that. When I check /proc//cmdline, I see the '@' character. It is not clear to me if the '@' is necessary because the process is invoked using the "init" kernel parameter. Doesn't that mean it gets executed after the switch-root killing spree? (i.e. it gets executed when PID 1 from the initial ramdisk re-executes to run the new init). I think the SIGHUP must come from the new PID 1. > Note that we'll also possibly reinitialize the tty on switch root, > maybe your tool has the tty open and gets a SIGHUP because of that. Brian M also suggested that might be the cause, but I don't see any ttys when I do "ls -l /proc//fd". This pattern (fork and exec /usr/lib/systemd/systemd in the parent) is used by systemd-bootchart. If you invoke systemd-bootchart using the "init" kernel parameter on Fedora 41 (which is the recommended way), then you will hit this SIGHUP (and bootchart exits when it receives SIGHUP, so this breaks it). That appears to have been not working for a while (e.g. it does not work with systemd v251). There is an strace log here: https://github.com/systemd/systemd-bootchart/issues/58 -James M
Re: [systemd-devel] switch-root, init, SIGHUP
On Sa, 18.01.25 05:40, James Muir (jamesmui) ([email protected]) wrote: > I have a program, init-jm, that forks and executes > /usr/lib/systemd/systemd in the parent (using execl() ) while the > child collects some stats in a loop. Uh, you can do this, but it's not trivial to get right: you need to move yourself down the cgroup tree, because otherwise you'll collide with cgroup's no-processes-in-inner-cgroup-nodes rule. Create two cgroups in the root of the cgroup tree, then put your stuff in one, and invoke systemd in the other. That's the only "correct" way to do this, so that your stuff runs reasonably isolated from systemd. > The child sets its argv[0][0] to ‘@’. > > init-jm is invoked using the “init” kernel parameter on a switch-root system > (it’s Fedora 41). > > Can someone explain why systemd (PID 1) sends SIGHUP to the child? Are you sure you are setting argv[0][0] properly? the killing spree we do on switch root should exclude processes marked like that. Note that we'll also possibly reinitialize the tty on switch root, maybe your tool has the tty open and gets a SIGHUP because of that. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] switch-root, init, SIGHUP
> I suspect that your process has a controlling tty based on this comment
> in the killall() function at src/shared/killall.c:
>
>if (send_sighup) {
>/* Optionally, also send a SIGHUP signal, but only if the process
> has a controlling
> * tty. This is useful to allow handling of shells which ignore
> SIGTERM but react to
> * SIGHUP. We do not send this to processes that have no
> controlling TTY since we
> * don't want to trigger reloads of daemon processes. Also we make
> sure to only send
> * this after SIGTERM so that SIGTERM is always first in the
> queue. */
> ...
>}
>
> Also systemd >= v255 (Fedora 41 has v256) has the directive
> SurviveFinalKillSignal=yes that can be used instead of setting
> argv[0][0] = '@'.
Hi Brian. Thanks very much for your reply.
When I grepped for sighup in the systemd source, I did find the function
killall().
The sighup should not be coming from there because my process has argv[0][0] ==
‘@’ and there is a check for that in the killall() for-loop (it is done in the
static function ignore_proc()).
Also, my process does not have a controlling tty (it is being invoked using the
“init” kernel param).
There is no tty shown when I view its details using ps, and I also don’t see
any ttys when I do “ls -l /proc//fd”.
Any other suggestions?
Maybe I should open a github issue for this.
-James M
Re: [systemd-devel] switch-root, init, SIGHUP
Hi James,
On Sat, Jan 18, 2025 at 05:40:17AM +, James Muir (jamesmui) wrote:
> I have a program, init-jm, that forks and executes /usr/lib/systemd/systemd
> in the parent (using execl() ) while the child collects some stats in a loop.
>
> The child sets its argv[0][0] to ‘@’.
>
> init-jm is invoked using the “init” kernel parameter on a switch-root system
> (it’s Fedora 41).
>
> Can someone explain why systemd (PID 1) sends SIGHUP to the child?
>
> This happens just before the statement log_execution_mode(&first_boot) is
> executed inside main().
>
> Sending SIGHUP to the child seems to be particular to the re-execution of
> the systemd binary during switch-root because it is not sent when init-jm is
> invoked using “rdinit”.
I suspect that your process has a controlling tty based on this comment
in the killall() function at src/shared/killall.c:
if (send_sighup) {
/* Optionally, also send a SIGHUP signal, but only if the process
has a controlling
* tty. This is useful to allow handling of shells which ignore
SIGTERM but react to
* SIGHUP. We do not send this to processes that have no
controlling TTY since we
* don't want to trigger reloads of daemon processes. Also we make
sure to only send
* this after SIGTERM so that SIGTERM is always first in the queue.
*/
...
}
Also systemd >= v255 (Fedora 41 has v256) has the directive
SurviveFinalKillSignal=yes that can be used instead of setting
argv[0][0] = '@'.
Brian
