Journey to s6-svscan as PID 1 on FreeBSD (almost there)
On 8/04/2021 11:52 pm, Laurent Bercot wrote:
>> Finally I sucked up the courage and defined in /boot/loader.conf
>> init_exec=/root/bin/init_exec.sh
>> which contains
>> #!/usr/local/bin/execlineb -S0
>> redirfd -wnb 1 /m/fifo/catch_all
>> redirfd -r - /dev/null
>> fdmove -c 2 1
>> exec -c /sbin/s6/svscan -t 0 -s /s/scan
>>
>> And finally /sbin/s6-svscan runs as pid 1.
>
> That's awesome! So, if I understand correctly:
>
> - /sbin/init is still the first program loaded by the kernel, it's
> the stock FreeBSD thing that performs FreeBSD-specific setups
> - if there is an init_exec entry in /boot/loader.conf, /sbin/init
> execs into its value, which allows you to run a different pid 1
> for the lifetime of the machine.
Yes. That is correct.
Sequence: bios -> boot (MBR & GPT differ slightly) -> loader -> kernel
-> /sbin/init
(loader reads and executes /boot/loader.conf this enables things like
setting kernel environment variables, read-only kernel settings,
attaching encrypted devices, some controls over init and human
interaction with loader. My apologies if its obvious that I have no
knowledge of Linux, as I went from IBM OS, CP/M, VMS, FreeBSD)
>
> If that's really how it works, it's incredibly good, and would make
> s6-freebsd-init a breeze to write.
> The necessary modifications to turn s6-linux-init into s6-freebsd-init
> would just be:
>
> - Comment out kbspecials() and its invocations:
> https://github.com/skarnet/s6-linux-init/blob/master/src/init/s6-linux-init.c#L70
>
I really appreciate the direct links. I'd need to set aside time to
understand what s6-linux-init does, though kbspecials looks like its
performs an open console (been more than 30+ years since I worked in C)
> (does FreeBSD's /sbin/init set up CAD handling and other related
> console things?)
Yes. /sbin/init normally owns the console and runs getty (stuff).
So from init.c
init_exec invokes replace_init(path) -> execute_script() ->
open_console() & execv()
while
init_script -> run_script -> forks -> execute_script -> open_console & execv
> - Port the mount() and umount() invocations, replacing mount with the
> correct FreeBSD way of mounting a tmpfs:
> https://github.com/skarnet/s6-linux-init/blob/master/src/init/s6-linux-init.c#L209>
init.c creates and calls "mount_tmpfs" (using nmount) which should be a
cut/paste to mount at S6_LINUX_INIT_TMPFS, the purpose for it in FreeBSD
is to enable a reroot to occur :)
I notice runlevels which FreeBSD doesn't have. I don't think the 5
security modes is the same -1 is default, insecure.
> - Port the final reboot() call:
> https://github.com/skarnet/s6-linux-init/blob/master/src/shutdown/s6-linux-init-hpr.c#L81
>
Yes, I think this is the challenge. I need to figure out why the signals
aren't getting caught (& handled) by s6-svscan. :-/
init.c is 2043 lines of code and a lot of that is in managing ttys and
shutdown. The reboot command should be fairly straight-forward.
Humorous comment re glibc multilib.
>> and all the rest should work as is. It's a game-changer.
>
>
>> foreground { /sbin/shutdown -o -r now } # this invokes reboot rather
>> than send a signal to init.
>>
>> I'd greatly appreciate advice as to completing a clean shutdown?
>
> My current approach is, indeed, to keep s6-svscan running as pid 1,
> without making it exec into anything else, for the whole lifetime of
> the machine, shutdown procedure included. So yes, for a clean
> shutdown, you'd just make sure to kill all processes, then unmount
> filesystems (and remount / read-only), then invoke reboot/poweroff
> directly. That is how the current s6-linux-init works; I don't see a
> reason why it shouldn't work on FreeBSD.
>
This is where I'm stuck. If I can achieve a clean shutdown, then init
is a non-issue. Largely because my servers are headless, so I don't
need a console (or ttys) unless I have a hardware fault, in which case
I'd boot from a clean device (if possible).
I've given some thought to modify s6-svscan to include the shutdown
code, but we're not desperate yet.
>
>> PS This journey commenced with the simple wish to safely rotate apache
>> logs, now I'm hooked on s6-rc (and the peace of mind it brings)
>
> Glad you like it :)
>
> --
> Laurent
Thank-you for your informative reply. I'll schedule some time to work
through this properly and come back.
Re: Journey to s6-svscan as PID 1 on FreeBSD (almost there)
On 9/04/2021 4:29 am, Crest wrote: > There is a cleaner solution. I prefer to set the init_path kernel env > var in /boot/loader.conf and prepend the s6 stage1 script to the normal > search path. That way the kernel starts my execline script as PID 1 and > if I mess it I just have to clear the variable from the loader prompt. Thank-you for the reminder, I'd forgotten about kern.init_path. Though in both cases its really a matter of changing a variable after the loader has completed its chores (For non-FreeBSDers, there are two points where a boot can be interrupted prior to the kernel starting) I use both init_script to establish all the prerequisite stuff, check boot device, enable MAC(biba), enable firewall, start network, start ssh, create logging environment; then handoff to s6-svscan, via init_exec. Yes it could be done in one script which init_path points to, and I agree that would be cleaner. But the prerequisite stuff really needs a sh, and I like execlineb doing its redirection prep and handing off to s6-svscan :)
Re: execlineb ELF executable stack on Linux
Fair enough. That's totally fine for self-contained packages in which you control the build parameters from end-to-end. This may become an issue with skarnet libraries, such as utmps, that are used with other, non-skarnet packages. Libutmps pulls in libskarnet and libskarnet 'pollutes' the other packages by making their stack executable. To make things perfectly clear: the modification I pushed to git also applies to the ld invocations creating shared libraries, including libskarnet.so. So with that change, skalibs or libutmps will not pollute anything else. ;) -- Laurent
Re: execlineb ELF executable stack on Linux
On Thu, Apr 8, 2021 at 9:57 PM Laurent Bercot wrote: > > My toolchain also creates E stack binaries by default, no matter > whether they're static or dynamic. It may be that my build of musl is > bad. That's unexpected. > I am not interested enough in the details of what happens at the ld > level to try and figure out if there's *something* that causes it to > mark E stack when it should not; it requires spending much more > quality time with binutils than I am comfortable with. All I know is > that none of the object files in my software needs E stack, and > bullying ld into doing the right thing works, so I'm content with that > solution. Fair enough. That's totally fine for self-contained packages in which you control the build parameters from end-to-end. This may become an issue with skarnet libraries, such as utmps, that are used with other, non-skarnet packages. Libutmps pulls in libskarnet and libskarnet 'pollutes' the other packages by making their stack executable. I'm currently going through the process of linking some packages (e.g. coreutils, util-linux, and the like) with libutmps, and now the executables that call libutmps have their stack executable. I can deal with it on my own by tweaking the LDFLAGS (-noexecstack) of those packages, it's not too much of an issue for me. But let's take Alpine Linux for example. I see that they now link coreutils, and probably other packages I didn't check, with libutmps and libskarnet. I don't know if they're affected by the same GNU_STACK weirdness in skalibs, or if they scan their distribution binaries for executable stacks, but they may be shipping a bunch of innocuous-looking binaries that have an executable stack and may be vulnerable as a result. So IMHO it may be worthwhile spending some time at some point to figure out what's going on and eventually make it not happen anymore. Just my 2 cents.
Re: execlineb ELF executable stack on Linux
On 08/04/2021 21.57, Laurent Bercot wrote: >> 1) I think I misunderstood how the kernel warning works. It may only >> apply to images that the kernel execve() itself directly rather than >> to any execve() syscall. > > Looks like it. Otherwise, I would have seen zillions of warning messages > in my kernel logs, with all the executable-stack binaries I have > apparently been running. XD It's checked for every execve, but it's a pr_warn_once(), so if (one of the incarnations of) PID1 triggers the warning, nothing else will during that boot. Rasmus
Re: execlineb ELF executable stack on Linux
1) I think I misunderstood how the kernel warning works. It may only apply to images that the kernel execve() itself directly rather than to any execve() syscall. Looks like it. Otherwise, I would have seen zillions of warning messages in my kernel logs, with all the executable-stack binaries I have apparently been running. XD 2) Simply compiling a Hello World program with gcc without any option and linked with musl libc produces an executable with a non-executable stack by default. > (...) So it seems there is something in one of the skalibs that tells the linker to mark the stack as executable. I wish I could tell you what, but that is beyond my expertise. I double-checked all the objects in skalibs, and none of them require an executable stack. But when creating libskarnet.so without "-Wl,-z,noexecstack", the resulting shared library is marked E stack. I suppose that if you link a binary against that shared library, it will be marked E stack as well. My toolchain also creates E stack binaries by default, no matter whether they're static or dynamic. It may be that my build of musl is bad. I am not interested enough in the details of what happens at the ld level to try and figure out if there's *something* that causes it to mark E stack when it should not; it requires spending much more quality time with binutils than I am comfortable with. All I know is that none of the object files in my software needs E stack, and bullying ld into doing the right thing works, so I'm content with that solution. -- Laurent
Re: Journey to s6-svscan as PID 1 on FreeBSD (almost there)
On 08.04.21 15:15, Dewayne Geraghty wrote: First we started with the documented approach of appending to /etc/ttys "" "/usr/local/bin/s6-svscan /run/scan""" on Which worked nicely under FreeBSD's /sbin/init. Then we added to loader.conf an init_script which is invoked via /sbin/init. This also worked well, but init remained as pid 1. The init_script variable is defined in /boot/loader.conf, in my case as: init_script="/root/bin/init_script.sh /sbin/init calls "init_script" This ran nicely, but still under init as pid=1. Finally I sucked up the courage and defined in /boot/loader.conf init_exec=/root/bin/init_exec.sh which contains #!/usr/local/bin/execlineb -S0 redirfd -wnb 1 /m/fifo/catch_all redirfd -r - /dev/null fdmove -c 2 1 exec -c /sbin/s6/svscan -t 0 -s /s/scan And finally /sbin/s6-svscan runs as pid 1. There is a cleaner solution. I prefer to set the init_path kernel env var in /boot/loader.conf and prepend the s6 stage1 script to the normal search path. That way the kernel starts my execline script as PID 1 and if I mess it I just have to clear the variable from the loader prompt.
Re: execlineb ELF executable stack on Linux
On Thu, Apr 8, 2021 at 2:25 PM Laurent Bercot wrote: > What works is setting the attribute at the link level > (-Wl,-z,noexecstack in LDFLAGS), and if it's not done, apparently > GNU toolchains still mark the stack as executable by default in the > binaries. Yes that is effective at marking the stack non-executable in the ELF image, but I'm concerned that may just be hiding the source of this weird behavior and might break something in the process. I did a bit more investigation and here are my findings. 1) I think I misunderstood how the kernel warning works. It may only apply to images that the kernel execve() itself directly rather than to any execve() syscall. It turns out that *all* skarnet ELF executables (at least from the execline, s6, s6-src, s6-linux-init and utmps packages) have an executable stack. Except one: s6-linux-init-nuke (more on that further). 2) Simply compiling a Hello World program with gcc without any option and linked with musl libc produces an executable with a non-executable stack by default. 3) I went back and triple-checked all objects, static and shared libraries from execline, skalibs, musl, and gcc. None of them have the executable stack flag. readelf -l | grep GNU_STACK on shared libraries shows 'RW' not 'RWE' in the flags column readelf -S | grep GNU-stack on objects and static libraries shows no 'X' in the flags column And yet somehow the resulting skarnet linked executables end up with a 'RWE' GNU_STACK. Very strange. 4) s6-linux-init-nuke's GNU_STACK flags are 'RW' (so not executable) and it is the only skarnet executable I have built that is not linked with skalibs! So it seems there is something in one of the skalibs that tells the linker to mark the stack as executable. I wish I could tell you what, but that is beyond my expertise.
Re: Journey to s6-svscan as PID 1 on FreeBSD (almost there)
Finally I sucked up the courage and defined in /boot/loader.conf
init_exec=/root/bin/init_exec.sh
which contains
#!/usr/local/bin/execlineb -S0
redirfd -wnb 1 /m/fifo/catch_all
redirfd -r - /dev/null
fdmove -c 2 1
exec -c /sbin/s6/svscan -t 0 -s /s/scan
And finally /sbin/s6-svscan runs as pid 1.
That's awesome! So, if I understand correctly:
- /sbin/init is still the first program loaded by the kernel, it's
the stock FreeBSD thing that performs FreeBSD-specific setups
- if there is an init_exec entry in /boot/loader.conf, /sbin/init
execs into its value, which allows you to run a different pid 1
for the lifetime of the machine.
If that's really how it works, it's incredibly good, and would make
s6-freebsd-init a breeze to write.
The necessary modifications to turn s6-linux-init into s6-freebsd-init
would just be:
- Comment out kbspecials() and its invocations:
https://github.com/skarnet/s6-linux-init/blob/master/src/init/s6-linux-init.c#L70
(does FreeBSD's /sbin/init set up CAD handling and other related
console things?)
- Port the mount() and umount() invocations, replacing mount with the
correct FreeBSD way of mounting a tmpfs:
https://github.com/skarnet/s6-linux-init/blob/master/src/init/s6-linux-init.c#L209
- Port the final reboot() call:
https://github.com/skarnet/s6-linux-init/blob/master/src/shutdown/s6-linux-init-hpr.c#L81
and all the rest should work as is. It's a game-changer.
foreground { /sbin/shutdown -o -r now } # this invokes reboot rather
than send a signal to init.
I'd greatly appreciate advice as to completing a clean shutdown?
My current approach is, indeed, to keep s6-svscan running as pid 1,
without making it exec into anything else, for the whole lifetime of
the machine, shutdown procedure included. So yes, for a clean
shutdown, you'd just make sure to kill all processes, then unmount
filesystems (and remount / read-only), then invoke reboot/poweroff
directly. That is how the current s6-linux-init works; I don't see a
reason why it shouldn't work on FreeBSD.
PS This journey commenced with the simple wish to safely rotate apache
logs, now I'm hooked on s6-rc (and the peace of mind it brings)
Glad you like it :)
--
Laurent
Journey to s6-svscan as PID 1 on FreeBSD (almost there)
First we started with the documented approach of appending to /etc/ttys
"" "/usr/local/bin/s6-svscan /run/scan""" on
Which worked nicely under FreeBSD's /sbin/init.
Then we added to loader.conf an init_script which is invoked via
/sbin/init. This also
worked well, but init remained as pid 1. The init_script variable is
defined in /boot/loader.conf, in my case as:
init_script="/root/bin/init_script.sh
/sbin/init calls "init_script"
This ran nicely, but still under init as pid=1.
Finally I sucked up the courage and defined in /boot/loader.conf
init_exec=/root/bin/init_exec.sh
which contains
#!/usr/local/bin/execlineb -S0
redirfd -wnb 1 /m/fifo/catch_all
redirfd -r - /dev/null
fdmove -c 2 1
exec -c /sbin/s6/svscan -t 0 -s /s/scan
And finally /sbin/s6-svscan runs as pid 1.
I'm very pleased with the result except... after a reboot, I have to
fsck -y /
because when I shutdown, the system crashes due to pid 1 being killed.
At the moment I have
/s/scan/.s6-svscan/SIGTERM and /s/scan/.s6-svscan/finish containing
#!/usr/local/bin/execlineb -S0
foreground { /bin/sync }
foreground { /sbin/mount -ur / }
foreground { /sbin/shutdown -o -r now } # this invokes reboot rather
than send a signal to init.
I'd greatly appreciate advice as to completing a clean shutdown?
(FYI for other FreeBSD folk, I hacked the order of calling init_script
then init_exec in init.c because the order is incorrect - I'll pursue
with the FreeBSD devs. Its only a few lines moved around. I use
init_script to prepare the environment for init_exec, though it could
all be done within init_exec script.)
Regards, Dewayne
PS This journey commenced with the simple wish to safely rotate apache
logs, now I'm hooked on s6-rc (and the peace of mind it brings)
Re: execlineb ELF executable stack on Linux
Okay, so the problem is that setting the noexecstack attribute at the asm level (-Wa,--noexecstack in CFLAGS) is useless: even without it, .note.GNU-stack is never marked as executable unless you have nested functions. What works is setting the attribute at the link level (-Wl,-z,noexecstack in LDFLAGS), and if it's not done, apparently GNU toolchains still mark the stack as executable by default in the binaries. I will change the configure scripts to specify noexecstack at the LDFLAGS level. Thanks for bringing this to my attention :) -- Laurent
Re: Small bug in utmps configure script
As of version 0.1.0.0, utmps' configure script ignores the --with-xtmpd-socket options. Indeed, thanks for the report, and the patch. Fixed in git. Bugfix release coming out, uh, at some point in the future when I do the next batch :) -- Laurent
Re: execlineb ELF executable stack on Linux
Recent versions of the Linux kernel issue a warning when a process executes an image with an executable stack, as an indicator of a _potential_ security vulnerability. Such a warning is issued when execlineb is executed. I just checked my kernel logs and indeed, I get such a warning as well. That's pretty weird, because there is nothing in execlineb that should require an executable stack. I will take a look at all the source files involved in the final execlineb binary, to see whether something is poisoning it. Honestly, I'm baffled - and I hope to all the gods it's just a stupid mistake I've made somewhere, and not a complex issue involving compiler and linker details that will take two weeks to untangle. Thanks for the report. I'll keep you informed. -- Laurent
