Journey to s6-svscan as PID 1 on FreeBSD (almost there)
On 8/04/2021 11:52 pm, Laurent Bercot wrote:
>> Finally I sucked up the courage and defined in /boot/loader.conf
>> init_exec=/root/bin/init_exec.sh
>> which contains
>> #!/usr/local/bin/execlineb -S0
>> redirfd -wnb 1 /m/fifo/catch_all
>> redirfd -r - /dev/null
>> fdmove -c 2 1
>> exec -c /sbin/s6/svscan -t 0 -s /s/scan
>>
>> And finally /sbin/s6-svscan runs as pid 1.
>
> That's awesome! So, if I understand correctly:
>
> - /sbin/init is still the first program loaded by the kernel, it's
> the stock FreeBSD thing that performs FreeBSD-specific setups
> - if there is an init_exec entry in /boot/loader.conf, /sbin/init
> execs into its value, which allows you to run a different pid 1
> for the lifetime of the machine.
Yes. That is correct.
Sequence: bios -> boot (MBR & GPT differ slightly) -> loader -> kernel
-> /sbin/init
(loader reads and executes /boot/loader.conf this enables things like
setting kernel environment variables, read-only kernel settings,
attaching encrypted devices, some controls over init and human
interaction with loader. My apologies if its obvious that I have no
knowledge of Linux, as I went from IBM OS, CP/M, VMS, FreeBSD)
>
> If that's really how it works, it's incredibly good, and would make
> s6-freebsd-init a breeze to write.
> The necessary modifications to turn s6-linux-init into s6-freebsd-init
> would just be:
>
> - Comment out kbspecials() and its invocations:
> https://github.com/skarnet/s6-linux-init/blob/master/src/init/s6-linux-init.c#L70
>
I really appreciate the direct links. I'd need to set aside time to
understand what s6-linux-init does, though kbspecials looks like its
performs an open console (been more than 30+ years since I worked in C)
> (does FreeBSD's /sbin/init set up CAD handling and other related
> console things?)
Yes. /sbin/init normally owns the console and runs getty (stuff).
So from init.c
init_exec invokes replace_init(path) -> execute_script() ->
open_console() & execv()
while
init_script -> run_script -> forks -> execute_script -> open_console & execv
> - Port the mount() and umount() invocations, replacing mount with the
> correct FreeBSD way of mounting a tmpfs:
> https://github.com/skarnet/s6-linux-init/blob/master/src/init/s6-linux-init.c#L209>
init.c creates and calls "mount_tmpfs" (using nmount) which should be a
cut/paste to mount at S6_LINUX_INIT_TMPFS, the purpose for it in FreeBSD
is to enable a reroot to occur :)
I notice runlevels which FreeBSD doesn't have. I don't think the 5
security modes is the same -1 is default, insecure.
> - Port the final reboot() call:
> https://github.com/skarnet/s6-linux-init/blob/master/src/shutdown/s6-linux-init-hpr.c#L81
>
Yes, I think this is the challenge. I need to figure out why the signals
aren't getting caught (& handled) by s6-svscan. :-/
init.c is 2043 lines of code and a lot of that is in managing ttys and
shutdown. The reboot command should be fairly straight-forward.
Humorous comment re glibc multilib.
>> and all the rest should work as is. It's a game-changer.
>
>
>> foreground { /sbin/shutdown -o -r now } # this invokes reboot rather
>> than send a signal to init.
>>
>> I'd greatly appreciate advice as to completing a clean shutdown?
>
> My current approach is, indeed, to keep s6-svscan running as pid 1,
> without making it exec into anything else, for the whole lifetime of
> the machine, shutdown procedure included. So yes, for a clean
> shutdown, you'd just make sure to kill all processes, then unmount
> filesystems (and remount / read-only), then invoke reboot/poweroff
> directly. That is how the current s6-linux-init works; I don't see a
> reason why it shouldn't work on FreeBSD.
>
This is where I'm stuck. If I can achieve a clean shutdown, then init
is a non-issue. Largely because my servers are headless, so I don't
need a console (or ttys) unless I have a hardware fault, in which case
I'd boot from a clean device (if possible).
I've given some thought to modify s6-svscan to include the shutdown
code, but we're not desperate yet.
>
>> PS This journey commenced with the simple wish to safely rotate apache
>> logs, now I'm hooked on s6-rc (and the peace of mind it brings)
>
> Glad you like it :)
>
> --
> Laurent
Thank-you for your informative reply. I'll schedule some time to work
through this properly and come back.
Re: Journey to s6-svscan as PID 1 on FreeBSD (almost there)
On 9/04/2021 4:29 am, Crest wrote: > There is a cleaner solution. I prefer to set the init_path kernel env > var in /boot/loader.conf and prepend the s6 stage1 script to the normal > search path. That way the kernel starts my execline script as PID 1 and > if I mess it I just have to clear the variable from the loader prompt. Thank-you for the reminder, I'd forgotten about kern.init_path. Though in both cases its really a matter of changing a variable after the loader has completed its chores (For non-FreeBSDers, there are two points where a boot can be interrupted prior to the kernel starting) I use both init_script to establish all the prerequisite stuff, check boot device, enable MAC(biba), enable firewall, start network, start ssh, create logging environment; then handoff to s6-svscan, via init_exec. Yes it could be done in one script which init_path points to, and I agree that would be cleaner. But the prerequisite stuff really needs a sh, and I like execlineb doing its redirection prep and handing off to s6-svscan :)
Journey to s6-svscan as PID 1 on FreeBSD (almost there)
First we started with the documented approach of appending to /etc/ttys
"" "/usr/local/bin/s6-svscan /run/scan""" on
Which worked nicely under FreeBSD's /sbin/init.
Then we added to loader.conf an init_script which is invoked via
/sbin/init. This also
worked well, but init remained as pid 1. The init_script variable is
defined in /boot/loader.conf, in my case as:
init_script="/root/bin/init_script.sh
/sbin/init calls "init_script"
This ran nicely, but still under init as pid=1.
Finally I sucked up the courage and defined in /boot/loader.conf
init_exec=/root/bin/init_exec.sh
which contains
#!/usr/local/bin/execlineb -S0
redirfd -wnb 1 /m/fifo/catch_all
redirfd -r - /dev/null
fdmove -c 2 1
exec -c /sbin/s6/svscan -t 0 -s /s/scan
And finally /sbin/s6-svscan runs as pid 1.
I'm very pleased with the result except... after a reboot, I have to
fsck -y /
because when I shutdown, the system crashes due to pid 1 being killed.
At the moment I have
/s/scan/.s6-svscan/SIGTERM and /s/scan/.s6-svscan/finish containing
#!/usr/local/bin/execlineb -S0
foreground { /bin/sync }
foreground { /sbin/mount -ur / }
foreground { /sbin/shutdown -o -r now } # this invokes reboot rather
than send a signal to init.
I'd greatly appreciate advice as to completing a clean shutdown?
(FYI for other FreeBSD folk, I hacked the order of calling init_script
then init_exec in init.c because the order is incorrect - I'll pursue
with the FreeBSD devs. Its only a few lines moved around. I use
init_script to prepare the environment for init_exec, though it could
all be done within init_exec script.)
Regards, Dewayne
PS This journey commenced with the simple wish to safely rotate apache
logs, now I'm hooked on s6-rc (and the peace of mind it brings)
Can s6-tlsd use openssl, rather than libressl?
Thanks to Ilaia's email, I looked into using s6-tlsd, but I'm a bit confused about what libraries are needed, and hopefully not libressl? https://www.skarnet.org/software/s6-networking/ indicates dependency on stls, sbearlssl. https://www.skarnet.org/software/s6-networking/libstls/ requires libressl https://www.skarnet.org/software/s6-networking/libsbearssl/ requires bearssl We tried to use libressl for approx 8 months after heartbleed, but there were too many incompatibilities with other applications' Makefiles, which may have been fixed over the last 2 years? Now we're locked into both openssl, bearssl is only used in boot code. Does anyone use s6-networking with only openssl, or better, does anyone use s6-networking on a BSD (I'm FreeBSD based) which uses openssl by default? (If s6-* has libtls is a dependency, that's a showstopper.) Regards, Dewayne.
Re: False positive in skalibs system feature test
Thankyou Laurent. So cat /usr/ports/devel/skalibs/work/skalibs-2.9.1.0/sysdeps.cfg/sysdeps reveals getrandom: yes Now adding --with-sysdep-getrandom=no to the configure line, we get from: /usr/ports/devel/skalibs/work/skalibs-2.9.1.0/sysdeps.cfg/sysdeps getrandom: no and we obtained a 211K stripped image libskarnet.so.2.9.1.0 This on FreeBSD 12.1 using clang 8.0.1 and ld 2.32 (note not the default lld). I hope that helps? However there were some (common) warning messages: > src/libstddjb/child_spawn1_internal.c:35:23: warning: & has lower precedence than !=; != will be evaluated first [-Wparentheses] > if (p[to & 1] != to & 1) > ^ > src/libstddjb/child_spawn1_internal.c:35:23: note: place parentheses around the '!=' expression to silence this warning > if (p[to & 1] != to & 1) > ^ > ( ) > src/libstddjb/child_spawn1_internal.c:35:23: note: place parentheses around the & expression to evaluate it first > if (p[to & 1] != to & 1) > ^ >( ) > 1 warning generated.
Re: False positive in skalibs system feature test
While building skalibs-2.9.1.0, I notice that the getrandom option isn't available on FreeBSD 12.1 Stable. What I have from ./configure --help is: "... Sysdeps autodetection override: --with-sysdep-K=V assume sysdep K has the value V [autodetected] List of mandatory K for cross-compiling: devurandom (V=yes|no) " To your question Laurent about what other (non-Linux) people use, as FYI normal FreeBSD people (on FreeBSD 12.1Stable) use for cc, clang 8.0.1 # ld -v LLD 8.0.1 ; an LLVM-based linker For dev/testing I also use gcc 9.2.0 and ld (GNU Binutils) 2.32 as needed. On HardenedBSD which uses LLVM tooling, there are additional flags like -fPIC -fPIE and for linking -pie -z relro -z now; and optional settings for CFI but that's becoming esoteric. ;) Cheers, Dewayne.
