s6-rc-init verbose equivalent messages?
Is there anyway of tracking down the cause of the following fatal failure? # /usr/local/bin/s6-rc-init -c /s/comp -l /s/run /s/scan s6-rc-init: fatal: unable to supervise service directories in /s/run/servicedirs: No such file or directory I've completed a disk-disk copy, as I need to integrate s6 into hardenedbsd. The compile command works: # /usr/local/bin/s6-rc-compile -v9 /s/comp /usr/local/etc/s6 ... all good. But s6-rc-init is stuck, as above. There are no changes to the system except IP addresses. I'm baffled! Regards, Dewayne.
Re: Query on s6-log and s6-supervise
Apologies, I'd implied that we have multiple s6-supervise processes running and their children pipe to one file which is read by one s6-log file. You can achieve this outcome by using s6-rc's, where one consumer can receive multiple inputs from producers. There is a special (but not unique) case where a program, such as apache which will have explicit log files (defined in apache's config file) to record web-page accesses and error logs, on a per server basis. Because all the supervised apache instances can write to one error logfile, I instructed apache to write to a pipe. Multiple supervised apache instances using the one pipe (aka funnel), which was read by one s6-log. This way reducing the number of (s6-log) processes. I could do the same with the access logs and use the regex function of s6-log, but I tend to simplicity.
Re: Query on s6-log and s6-supervise
Thanks Laurent, that's really interesting. By comparison, my FBSD system uses: # ps -axw -o pid,vsz,rss,time,comm | grep s6 virt KB resident cpu total 38724 10904 1600 0:00.02 s6-log 41848 10788 1552 0:00.03 s6-log 42138 10848 1576 0:00.01 s6-log 4 10888 1596 0:00.02 s6-log 45878 10784 1516 0:00.00 s6-svscan 54453 10792 1544 0:00.00 s6-supervise ... lots ... 67937 10792 1540 0:00.00 s6-supervise 76442 10724 1484 0:00.01 s6-ipcserverd 76455 11364 1600 0:00.01 s6-fdholderd 84229 10896712 0:00.01 s6-log Processes pull-in both ld-elf and libc.so, from procstat -v start end path 0x1021000 0x122a000 /usr/local/bin/s6-supervise 0x801229000 0x80124f000 /libexec/ld-elf.so.1 0x801272000 0x80144c000 /lib/libc.so.7 Yes - libc is ... large. Arjun, if you want to reduce the number of s6-log processes perhaps consider piping them to a file which s6-log reads from. For example we maintain various web servers, the accesses are unique and of interest to customers, but they don't (really) care about the errors so we aggregate this with one s6-log. Works very well :)
Re: s6-rc : Anomalies or normal behaviour
On 4/10/2020 1:14 pm, Laurent Bercot wrote:
>> 1. I expected to see the date in seconds since time epoch, but result is
>> variable name
>> # execlineb -Pc 'backtick D { date "+%s" } echo $D'
>> $D
>
> Normal behaviour, since there's no shell to interpret $D as the
> contents of variable D. Try using "importas D D" before the echo:
> it will read the value of D and substitute $D with this value, so
> echo will print the value. Yeah, execline is annoying like that, it's
> just a habit to take.
> Also, you generally want "backtick -n", to chomp the newline at
> the end of your input.
>
>
>> ---
>> 2. When I use emptyenv within an execlineb script, I have a "defunct"
>> zombie process
>> 89685 3 S< 0:00.01 |-- s6-supervise base:time-srv
>> 3020 - S> -N -g -u ntpd --nofork
>> 3601 - Z< 0:00.00 | `--
>>
>> The time server script is
>> #!/usr/local/bin/execlineb -P
>> emptyenv
>> multidefine -d " " "base time ntpd /usr/local/sbin/ntpd" { JAIL SERVICE
>> USER PROGRAM }
>> background { echo Starting service $SERVICE using $PROGRAM on $JAIL
>> under user $USER }
>> fdmove 2 1
>> redirfd -w 1 /m/base:time/fifo
>> $PROGRAM -c /etc/ntp.conf -N -g -u $USER --nofork
>>
>> removing emptyenv, prevents the zombie from being created. Is this
>> normal?
>
> The zombie is the echo program in your background block, since it's a
> direct child of your run script and there's nothing that reaps it
> after it's forked (fdmove, redirfd, ntpd - those programs don't expect
> to inherit a child). So the zombie is expected. To prevent that, use
> "background -d", which will doublefork your echo program, so it will
> be reparented to pid 1 which will reap it properly.
>
EDIT My error, the problem was background, and -d fixes this.
> The anomaly is that you *don't* have that zombie without emptyenv;
> my first guess is that there's something in your environment that changes
> the behaviour of ntpd and makes it reap the zombie somehow.
>
>
>> ---
>> 3. Is it normal/standard/good practice to include a dependency in a
>> bundle. For example, I have a "time" bundle whose contents are
>> time-srv. time-srv starts the ntpd service, and has as a dependency
>> time-log.
>>
>> Using "s6-rc -u change time", everything behaves as documented, ie
>> starts "time" which starts time-log, then time-srv. However
>>
>> # s6-rc -v 9 -d change base:time
>> s6-rc: info: bringing selected services down
>> s6-rc: info: processing service base:time-srv: stopping
>> s6-rc: info: service base:time-srv stopped successfully
>> # Starting logging service time for base with user s6log folder
>> /var/log/time
>>
>> and the time-log continues running.
>
> If you only have time-srv in your 'time' bundle, then time-srv and
> time are equivalent. Telling s6-rc to bring down time will do the
> exact same thing as telling it to bring down time-srv. time-log is
> not impacted. So the behaviour is expected.
>
> If you want "s6-rc -d change time" to also bring down time-log, then
> yes, you should add time-log to the time bundle. Then 'time' will
> address both time-srv and time-log.
>
>
>> y 6 seconds # This is time-srv
>> up (pid 85131) 6 seconds # This is time-log,so it
>> has been restarted
>
> If you're using a manually created named pipe to transmit data
> from time-srv to time-log, that pipe will close when time-srv exits,
> and your logger will get EOF and probably exit, which is why it
> stopped; but time-log's supervisor has received no instruction that
> it should stop, so it will restart it. This is also expected.
>
> The simplest way of achieving the behaviour you want is s6-rc's
> integrated pipeline feature. Get rid of your named pipe and of your
> stdout (for time-srv) and stdin (for time-log) redirections; get rid
> of your time bundle definition. Then declare time-log as a consumer
> for time-srv and time-srv as a producer for time-log. In the
> time-log source definition directory, write 'time' into the
> pipeline-name file. Then recompile your database.
>
> This will automatically create a pipe between time-srv and time-log;
> the pipe will be held open so it won't close even if one of the
> processes exits; and it will automatically create a 'time' bundle
> that contains both time-srv and time-log.
>
> You're on the right track. :)
>
> --
> Laurent
>
>
Laurent,
Thank-you very much. Using your advise (re 1 & 2) I've redeployed our
testing platform and everything works as expected :)
re 3. Implementing the producer-for/consumer-for pair, we've gone from
(The application server in jail b3 to log server in jail b2 Ref1).
# cat b3:named-setup2/up
#!/usr/local/bin/execlineb -P
define D /m/b3/fifo/named
foreground { if -n { test -p $D } foreground { /usr/bin/mkfifo $D } }
foreground { /usr/sbin/chown s6log:named $D }
foreground { /bin/chmod 720 $D }
# cat b3:named2/run
#!/usr/local/bin/execlineb -P
fdmove 2 1
redirfd -w 1 /m/b3/fifo/named
/usr/sbin/jexec b3
Re: s6-rc : Anomalies or normal behaviour
Apologies, my earlier email, item 2, pointed to emptyenv as the cause of
zombie processes on FreeBSD 12.2S, actually it is due to background.
# execlineb -Pc 'background { echo hello } pipeline { ps -axw } grep
defunct'
hello
30144 0 Z+ 0:00.00
while the following tests both foreground and emptyenv
# execlineb -Pc 'emptyenv foreground { echo hello } pipeline { /bin/ps
-axw } /usr/bin/grep defunct'
hello
#
Software revision level (as available in the FreeBSD ports system)
execline-2.6.0.1
s6-2.9.1.0
s6-rc-0.5.1.2
skalibs-2.9.2.1
Further detail:
# execlineb -Pc 'emptyenv background { echo hello } pipeline { /bin/ps
-axwwdo pid,ppid,stat,command } /usr/bin/grep -B1 "defunct"'
hello
71212 70760 Ss | | `-- -csh (csh)
16885 71212 S+ | | `-- /usr/bin/grep -B1 defunct
17052 16885 Z+ | | |--
I've also placed a ktrace and kdump of
execlineb -Pc 'ktrace -f /tmp/bgnd.kt /usr/local/bin/background {
/bin/ps } echo a'
here
http://www.heuristicsystems.com/s6/
s6-rc : Anomalies or normal behaviour
Is this correct behaviour or are these just anomalies?
1. Use of backtick variable assignment on FreeBSD doesn't appear correct
2. Use of emptyenv results in a remnant "defunct" process
3. Should a bundle's contents file include the dependencies of its
contents file, for a down change to the bundle to bring the service's
components down?
1. I expected to see the date in seconds since time epoch, but result is
variable name
# execlineb -Pc 'backtick D { date "+%s" } echo $D'
$D
Note: this isn't how I intend to use backtick, but I try to use the
simplest case to understand how things work
---
2. When I use emptyenv within an execlineb script, I have a "defunct"
zombie process
89685 3 S< 0:00.01 |-- s6-supervise base:time-srv
3020 - S
The time server script is
#!/usr/local/bin/execlineb -P
emptyenv
multidefine -d " " "base time ntpd /usr/local/sbin/ntpd" { JAIL SERVICE
USER PROGRAM }
background { echo Starting service $SERVICE using $PROGRAM on $JAIL
under user $USER }
fdmove 2 1
redirfd -w 1 /m/base:time/fifo
$PROGRAM -c /etc/ntp.conf -N -g -u $USER --nofork
removing emptyenv, prevents the zombie from being created. Is this normal?
---
3. Is it normal/standard/good practice to include a dependency in a
bundle. For example, I have a "time" bundle whose contents are
time-srv. time-srv starts the ntpd service, and has as a dependency
time-log.
Using "s6-rc -u change time", everything behaves as documented, ie
starts "time" which starts time-log, then time-srv. However
# s6-rc -v 9 -d change base:time
s6-rc: info: bringing selected services down
s6-rc: info: processing service base:time-srv: stopping
s6-rc: info: service base:time-srv stopped successfully
# Starting logging service time for base with user s6log folder
/var/log/time
and the time-log continues running.
Admittedly
# s6-svstat /s/scan/base:time-srv ; s6-svstat /s/scan/base:time-log
down (exitcode 0) 6 seconds, ready 6 seconds # This is time-srv
up (pid 85131) 6 seconds # This is time-log,so it
has been restarted
To obtain the desired/expected behaviour and bring time-log down must it
also be added to the bundle's contents?
These observations were made using FreeBSD 12.2Stable on amd64.
Apologies for still asking newbie questions, but I'm trying to embed s6
here, which translates to properly understand.
Regards, Dewayne.
Re: runit SIGPWR support
Yes Colin, you're right. PWR is not in the FreeBSD list of signals, per # kill -l HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM URG STOP TSTP CONT CHLD TTIN TTOU IO XCPU XFSZ VTALRM PROF WINCH INFO USR1 USR2 LWP Also doesn't appear in Open Group Base Specifications Issue 7 list of signals https://pubs.opengroup.org/onlinepubs/9699919799/ Cheerio.
Re: s6 usability
On the question of PATH for BSD land (FreeBSD, TrueOS, HardenedBSD et al), binaries installed from packages (ports) live under /usr/local, with the exception of /var and /tmp. The wars were fought and /usr/local can easily be mounted read-only. Of the 1446 packages I have installed (no desktop stuff), the breakdown is # ls /usr/local/bin/ | wc -l 2857 # ls /usr/local/sbin/ | wc -l 349 # find /usr/local/bin/ -type d -depth 1 /usr/local/bin/db5; # No directories under /usr/local/sbin # ls /usr/local/libexec|wc -l 72 The placement of files, is more a "distribution" decision that I'm sure is already settled. On the documentation front, Laurent's work is very good, but I did find the examples from the gentoo docset for s6 & s6-rc, a life saver for someone with no prior process or service management background. (I'd only used monit previously, and still do to reload application configs and some other system state change events over s6 tools).
Re: s6 usability
Hi Steve, Does the *user* need to code execline scripts, or is it just something the program does? If the former, then make a point that one doesn't need to use execline for s6-rc to be a very powerful startup system. No the user doesn't need to write execline scripts. The following equally applies to s6-rc. Refer to:https://skarnet.org/software/s6/overview.html for: "execline makes it natural to handle long command lines made of massive amounts of chain loading. This is by no means mandatory, though: a run script can be any executable file you want, provided that running it eventually results in a long-lived process with the same PID." Regarding creating a s6 subdir of bin. I have some 1325 applications (FreeBSD people call them ports), only 1 has a separate directory under bin.
Re: runit patches to fix compiler warnings on RHEL 7
Jan, I'm also a virgin to process/service management software, learning s6-rc, s6, execlineb is not for the faint-hearted nor the time-poor. Getting a handle on the concepts, and the naming conventions - its really hard work. Execline enforces a discipline, a rigor demanding anticipatory planning (to get right). I ran some performance tests and execlineb is marginally better. So why persist? Largely because an execline script is immediately obvious and explicit. Seriously, at a glance you know what the script will achieve. Could I write a sh script to do the same task? Yes, and probably do it a lot quicker. But. I would loose the elegance and readability - where sh has an equivalence with assembler, execline is akin to BASIC, it makes you think differently :) I'm developing solutions for PROTECTED level security (its an Australian Govt thing), and skarnet's service management provides assurance, and s6-log provides near-certainty of logging completeness. I'm very happy with the toolset, worth the time investment.
Re: s6-log can create current with 640?
Hi Laurent, Answers embedded
On 26/10/2019 4:27 pm, Laurent Bercot wrote:
>> I'd mistakenly assumed execlineb knew where its friends were; though in
>> hindsight its a bit much to assume that execlineb internally changes the
>> PATH.
>
> The real question is, why is there a "umask" binary that's not the one
> from execline? Non-chainloading non-builtin umask is nonsense, just
> like non-chainloading non-builtin cd.
>
I can only shed a ray of light, though this /usr/bin/umask has been
around for some time. I fired up an old FreeBSD 9.2 system that has it.
Content being:
#!/bin/sh
# $FreeBSD: stable/9/usr.bin/alias/generic.sh 151635 2005-10-24
22:32:19Z cperciva $
# This file is in the public domain.
builtin ${0##*/} ${1+"$@"}
which is the same as 12.1. Why? I can only refer to src logs:
...
r151635 | cperciva | 2005-10-25 08:32:19 +1000 (Tue, 25 Oct 2005) | 10 lines
Use the "builtin" shell function to make sure that the requested
command is handled as a shell function. This avoids the following
peculiar behaviour when /usr/bin is on a case-insensitive filesystem:
# READ foo
(... long pause, depending upon the amount of swap space available ...)
sh: Resource temporarily unavailable.
...
r100200 | wollman | 2002-07-17 08:16:05 +1000 (Wed, 17 Jul 2002) | 5 lines
A little bit more thought has resulted in a generic script which can
implement any of the useless POSIX-required ``regular shell builtin''
utilities, saving one frag and one inode each. The script moves to
usr.bin/alias which is alphabetically the first of these commands.
the last entry before the cvs logs were transferred to svn.
Examining the Makefile/usr/src/usr.bin/alias/Makefile
# $FreeBSD: stable/12/usr.bin/alias/Makefile 284255 2015-06-11 04:22:17Z
sjg $
SCRIPTS=generic.sh
SCRIPTSNAME=alias
LINKS= ${BINDIR}/alias ${BINDIR}/bg \
${BINDIR}/alias ${BINDIR}/cd \
${BINDIR}/alias ${BINDIR}/command \
${BINDIR}/alias ${BINDIR}/fc \
${BINDIR}/alias ${BINDIR}/fg \
${BINDIR}/alias ${BINDIR}/getopts \
${BINDIR}/alias ${BINDIR}/hash \
${BINDIR}/alias ${BINDIR}/jobs \
${BINDIR}/alias ${BINDIR}/read \
${BINDIR}/alias ${BINDIR}/type \
${BINDIR}/alias ${BINDIR}/ulimit \
${BINDIR}/alias ${BINDIR}/umask \
${BINDIR}/alias ${BINDIR}/unalias \
${BINDIR}/alias ${BINDIR}/wait
and yes they exist in /usr/bin/
>From tcsh
# which cd
cd: shell built-in command.
>From sh
# which echo
/bin/echo
Ok - that's done my head in.
>
>> Unfortunately it seems that the path can't be set within execlineb
>> context.
>
> Of course it can. What's happening is that both export and envfile
> set the PATH *on execution of the next command*, so the command that's
> right after them will still be searched with the old PATH. In other words:
>
> "export PATH /usr/local/bin umask 033 echo blah" will not work, because
> umask will still be searched for in the old PATH (but echo would be
> searched in the new PATH), but
> "export PATH /usr/local/bin exec umask 033 echo blah" will work, because
> exec is searched in the old PATH and umask is searched in the new PATH.
>
> (exec is an execline nop that can be useful in those cases, if you
> don't have another command to put between the export PATH and the
> command you need to search in the new PATH.)
>
Thank-you, the explanation helps. I guess through bad experiences with
(non-execline) exec, I avoid it unless I really do want to transfer
control to it.
I think with this example
# rm -v /tmp/t1 ; setenv PATH
/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin ; execlineb -Pc 'importas
op PATH emptyenv export PATH /usr/local/bin:$op exec umask 002 redirfd
-w 1 /tmp/t1 echo blah4' ; /bin/ls -l t1 ; cat /tmp/t1
/tmp/t1
-rw-rw-r-- 1 root wheel 6 26 Oct 18:02 t1
blah4
Due to the above discussion around umask, I will have to consider either
adding to my execline scripts :(
importas op PATH
emptyenv
export PATH /usr/local/bin:$op
or explicit paths as needed.
Thank-you to the contributors for your patience.
Re: s6-log can create current with 640?
On 26/10/2019 4:06 am, Guillermo wrote: ... > Let me guess: the value of PATH is > /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin, > execline's chain loading umask is in /usr/local/bin, and FreeBSD > happens to have an 'umask' shell script in /usr/bin. If that is > correct, then you'll have to either use the absolute pathname > /usr/local/bin/umask in the execlineb chain, or run it with a PATH in > which /usr/local/bin comes before /usr/bin. > > G. > Irk! Thank-you Guillermo, your guess is correct. I'd mistakenly assumed execlineb knew where its friends were; though in hindsight its a bit much to assume that execlineb internally changes the PATH. This works correctly # rm /tmp/t1 ; setenv PATH "/usr/local/bin:/bin:/sbin:/usr/sbin:/usr/bin" ; printenv | grep -i path ; ktrace -f /tmp/t1-pathok-umaskfirst \ /usr/local/bin/execlineb -Pc 'export path "/usr/local/bin" umask 033 redirfd -w 1 /tmp/t1 echo hello' ; echo $? ; ls -l /tmp/t1 PATH=/usr/local/bin:/bin:/sbin:/usr/sbin:/usr/bin 0 -rw-r--r-- 1 root wheel 6 26 Oct 11:19 /tmp/t1 as does /usr/local/bin/execlineb -Pc '/usr/local/bin/umask 033 /usr/local/bin/redirfd -w 1 /tmp/t1 echo hello' Unfortunately it seems that the path can't be set within execlineb context. I tried both "path" and "PATH" in the following export and envfile examples. export /usr/local/bin/execlineb -Pc '/usr/local/bin/export PATH /usr/local/bin umask 033 redirfd -w 1 /tmp/t1 echo hello' ; echo $? ; ls -l /tmp/t1 rm: /tmp/t1: No such file or directory PATH=/bin:/sbin:/usr/sbin:/usr/bin 0 ls: /tmp/t1: No such file or directory envfile /usr/local/bin/execlineb -Pc '/usr/local/bin/envfile /tmp/e umask 035 redirfd -w 1 /tmp/t1 echo hello' ; echo $? ; ls -l /tmp/t1 where /tmp/e path = /usr/local/bin:/bin:/sbin:/usr/sbin:/usr/bin I've moved the various kdumps into http://www.heuristicsystems.com/s6-umask/ However I don't think there is a problem with export as /usr/local/bin/execlineb -Pc 'export T1 /tmp/t1 umask 033 importas T1v T1 redirfd -w 1 $T1v echo hello' works correctly, but envfile doesn't; might be worth a look. (?) Kind regards, Dewayne
Re: s6-log can create current with 640?
Laurent, I've embedded responses: On 24/10/2019 10:58 am, Laurent Bercot wrote: >> My initial attempt >> >> #!/usr/local/bin/execlineb -P >> s6-setuidgid uucp >> redirfd -r 0 /services/ntp/fifo >> umask 037 >> /usr/local/bin/s6-log -b n14 r7000 s10 S300 !"/usr/bin/xz -7q" >> /var/log/ntpd > > Hi Dewayne, > > - Is there a reason why you're using a manually created fifo instead > of the built-in logging facility offered by s6-svscan? You could tell > ntpd to log to /dev/stdout, and have the logger in a ntpd/log service > directory, and your logs would automatically be available on your > logger's stdin. > That's really testing my memory! I recall using the producer/consumer pair; as well as s6-trig-notify and s6-ftrig-listen. But at the end of the day, processes like apache have multiple log files, and in my case 5 access log files and one error log. So either I use one aggregated log (input) with multiple regex expressions in an s6-log statement (which I was advised against doing) or multiple s6-log for each input, which I chose. In addition the log content need to traverse different VM contexts, where I usually use lo0; but in this case, a nullfs and fifo was the better choice per some early benchmarks (via the wrk tool). Re: ntp and its log file? Yes within a single machine context, one process writes to stdout and another takes that as input for s6-log to handle, makes sense (and easy). However the "logger" will relocate to a separate VM, when unrelated problems with an ASLR enabled ntpd are fixed. Again a nullfs and fifo are sufficient - one VM writes to the fifo, a different user in a different VM reads & processes the log via s6-log. Aside: If someone isn't familiar with FreeBSD, it can be hard to get a handle on FreeBSD jails, its a bit more than a simple chroot. I think its reasonable to consider them as very lightweight VMs. Each can have separate security levels, shared memory; multiple network interfaces and IP addresses etc; all sharing the one kernel. The only con, is that the first IP address is treated as routable and localhost points to it. > - About umask: that's very strange. Can you strace, or ktrace, or > whatever tool gives you a list of system calls, the script? It will > show exactly what's going on. > Apologies for the delay. Needed to rebuild the kernel to enable ktrace. Results for umask rm -f /tmp/t1 ; /usr/local/bin/execlineb -Pc 'redirfd -w 1 /tmp/t1 umask 037 echo hello' ; echo $? ; ls -l /tmp/t1 0 -rw-r- 1 root wheel 0 25 Oct 18:33 /tmp/t1 rm -f /tmp/t1 ; /usr/local/bin/execlineb -Pc 'redirfd -w 1 /tmp/t1 umask 023 echo hello' ; echo $? ; ls -l /tmp/t1 0 -rw-r- 1 root wheel 0 25 Oct 18:36 /tmp/t1 I've placed the ktrace's dumpfile in txt format so its readable for you at http://www.heuristicsystems.com/s6-umask/t1.kdump The OS umask is 027. I also tried what I thought was the more logical # rm -f /tmp/t1 ; /usr/local/bin/execlineb -Pc 'umask 033 redirfd -w 1 /tmp/t1 echo hello' ; echo $? ; ls -l /tmp/t1 0 ls: /tmp/t1: No such file or directory This ktrace is http://www.heuristicsystems.com/s6-umask/t1-umask-redirfd.kdump I think the relevant lines are 276 sh CALL umask(0) 276 sh RET umask 23/0x17 276 sh CALL umask(027) 276 sh RET umask 0 276 sh CALL umask(033) 276 sh RET umask 23/0x17 276 sh CALL read(0xa,0x104c9d0,0x400) Platform details: # uname -pivKU; cc -v ; ld -v FreeBSD 12.1-STABLE #5 r353671M: Fri Oct 25 09:12:32 AEDT 2019 amd64 hqdev-amd64-smp-vga 1201500 1201500 FreeBSD clang version 8.0.1 (tags/RELEASE_801/final 366581) (based on LLVM 8.0.1) Target: x86_64-unknown-freebsd12.1 Thread model: posix InstalledDir: /usr/bin LLD 8.0.1 (FreeBSD 366581-128) (compatible with GNU linkers) If there is anything I can do to assist please let me know. (Though I havent worked in C since 1990...) Kind regards, Dewayne
Re: s6-log can create current with 640?
Thanks Jonathon. Both attempts were within the execlineb context, so I assume that umask was correctly employed ;). I tried My initial attempt #!/usr/local/bin/execlineb -P s6-setuidgid uucp redirfd -r 0 /services/ntp/fifo umask 037 /usr/local/bin/s6-log -b n14 r7000 s10 S300 !"/usr/bin/xz -7q" /var/log/ntpd which results in notice of "# echo: write: Broken pipe"; the ntp process properly starts but there is no ntp logging process. --- The usual flow is echo: write: Broken pipe ... and in quick succession (<1 sec): # ps -axww | grep ntpd 28869 - Rs 0:00.00 s6-setuidgid uucp redirfd -r 0 /service/fifo umask 037 /usr/local/bin/s6-log -b n14 r7000 s10 S300 !/usr/bin/xz -7q /var/log/ntpd # ps -axww | grep ntpd 29457 - Ss 0:00.03 /usr/local/sbin/ntpd -c /etc/ntp.conf -u ntpd -x -G --nofork and /var/log/ntpd folder is empty (with mode rwx--) --- If I move umask up one line (before redirfd), the logging process doesn't start; and blocks the start of ntp. There is a dependency relationship ntp-log <- ntp so this is expected. I was surprised that umask didn't work, but not overly concerned; as Colin, quite rightly, pointed to my directory permissions as being adequate. Now to migrate sendmail and its milters... Regards, Dewayne
Re: s6-log can create current with 640?
Thank-you, Colin. My brain turned to mush integrating logging with fifo queues across multiple jails (aka very lightweight VMs) and disjoint users (userA writes, userB reads). Unfortunately they're across various jailed systems, so the s6 fifo tools aren't applicable. I appreciate your advice, and yes, if there was anyone in the uucp group, I could be labelled "overly permissive"! ;^) Kind regards, Dewayne. PS I've gotten to like s6, it helped me discover a "workaround" for an aslr issue with ntp, which under normal circumstance I would've given up. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=241421
s6-log can create current with 640?
Is there any way to tell s6-log to set the mode to ./current to something other than 644? 640 is preferred? For example: I write to the logdir /var/log/httpd/error which has privs: /var/log/http drwx-- 2 uucp uucp 1.0K Oct 23 12:37 error/ Within /var/log/httpd/error -rwxr--r-- 1 uucp uucp 190K Oct 23 12:37 @40005dafaf1b180d862c.s* -rw-r- 1 uucp uucp 0B Oct 23 12:37 state -rw-r--r-- 1 uucp uucp 0B Oct 23 12:37 current I did try umask 037 but that just broke the pipe. All my log files are of this form #!/usr/local/bin/execlineb -P s6-setuidgid uucp redirfd -r 0 /services/ntp/fifo /usr/local/bin/s6-log -b n28 r7000 s20 S700 !"/usr/bin/xz -7q" /var/log/ntpd This is a big deal as I'm about to move my audit processing under s6-rc. (Aside: Actually I write to a fifo and then redirfd for s6-log to pick up the content and manage the log files. All works very nicely :) )
Re: A better method than daisy-chaining logging files?
Thanks Pica. I've sent the ktraces to Laurent and will update when possible.
Re: A better method than daisy-chaining logging files?
Good point. # ls -lrtha /var/log/httpd | grep error; ls -lrtha /var/log/httpd/error" drwx-- 2 mylogger www 512B Jun 18 17:31 error total 12 -rw-r--r-- 1 mylogger www 0B Jun 18 17:31 state -rw-r--r-- 1 mylogger www 0B Jun 18 17:31 lock drwxrwxrwx 6 mylogger www 512B Jun 18 17:31 .. drwx-- 2 mylogger www 512B Jun 18 17:31 . -rw-r--r-- 1 mylogger www 329B Jun 18 17:32 current Reflects my effort in previous testing that I forgot to reset! :) And there are no mandatory security labels in effect, for completeness ;)
Re: A better method than daisy-chaining logging files?
Thanks Joan, I appreciate the advise, unfortunately the box I'm working returns: # ps -axw | grep s6-l|grep erro 83417 - Is 0:00.01 s6-log n14 r7000 s10 S300 n14 -.* +fatal: 2 -.* +^STAT =/var/log/httpd/error/status f !/usr/bi # ktrace -f /tmp/s-log.txt -p 83417 ktrace: /tmp/s-log.txt: Function not implemented Its a preproduction box, everything optimised and stripped (no debug symbols). I've worked with nullfs since 2004, probably a little delicate then, but I've used extensively on customer sites and its proven to be ok. :) The nullfs component is where the files are piped through, and not the end-point destination which is ufs2 on an SSD. Regards, Dewayne.
Re: A better method than daisy-chaining logging files?
Sure. I don't think the permissions are particularly weird? ;) Remember we're effectively talking about two VM's one running apache and the other being a log recipient, so priv's aren't a big deal in this latter's context. On the logger, the files, as requested are: # ls -lrth /var/log/httpd | grep error ; ls -lrth /var/log/httpd/error drwx-- 2 mylogger www 512B Jun 18 15:06 error/ total 44 -rw-r--r-- 1 mylogger www 0B Jun 18 15:06 state -rw-r--r-- 1 mylogger www 0B Jun 18 15:06 lock -rw-r--r-- 1 mylogger www41K Jun 18 16:04 current When I send s6-svc -a /run/scan/apache24-error-log the processor does its job correctly. And while the systems are all running, and simply remove mylogger from the www group, then sending an alarm to the service works correctly. -rw-r--r-- 1 mylogger www 0B Jun 18 15:06 lock -rwxr--r-- 1 mylogger www 2.7K Jun 18 16:59 @40005d088c11012cc9f4.s* -rw-r--r-- 1 mylogger www 0B Jun 18 17:03 state -rw-r--r-- 1 mylogger www 0B Jun 18 17:03 current -rwxr--r-- 1 mylogger www64B Jun 18 17:03 @40005d088cd6113d5a5c.s* However when I remove mylogger from the www group and restart (into a relatively pristine test environment), it all works well but we return to the original problem: # s6-svc -a /run/scan/apache24-error-log # lh /var/log/httpd | grep error ; lh /var/log/httpd/error drwx-- 2 mylogger www 512B Jun 18 17:05 error/ total 4 -rw-r--r-- 1 mylogger www 0B Jun 18 17:04 lock -rw-r--r-- 1 mylogger www 0B Jun 18 17:05 state -rwxr--r-- 1 mylogger www 304B Jun 18 17:05 processed* -rw-r--r-- 1 mylogger www 0B Jun 18 17:05 current with the resulting s6-log: warning: unable to finish processed .s to logdir /var/log/httpd/error: Operation not permitted This is on a box that lacks development tools, so tracing will take some time to sort out; sorry. :/ FreeBSD does have tweakable knobs to prevent seeing other uids or gids which were enabled, but disabling made no difference (I thought we were onto something for a minute there). Cheers, Dewayne
Re: A better method than daisy-chaining logging files?
Laurent, if you keep this up, I'm going to think you're mystical. Putting mylogger into the www group did fix the "problem". And not without a sigh of relief! FYI: The fifo queue permissions, which the jail sees pr---w 1 mylogger www 0B May 31 13:27 apache24-error| The final log repository, in the logging jail # ls -lrth /var/log/httpd/error -rw-r--r-- 1 mylogger www 0B Jun 18 07:43 state -rw-r--r-- 1 mylogger www 0B Jun 18 07:43 lock -rw-r--r-- 1 mylogger www 0B Jun 18 07:43 current -rwxr--r-- 1 mylogger www 329B Jun 18 07:43 @40005d08099e33134f4c.s* but now with www:*:80:mylogger Thank-you. Regards, Dewayne PS. I'll investigate the u:x and o:r permissions later.
A better method than daisy-chaining logging files?
I'm still working the Apache problem, and I'm using s6-log to manage the logs. I just attempted to link an apache24 instance to its log files via a bundle, which isn't acceptable to s6-rc-compile. The approach attempted was to chain: 1. apache24 (longrun) and is a producer-for apache24-log 2. apache24-log (bundle) is a consumer-for apache24, with contents, the following, two longrun's for logging) 3. apache24-access-log (longrun) & apache24-error-log (longrun) Is it envisaged that s6-rc would enable something like this in the future, or will the following method remain: 1. apache24 (longrun) and is a producer-for apache24-access-log -. apache24-log (bundle) [ only for admin, though largely redundant ] 2. apache24-access-log (longrun) consumer-for apache24, producer-for apache24-error-log 3. apache24-error-log (longrun) consumer-for apache24-access-log The link between items 2 and 3 is fictional as is the absence of a connection between 1 and 3. Ideally having: apache24 as producer-for (both) apache24-access-log and apache24-error-log might be another option as it reflects reality. But this also isn't acceptable to the s6-rc-compile. This is a very simplified example as I have 6 sites to manage, and its seems wrong to complicate the setup with artificial links with s6-rc? I'm very interested to understand the reasoning. Kind regards, Dewayne
Re: s6-log problem with +regex
Thank-you Guillermo & Laurent. I appreciate the detail, being a virgin to daemontools this is a steep learning curve as I'm trying to ween off monit. The solution works nicely (& as intended) when using the workaround regex: redirfd -r 0 /tmp/af /usr/local/bin/s6-log n3 -.* +^a /tmp/a-only -.* +^b /tmp/b-only -.* +^c /tmp/c-only -.* +^\\\[ /tmp/date-only f /tmp/default However without any control directive, the result is: s6-log: usage: s6-log [ -d notif ] [ -q | -v ] [ -b ] [ -p ] [ -t ] [ -e ] [ -l linelimit ] logging_script Though running s6-log without a control directive is probably a little silly, perhaps the requirement to have one may be worthwhile mentioning in the doc. Aside: I had orginally placed ErrorLog "|/usr/local/bin/s6-log -b n32 s5 S700 /var/log/httpd-error T !'/usr/bin/xz -7q' /var/log/httpd-error" into apache24 which worked well in testing (one httpd), but of course in production there are lots of httpd that do NOT use the parent for logging errors, so locking is a problem. Because I have three websites (3x error files, 3x access files) I was looking at using 6 pipelines into two s6-log processes and regex's to route the content. (hence my original example). Is this a good use of resources or better to pipeline (funnel) to their own s6-log? Kind regards, Dewayne.
s6-log problem with +regex
Thank-you for s6-rc and friends. I came across two items: 1. the s6-log in testing requires a control directive. In my testing without a T, t, n $VALUE for example, the s6-log command failed. 2. The regular expression preceded by a + unintentionally matches everything, which was surprising as I intended to maintain separate logs which emanated from the same process (apache24) So to testing. My test comprised an input /tmp/af with the intention of four separate s6-log managed logdirs. The logdirs' structure was correctly created, however three contained the same content, and the default contained nothing. So the reduced form of testing looks like this: # cat /tmp/af # a file of text to be processed a line of text not selected b nother line of text c more lines [date] with line of text The intention is to create a separate log based on a regex of the first character. Unfortunately the selection mechanism failed and the entire content of the test file (/tmp/af) was duplicated, ie unselectively. The following places the content of /tmp/af into /tmp/date-only. #!/usr/local/bin/execlineb -P ; # The testing script redirfd -r 0 /tmp/af /usr/local/bin/s6-log -b n3 +^\\\[ /tmp/date-only f s100 S1500 T /tmp/default The following works correctly, in that date-only does not contain the [date] line. It does contain the rest of /tmp/af, while /tmp/default does contain only the date (input line). # cat s6-test.sh #!/usr/local/bin/execlineb -P redirfd -r 0 /tmp/af /usr/local/bin/s6-log -b n3 -^\\\[ /tmp/date-only f s100 S1500 T /tmp/default Incidentally using the s6-log command /usr/local/bin/s6-log -b n3 +^\\\[ /tmp/date-only s10 S150 T +^b /tmp/b-only f s100 S1500 T /tmp/default results in both /tmp/date-only/current and /tmp/b-only/current containing all of /tmp/af, /tmp/default had the expected structure but empty current file. The platform is amd64 FreeBSD 11.2Stable built May 6, clang v7.0.0 and clang 8.0.0 PS I haven't made sense of the hyphen in the example, after "E500 - " on page https://www.skarnet.org/software/s6/s6-log.html. Testing resulted in s6-log: fatal: unrecognized directive: -
