Re: Understanding the syslogd-linux Service Script
On Tue, Sep 08, 2020 at 12:53:37PM -0400, Scott Colby wrote: > Hello, > > I am faced with running a program in a container that will only log > to syslog and cannot be configured otherwise. I am looking to using > s6 within the container to supervise this program and some > implementation of syslog. I thought that there must be something > simpler than rsyslog or syslog-ng, and my investigations led me to > the s6/examples/syslogd-linux service directory. > > I am only slightly experienced with writing execline scripts and > would like to better understand exactly what each line in the example > run script is doing. Here it is, annotated with my understanding > and questions. > Responses inline. > > #!/command/execlineb -P > # Redirects stderr to stdout, but why is this necessary? > fdmove -c 2 1 This is necessary because by default only stdout goes to the appendant logger managed by s6-supervise. stderr is forwarded up to s6-svscan (and the catch-all logger). It is also needed to prepare for stuff later on in the script. Semanticaly this is actually pointing fd2 at the target of fd1. > # Clears the environment, I assume for general > # security/isolation/cleanliness reasons? > exec -c Yup. Mostly cleanup, a little security. > # Prepares for setting uid/gid later > s6-envuidgid nobody > # Redirects stdout to fd 3, I think because s6-ipcserver closes fd > # 1; what happens to things sent to fd 3? > # Also, why is the -c option not used here? > fdmove 1 3 Welcome to Being Confused By Shell Redirection (don't worry, it happens to everyone). fdmove 1 3 does not redirecct stdout to fd3, it points fd1 to fd3's target. Omitting the -c means that fd3 is closed after we rewrite fd1's target. We close fd3 after this point because we don't need it anymore. > # Listens on /dev/log, this makes sense to me > s6-ipcserver -U -1 -- /dev/log > # Redirects stdout to stderr, because this is where log messages > # are expected to go > fdmove -c 1 2 s6-ipcserver -1 signals readyness on fd 1 and then closes it. This re-opens fd1 (pointed at fd2's target, which was fd1's original target at the start of the script) before launching ucspilogd log handlers. This is needed because ucspilogd reads syslog type messages on stdin (fd0) and writes the processed output on stdout (fd1). Note that this happens individually for every new message flow established by s6-ipcserver and not within the main control flow of the program, though this doesn't matter here since the last thing the main program does is establish an s6-ipcserver listener on /dev/log. > # writes stdin to stdout with the values of the remote UID and GID > # prepended, plus whatever other functionality of ucspidlogd > ucspilogd IPCREMOTEEUID IPCREMOTEEGID > > Please let me know if I have made any mistakes in my annotation and > what the answers to my questions are. > The fully annotated flow of the script is: start: fd1 open, pointed at s6-log; fd2 open, pointed at catch-all; fd3 open, pointed at s6-supervise readyness reader fdmove -c 2 1: fd1 open, pointed at s6-log; fd2 open, pointed at s6-log; fd3 open, pointed at s6-supervise readyness reader fdmove 1 3: fd1 open, pointed at s6-supervise readyness reader; fd2 open, pointed at s6-log; fd3 closed s6-ipcserver -U -1 -- /dev/log fd1 signaled ready and then closed; fd2 open, pointed at s6-log; fd3 closed --- for each new sender attached to /dev/log --- fd0 open, reading from /dev/log, fd1 open, writing to s6-log, fd2 open, writing to s6-log, fd3 closed. Cheers! -- Colin Booth
Re: Understanding the syslogd-linux Service Script
On Tue, Sep 08, 2020 at 12:53:37PM -0400, Scott Colby wrote: > # Listens on /dev/log, this makes sense to me > s6-ipcserver -U -1 -- /dev/log > One thing that I forgot to mention here is that depending on the age of the libc this might not work properly as written. Newer libc's use datagram connections for syslog and because of that you need to hand-write the entire s6-ipcserver chain (since the s6-ipcserver entrypoint doesn't currently support passing the SOCK_DGRAM option to s6-ipcserver-socketbinder. If you need to write it by hand, you should change that line to: s6-ipcserver-socetbinder -m -b0 /dev/log s6-applyuidgid -U -z s6-ipcserverd -1 then the rest of the program (the fdmove and ucspilogd commands). -- Colin Booth
Re: Understanding the syslogd-linux Service Script
One thing that I forgot to mention here is that depending on the age of the libc this might not work properly as written. Newer libc's use datagram connections for syslog and because of that you need to hand-write the entire s6-ipcserver chain (since the s6-ipcserver entrypoint doesn't currently support passing the SOCK_DGRAM option to s6-ipcserver-socketbinder. The syslogd-linux example is a pretty old one, I should probably rewrite it. It will work as is with glibc, because glibc's syslog() has a SOCK_STREAM fallback, so the s6-ipcserver chain is fine. It will *not* work as is with musl, because musl's syslog() only opens a SOCK_DGRAM socket. In that case, I recommend using socklog instead: http://smarden.org/socklog/ and the run script would just be: "fdmove -c 2 1 fdclose 0 socklog" (you lose readiness notification but that's not a big problem for socklog, which listens to /dev/log very fast, and the worst case is the first log lines are sent to /dev/console) Ultimately the syslogd service just needs to listen to the /dev/log socket (as SOCK_DGRAM, as socklog does, or SOCK_STREAM, as s6-ipcserver does) and translate the loglevel/priority in the log lines to human- readable text and print them to stdout (as socklog or ucspilogd does). The real work is done by the logger part, which is a s6-log process that reads the translated log lines, matches them against some regexes and logs them to various logdirs depending on what they match. The s6-log script is the equivalent of syslogd.conf: that's where you configure how to dispatch the syslog into different places. -- Laurent
Re: Understanding the syslogd-linux Service Script
Dear Colin and Laurent, Thank you for the thorough explanations it took me a minute with a pen and paper to work out the fd movements, but it makes sense to me now. My particular environment uses glibc, so the old version of the example seems to work well. I omit the syslog service's log directory and the log lines appear on my container's stdout, which was my goal. I tried out Colin's SOCK_DGRAM version, but this was unsuccessful; I received this line at a very high rate (20k lines in just a second or two) on my stdout: s6-ipcserverd: warning: unable to accept: Operation not supported The s6-ipcserverd docs specifically state that it expects a bound and listening SOCK_STREAM socket, and this led me to the `ipc_accept()` call that the program makes. Out of curiosity, is there another s6-family program that handles SOCK_DGRAM sockets? Otherwise, I'll look in to socklog as Laurent suggested. Thanks, Scott
Re: Understanding the syslogd-linux Service Script
The s6-ipcserverd docs specifically state that it expects a bound and listening SOCK_STREAM socket, and this led me to the `ipc_accept()` call that the program makes. Out of curiosity, is there another s6-family program that handles SOCK_DGRAM sockets? Otherwise, I'll look in to socklog as Laurent suggested. There is not, because the UCSPI model - the one that s6-ipcserver implements - is about handling data *streams*, and forking a child to handle each separate client (as the venerable inetd does). There are workarounds and hacks to handle datagram sequences with a similar architecture, but it's never perfect: clients have to connect (which an additional restriction), you lose natural message boundaries, and you don't have an explicit EOF so the child needs to die on a timeout - which makes the thing awkward and actually *adds* complexity, when the whole point of UCSPI is to make every component as simple as possible. When you need to listen to a /dev/log datagram socket, socklog is definitely the simplest, and correct, approach. -- Laurent
Re: Understanding the syslogd-linux Service Script
My particular environment uses glibc, so the old version of the example seems to work well. I omit the syslog service's log directory and the log lines appear on my container's stdout, which was my goal. If your goal is to have syslog messages appear on your container's stdout, then you should actually be able to get away with not having any syslogd service at all, which is always the preferred solution! When syslog() can't find a server that listens on /dev/log, it writes its message to /dev/console. In a container environment, /dev/console by definition should be your container's stdout. So, not handling syslog messages in your container at all should naturally make them fall through to where you want them. -- Laurent
Re: Understanding the syslogd-linux Service Script
> There is not, because the UCSPI model - the one that s6-ipcserver implements - is about handling data *streams*, and forking a child to handle each separate client (as the venerable inetd does). That makes sense to me. > If your goal is to have syslog messages appear on your container's > stdout, then you should actually be able to get away with not having > any syslogd service at all, which is always the preferred solution! That is my goal and it's interesting that I didn't observe the logs ending up on the container's stdout. I was considering that the program I want to run (OpenDNSSEC, to be exact) does something funny that would prevent this when I came across an undocumented configuration option that allows logging to a file or stdout instead of using syslog. I have to sigh, for this is not the first time I've had to read the source of this program in order to figure out what is going on. In any case, thank you for all the explanation. I'm certain my improved understanding of both s6 and syslog will come in handy in the future. Scott
