Vincent de RIBOU:
Is it possible to perform the same mechanism but having concatenated
file along all the others at all times ?
There's always tail with the -F option on a list of "current" files.
But you're then limited to just the "current" log data, it's tricky to
track which output comes from which log, and it's tricky to track over
missed log rotations when tail wasn't running.
One way to track things better when one knows that one is specifically
looking at a TAI64N log directory (as produced by
cyclog/multilog/s6-log) is exemplified by Russ Allbery's multilog-watch
(https://www.eyrie.org/~eagle/software/multilog-watch/) and my
export-to-rsyslog
(http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/).
It is the idea of log cursors. multilog-watch maintains a single cursor
because it only watches one log directory. export-to-rsyslog has a suite
of maintained log cursors which it holds in a cursor directory, because
it can watch multiple log directories. A cursor is roughly the TAI64N
stamp of the last log entry to be processed. (Read the export-to-rsyslog
manual page for details.)
Watching a TAI64N log directory with cursors has the advantage over some
other mechanisms that it entirely de-couples the log watching from the
logging daemon. There's no back pressure on the running service (back
down a pipeline) if the log watcher is very behind. One can have
multiple quite independent log watchers with their own sets of cursors.
One can run the log watcher when the logging daemon isn't running, and
vice versa. The caveats of (say) making sure that the reading end of a
FIFO never goes away never arise.
multilog-watch mails logs, which isn't really what you want.
export-to-rsyslog writes log entries in RFC 5424 form to a (datagram)
socket. Its major usage is providing near real time export of multiple
TAI64N log directories to something like logstash, with the individual
log cursor names encoded in the RFC 5424 data. You are not required to
point it at logstash, though, and can point it at something simpler.
However, RFC 5424 is still a lossy protocol, not least because it
doesn't have full TAI64N resolution. So pointing this at an RFC 5424
receiver is to introduce codec loss. Nonetheless, it's one way to
obtain a centralized combined log, in a properly size limited and
rotated TAI64N log directory no less, that you can manipulate with all
of the log directory tools that you are used to.
For those interested despite the aforementioned, the nosh service
bundles package has pre-made service bundles for an RFC 5424 server over
either a local datagram socket or a RFC 5426 UDP socket. The
local-syslog-read service bundle is an instance of
local-datagram-socket-listen opening /run/log (/dev/log on Linux
operating systems) and chaining to syslog-read. The udp-syslog-read
service bundle is an instance of udp-socket-listen binding to the
well-known "syslog" UDP port and chaining to syslog-read. syslog-read
itself is a UCSPI-style tool that reads RFC 5424 from the already-opened
socket and writes it to standard log suitable for piping into
cyclog/multilog/s6-log. There are accompanying cyclog@local-syslog-read
and cyclog@udp-syslog-read service bundles for that very thing.
There's no reason that someone cannot take the idea of log cursors and
make a tool like export-to-rsyslog but with a non RFC 5424 back-end, though.
