On Tue, May 20, 2025 at 10:13:32AM +0200, Etienne Doms wrote: > Hi, > > We're developing an embedded application which is run through a > systemd service, and we use sd-journal for logging. > > We just figured out that something is not scaling up: we log at INFO > level things like "user pushed that button so we did that" (which > occurs once in a while) and at DEBUG level "received that frame from > /dev/ttyUSB42" which occurs every 40ms on various serial lines. There > is no I/O issue since we use Storage=volatile, but the journal is > rotating way too fast. > > At first sight, it seems we just need to dynamically enable/disable > DEBUG messages but the problem is that we can't: in the wild, the > operator of our system can press a "something went wrong" button which > basically does a journal export that we can inspect later, and > obviously we can't predict when a problem is about to happen... > > So basically, we need to be able at any time to persist one hour of > INFO messages and the last five minutes of DEBUG messages. From what I > understood, the retention strategy is global to a systemd-journald > instance, I read about sd-journal namespaces, and my first intent was > to log INFO stuff in the system journal and DEBUG stuff in dedicated > namespaces (maybe one per serial lines), so I've been looking for some > sd_journal_print_with_namespace() or sd_journal_sendv([MESSAGE="foo", > NAMESPACE="bar"])... > > Now, two questions: > > * What I'm looking for does not exist, a systemd unit is bound to a > single journal namespace, and it's actually the .service that defines > to which namespace the application logs. Am I right? To achieve what I > want, I need to split my application into several services? > * Is systemd-journald designed so that it can record hundreds (if not > thousands...) of records per second? > > $COLLEAGUE proposed to switch to rsyslog, but I'd like your reading about > this. >
In your shoes I would agree with your colleague. Unless you have a good reason to need journald-specific features, in an embedded situation you're likely better served by a dead simple sysklogd like busybox provides. It's smaller in both code and execution footprint, easier to reason about, and less likely to result in lost logs. There are _still_ problems when it comes to journald/journalctl reliably associating metadata with the logged data. So when your team starts coming to depend on the features listed on the box, like `journalctl -u $unit`, then assuming all the related logs will be found using such filters, you'll be sorely disappointed. If journalctl drops messages because the metadata isn't there, the logs are effectively lost. None of these problems exist when the on-device logging is little more than a pipe redirected to an O_APPEND file in /var/log. Get the files off the device in that form, and do whatever you want in post. It's not perfect, but neither is journald. At least this way you minimize loss potential, minimize on-device load, and avoid driving everyone insane debugging problems with lossy logs. - Vito
