By embedded I was pointing more the idea that we can't access the device easily, CPU and memory usage is not really an issue.
However you're right that sd-journal also has this rate limit feature that already bit us, I get your point of people going crazy with partial logs. Thank you for your reply. Le mar. 20 mai 2025 à 16:22, Vito Caputo <vcap...@pengaru.com> a écrit : > > On Tue, May 20, 2025 at 10:13:32AM +0200, Etienne Doms wrote: > > Hi, > > > > We're developing an embedded application which is run through a > > systemd service, and we use sd-journal for logging. > > > > We just figured out that something is not scaling up: we log at INFO > > level things like "user pushed that button so we did that" (which > > occurs once in a while) and at DEBUG level "received that frame from > > /dev/ttyUSB42" which occurs every 40ms on various serial lines. There > > is no I/O issue since we use Storage=volatile, but the journal is > > rotating way too fast. > > > > At first sight, it seems we just need to dynamically enable/disable > > DEBUG messages but the problem is that we can't: in the wild, the > > operator of our system can press a "something went wrong" button which > > basically does a journal export that we can inspect later, and > > obviously we can't predict when a problem is about to happen... > > > > So basically, we need to be able at any time to persist one hour of > > INFO messages and the last five minutes of DEBUG messages. From what I > > understood, the retention strategy is global to a systemd-journald > > instance, I read about sd-journal namespaces, and my first intent was > > to log INFO stuff in the system journal and DEBUG stuff in dedicated > > namespaces (maybe one per serial lines), so I've been looking for some > > sd_journal_print_with_namespace() or sd_journal_sendv([MESSAGE="foo", > > NAMESPACE="bar"])... > > > > Now, two questions: > > > > * What I'm looking for does not exist, a systemd unit is bound to a > > single journal namespace, and it's actually the .service that defines > > to which namespace the application logs. Am I right? To achieve what I > > want, I need to split my application into several services? > > * Is systemd-journald designed so that it can record hundreds (if not > > thousands...) of records per second? > > > > $COLLEAGUE proposed to switch to rsyslog, but I'd like your reading about > > this. > > > > In your shoes I would agree with your colleague. Unless you have a good > reason to need journald-specific features, in an embedded situation > you're likely better served by a dead simple sysklogd like busybox > provides. > > It's smaller in both code and execution footprint, easier to reason > about, and less likely to result in lost logs. > > There are _still_ problems when it comes to journald/journalctl reliably > associating metadata with the logged data. So when your team starts > coming to depend on the features listed on the box, like `journalctl -u > $unit`, then assuming all the related logs will be found using such > filters, you'll be sorely disappointed. If journalctl drops messages > because the metadata isn't there, the logs are effectively lost. > > None of these problems exist when the on-device logging is little more > than a pipe redirected to an O_APPEND file in /var/log. Get the files > off the device in that form, and do whatever you want in post. It's not > perfect, but neither is journald. At least this way you minimize loss > potential, minimize on-device load, and avoid driving everyone insane > debugging problems with lossy logs. > > - Vito
