On Tue, Mar 1, 2022 at 4:39 PM eric.zalu...@vertiv.com < eric.zalu...@vertiv.com> wrote:
> #### [ System Environment ] #### > On an embedded x86-64 Linux system, I’m running systemd v241. I have > Systemd-Journald logging set to persistent with SystemMaxUse and > RuntimeMaxUse both set to 512MB. > > The Linux system mount loops /var/log on start-up from a var-log.ext4 > file. The /var/log mount is given a fixed size of the disk (976MB). Systemd > creates a journal entry directory given a hash name in > /var/log/journal/<hash1>. > > > > #### [ System Information ] #### > > Linux 4.19.0-6-2-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 > GNU/Linux > > > #### [ Problem ] #### > > When the system firmware is upgraded, Systemd creates a new journal entry > directory with a different hash name and no longer recognizes the previous > hash directory. > > The old logs from the previous journal entry can no longer be managed. The > old logs are never rotated and cannot be manually rotated using the > journaldctl cli. The disk usage calculator by Journald does not account for > the previous journal entry meaning if there are two previous entries in > /var/log that consume 900MB of space; then the new journal entry only has > 76MB of space to work with. Eventually, disk space will be full. Journald > cannot automatically flush, vacuum, clean, rotate previous logs because it > does not recognize the previous journal entries. > > There must be a systemd journald check that occurs where it determines > these other entries are not for it to manage. > This is not a 'hash' – it's the machine ID <https://www.freedesktop.org/software/systemd/man/machine-id.html>, which is directly read from /etc/machine-id. Normally the machine ID is randomly generated during *first *boot (it's just a random UUID), and it is supposed to be persistent afterwards. It sounds like your /etc doesn't have that file and is on a read-only rootfs, so systemd generates a new machine-id in tmpfs every boot. The device probably has *some* persistent storage though, so try to find a way to make /etc/machine-id persist as well (see the machine-id(5) <https://www.freedesktop.org/software/systemd/man/machine-id.html> manual page for a few possibilities). The intent of using machine-id subdirectories is to allow containers' journal directories to be symlinked on the host, or remote system journals to be collected on a single system – there's actually the journalctl -m option to make it look at "foreign" journals, so e.g. journalctl -m -f could be used to watch logs of all containers. (The machine ID is also used as the base for systemd-networkd's DHCPv4 Client ID, DHCPv6 DUID, and IPv6 address generation. But other than that, though, I can't think of any its uses that would be visible externally – aside from desktop-specific things like pulseaudio – so in some cases it might be "good enough" to pre-define a fixed ID in the template image as a last resort.) -- Mantas Mikulėnas
