Re: [systemd-devel] Journald retaining logs for only 10 days
On Mo, 16.11.20 20:29, Andrei Borzenkov (arvidj...@gmail.com) wrote: > > Hu? As far as I can tell, I told it to keep 300 MB and it's using 320 > > MB. > > For all I can tell, SystemMaxUse applies to inactive journal files only > - this sets target how many archived files to delete. systemd will > switch to new file and archive current if currently active file exceeds > max size. Max journal file size is by default set to 1/8th of max > allowed space. Which means with SystemMaxUse=300M max journal file size > is set to 37.5MiB and at any time you may have up to 300MiB of old files > and one currently active file which may grow up to 37.5MiB. So 337.5MiB > in total. > > And my understanding is that every user's journal adds to this - so > basically you will have additionally 37.5MiB x number of active user > sessions. > > If someone more familiar with journal internals will confirm it, it > calls for documentation update. While manual page says "Also note that > only archived files are deleted to reduce the space occupied by journal > files" it does not make clear that limits effectively apply to archived > files only and active files may additionally consume up to > SystemMaxFileSize each before something triggers rotation. Could you file an issue about this on github? Note that there already is a note indicating that the use can grow above SystemMaxUse=, but it should be updated about more than SystemMaxFiles=. Quoting the man page journald.conf(5): SystemMaxFiles= and RuntimeMaxFiles= control how many individual journal files to keep at most. Note that only archived files are deleted to reduce the number of files until this limit is reached; active files will stay around. This means that, in effect, there might still be more journal files around in total than this limit after a vacuuming operation is complete. This setting defaults to 100. Lennart -- Lennart Poettering, Berlin ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
Am 14.11.20 um 21:29 schrieb Vito Caputo: [Journal] SystemMaxUse=300M One thing to consider is journald allocates space per-file in 8MiB increments. On my laptop for example, there are 27 user journals, 8MiB each, where the last object offset is around 2MiB. This alone burns ~162MiB in allocated but unused space. We should probably have some lower level tooling for scrutinizing the journal files and reporting how much of the space is actually used vs. fallocated. yaeh, i own "full blown" setups using 500 MB for the OS and the complete workload. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
14.11.2020 22:18, Nikolaus Rath пишет: > On Nov 14 2020, Andrei Borzenkov wrote: >> 14.11.2020 14:32, Nikolaus Rath пишет: >> ... > > # grep -vE '^#' /etc/systemd/journald.conf > > [Journal] > SystemMaxUse=300M The number shown by disk usage (320 MB) is higher than 300 MB. Maybe also check the files in `/var/log/journal`. >>> >>> It's a bit bigger on disk too: >>> >>> # du -hs /var/log/journal >>> 321M/var/log/journal >>> >>> journalctl --verify does not find any errors. >>> >>> >>> Could that be related to the short retention, or is this an unrelated >>> problem? >>> >> >> It is not a "problem". You told journald to keep 300M of data and it >> does exactly that. > > Hu? As far as I can tell, I told it to keep 300 MB and it's using 320 > MB. For all I can tell, SystemMaxUse applies to inactive journal files only - this sets target how many archived files to delete. systemd will switch to new file and archive current if currently active file exceeds max size. Max journal file size is by default set to 1/8th of max allowed space. Which means with SystemMaxUse=300M max journal file size is set to 37.5MiB and at any time you may have up to 300MiB of old files and one currently active file which may grow up to 37.5MiB. So 337.5MiB in total. And my understanding is that every user's journal adds to this - so basically you will have additionally 37.5MiB x number of active user sessions. If someone more familiar with journal internals will confirm it, it calls for documentation update. While manual page says "Also note that only archived files are deleted to reduce the space occupied by journal files" it does not make clear that limits effectively apply to archived files only and active files may additionally consume up to SystemMaxFileSize each before something triggers rotation. This is actually in line with SystemMaxFiles which does not remove active files either. So setting SystemMaxFiles=0 will remove archived files but leave any active file untouched. > Furthermore, in these 320 MB it seemingly only stored 27 MB worth of > log data. > That's rather different issue. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, Nov 14, 2020 at 12:29:30PM -0800, Vito Caputo wrote: > On Sat, Nov 14, 2020 at 09:31:23AM +, Nikolaus Rath wrote: > > Hello, > > > > I just discovered that on one of my systems journald only retains log > > entries for about 10 days: > > > > # journalctl | head -1 > > -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 > > 09:28:19 UTC. -- > > > > I do not understand what could cause this, because I have no retention > > limit configured, and the logs take up way less space than I have > > reserved: > > > > # journalctl --disk-usage > > Archived and active journals take up 320.0M in the file system. > > > > # journalctl > alllogs > > # ls -lh alllogs > > -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > > > > > > Can someone help me understand where the log entries have gone? > > > > # journalctl --version > > systemd 241 (241) > > +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP > > +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN > > -PCRE2 default-hierarchy=hybrid > > > > # grep -vE '^#' /etc/systemd/journald.conf > > > > [Journal] > > SystemMaxUse=300M > > > > > > One thing to consider is journald allocates space per-file in 8MiB > increments. > > On my laptop for example, there are 27 user journals, 8MiB each, where > the last object offset is around 2MiB. This alone burns ~162MiB in > allocated but unused space. > > We should probably have some lower level tooling for scrutinizing the > journal files and reporting how much of the space is actually used vs. > fallocated. > FYI I've opened an issue proposing that journal rotate reclaim unused space since it seems it should be trivial, at least in scenarios where the rotate isn't in response to corruption. https://github.com/systemd/systemd/issues/17613 Regards, Vito Caputo ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, Nov 14, 2020 at 09:31:23AM +, Nikolaus Rath wrote: > Hello, > > I just discovered that on one of my systems journald only retains log > entries for about 10 days: > > # journalctl | head -1 > -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 09:28:19 > UTC. -- > > I do not understand what could cause this, because I have no retention > limit configured, and the logs take up way less space than I have > reserved: > > # journalctl --disk-usage > Archived and active journals take up 320.0M in the file system. > > # journalctl > alllogs > # ls -lh alllogs > -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > > > Can someone help me understand where the log entries have gone? > > # journalctl --version > systemd 241 (241) > +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP > +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN > -PCRE2 default-hierarchy=hybrid > > # grep -vE '^#' /etc/systemd/journald.conf > > [Journal] > SystemMaxUse=300M > > One thing to consider is journald allocates space per-file in 8MiB increments. On my laptop for example, there are 27 user journals, 8MiB each, where the last object offset is around 2MiB. This alone burns ~162MiB in allocated but unused space. We should probably have some lower level tooling for scrutinizing the journal files and reporting how much of the space is actually used vs. fallocated. Regards, Vito Caputo ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, Nov 14, 2020, 20:17 Mantas Mikulėnas wrote: > On Sat, Nov 14, 2020 at 11:31 AM Nikolaus Rath wrote: > >> Hello, >> >> I just discovered that on one of my systems journald only retains log >> entries for about 10 days: >> >> # journalctl | head -1 >> -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 >> 09:28:19 UTC. -- >> >> I do not understand what could cause this, because I have no retention >> limit configured, and the logs take up way less space than I have >> reserved: >> >> # journalctl --disk-usage >> Archived and active journals take up 320.0M in the file system. >> >> # journalctl > alllogs >> # ls -lh alllogs >> -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs >> > > That just shows the 'MESSAGE' field -- it does not show any other fields > that each entry will have stored, such as the unit name which generated the > message; the program's command line; and apparently even the original > unparsed packet that was received through /dev/log. Try `journalctl -o > export` to get a closer idea of what the messages in systemd-journal look > like. > > For example, on one of my servers, a plain `journalctl -a` outputs 260 MB > of data, but `journalctl -o export` is 1.9 GB. (Which is still not quite > the same as 2.4 GB of *.journal files, but there's always going to be some > discrepancy due to how a binary database allocates space.) > One specific reason is that journal files are indexed – you can search them by any field value, without needing to do a full linear grep. (This is how "systemctl status" shows just one unit's logs, for example.) The indexes are stored along with the data, so the .journal files will be larger than the corresponding "-o export" dump. > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Nov 14 2020, Uoti Urpala wrote: > On Sat, 2020-11-14 at 09:31 +, Nikolaus Rath wrote: >> # journalctl --disk-usage >> Archived and active journals take up 320.0M in the file system. >> >> # journalctl > alllogs >> # ls -lh alllogs >> -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > > The journal stores a lot of metadata for each log entry, so the > "alllogs" size is not a good indicator of disk space requirements. > Use "journalctl -o verbose" to see all information that is actually > stored. # journalctl -o verbose -a > alllogs # ls -lh alllogs -rw-r--r-- 1 root root 223M Nov 14 19:19 alllogs Wow, that is a lot of overhead. However, this still isn't 320 MB, and it doesn't take into account that journalctl is storing the data compressed, and much of the output (the field names) most likely isn't stored in this form. I remain unconvinced... Could someone else post an example of how many log entries they get per MB of journal size on their system? Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Nov 14 2020, Andrei Borzenkov wrote: > 14.11.2020 14:32, Nikolaus Rath пишет: > ... # grep -vE '^#' /etc/systemd/journald.conf [Journal] SystemMaxUse=300M >>> >>> The number shown by disk usage (320 MB) is higher than 300 MB. Maybe also >>> check the files >>> in `/var/log/journal`. >> >> It's a bit bigger on disk too: >> >> # du -hs /var/log/journal >> 321M /var/log/journal >> >> journalctl --verify does not find any errors. >> >> >> Could that be related to the short retention, or is this an unrelated >> problem? >> > > It is not a "problem". You told journald to keep 300M of data and it > does exactly that. Hu? As far as I can tell, I told it to keep 300 MB and it's using 320 MB. Furthermore, in these 320 MB it seemingly only stored 27 MB worth of log data. Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, Nov 14, 2020 at 11:31 AM Nikolaus Rath wrote: > Hello, > > I just discovered that on one of my systems journald only retains log > entries for about 10 days: > > # journalctl | head -1 > -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 > 09:28:19 UTC. -- > > I do not understand what could cause this, because I have no retention > limit configured, and the logs take up way less space than I have > reserved: > > # journalctl --disk-usage > Archived and active journals take up 320.0M in the file system. > > # journalctl > alllogs > # ls -lh alllogs > -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > That just shows the 'MESSAGE' field -- it does not show any other fields that each entry will have stored, such as the unit name which generated the message; the program's command line; and apparently even the original unparsed packet that was received through /dev/log. Try `journalctl -o export` to get a closer idea of what the messages in systemd-journal look like. For example, on one of my servers, a plain `journalctl -a` outputs 260 MB of data, but `journalctl -o export` is 1.9 GB. (Which is still not quite the same as 2.4 GB of *.journal files, but there's always going to be some discrepancy due to how a binary database allocates space.) -- Mantas Mikulėnas ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, 2020-11-14 at 09:31 +, Nikolaus Rath wrote: > # journalctl --disk-usage > Archived and active journals take up 320.0M in the file system. > > # journalctl > alllogs > # ls -lh alllogs > -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs The journal stores a lot of metadata for each log entry, so the "alllogs" size is not a good indicator of disk space requirements. Use "journalctl -o verbose" to see all information that is actually stored. So basically I believe this is (at least mostly) a case of the configured space not being enough to store more logs, given all the metadata space requirements. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
14.11.2020 14:32, Nikolaus Rath пишет: ... >>> >>> # grep -vE '^#' /etc/systemd/journald.conf >>> >>> [Journal] >>> SystemMaxUse=300M >> >> The number shown by disk usage (320 MB) is higher than 300 MB. Maybe also >> check the files >> in `/var/log/journal`. > > It's a bit bigger on disk too: > > # du -hs /var/log/journal > 321M /var/log/journal > > journalctl --verify does not find any errors. > > > Could that be related to the short retention, or is this an unrelated problem? > It is not a "problem". You told journald to keep 300M of data and it does exactly that. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Nov 14 2020, Paul Menzel wrote: >> I just discovered that on one of my systems journald only retains log >> entries for about 10 days: >> >> # journalctl | head -1 >> -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 09:28:19 >> UTC. -- >> >> I do not understand what could cause this, because I have no retention >> limit configured, and the logs take up way less space than I have >> reserved: >> >> # journalctl --disk-usage >> Archived and active journals take up 320.0M in the file system. >> >> # journalctl > alllogs >> # ls -lh alllogs >> -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > > What size does adding the switch `-a` result in? From journalctl(1): No significant difference: # journalctl -a > alllogs # ls -lh alllogs -rw-r--r-- 1 root root 28M Nov 14 11:31 alllogs >> Can someone help me understand where the log entries have gone? >> >> # journalctl --version >> systemd 241 (241) >> +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP >> +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN >> -PCRE2 default-hierarchy=hybrid >> >> # grep -vE '^#' /etc/systemd/journald.conf >> >> [Journal] >> SystemMaxUse=300M > > The number shown by disk usage (320 MB) is higher than 300 MB. Maybe also > check the files > in `/var/log/journal`. It's a bit bigger on disk too: # du -hs /var/log/journal 321M/var/log/journal journalctl --verify does not find any errors. Could that be related to the short retention, or is this an unrelated problem? Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
Dear Nikolaus, Am 14.11.20 um 10:31 schrieb Nikolaus Rath: I just discovered that on one of my systems journald only retains log entries for about 10 days: # journalctl | head -1 -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 09:28:19 UTC. -- I do not understand what could cause this, because I have no retention limit configured, and the logs take up way less space than I have reserved: # journalctl --disk-usage Archived and active journals take up 320.0M in the file system. # journalctl > alllogs # ls -lh alllogs -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs What size does adding the switch `-a` result in? From journalctl(1): -a, --all Show all fields in full, even if they include unprintable characters or are very long. By default, fields with unprintable characters are abbreviated as "blob data". (Note that the pager may escape unprintable characters again.) Can someone help me understand where the log entries have gone? # journalctl --version systemd 241 (241) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid # grep -vE '^#' /etc/systemd/journald.conf [Journal] SystemMaxUse=300M The number shown by disk usage (320 MB) is higher than 300 MB. Maybe also check the files in `/var/log/journal`. Kind regards, Paul ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Journald retaining logs for only 10 days
Hello, I just discovered that on one of my systems journald only retains log entries for about 10 days: # journalctl | head -1 -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 09:28:19 UTC. -- I do not understand what could cause this, because I have no retention limit configured, and the logs take up way less space than I have reserved: # journalctl --disk-usage Archived and active journals take up 320.0M in the file system. # journalctl > alllogs # ls -lh alllogs -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs Can someone help me understand where the log entries have gone? # journalctl --version systemd 241 (241) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid # grep -vE '^#' /etc/systemd/journald.conf [Journal] SystemMaxUse=300M Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
