Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, Nov 14, 2020 at 12:29:30PM -0800, Vito Caputo wrote: > On Sat, Nov 14, 2020 at 09:31:23AM +, Nikolaus Rath wrote: > > Hello, > > > > I just discovered that on one of my systems journald only retains log > > entries for about 10 days: > > > > # journalctl | head -1 > > -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 > > 09:28:19 UTC. -- > > > > I do not understand what could cause this, because I have no retention > > limit configured, and the logs take up way less space than I have > > reserved: > > > > # journalctl --disk-usage > > Archived and active journals take up 320.0M in the file system. > > > > # journalctl > alllogs > > # ls -lh alllogs > > -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > > > > > > Can someone help me understand where the log entries have gone? > > > > # journalctl --version > > systemd 241 (241) > > +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP > > +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN > > -PCRE2 default-hierarchy=hybrid > > > > # grep -vE '^#' /etc/systemd/journald.conf > > > > [Journal] > > SystemMaxUse=300M > > > > > > One thing to consider is journald allocates space per-file in 8MiB > increments. > > On my laptop for example, there are 27 user journals, 8MiB each, where > the last object offset is around 2MiB. This alone burns ~162MiB in > allocated but unused space. > > We should probably have some lower level tooling for scrutinizing the > journal files and reporting how much of the space is actually used vs. > fallocated. > FYI I've opened an issue proposing that journal rotate reclaim unused space since it seems it should be trivial, at least in scenarios where the rotate isn't in response to corruption. https://github.com/systemd/systemd/issues/17613 Regards, Vito Caputo ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, Nov 14, 2020 at 09:31:23AM +, Nikolaus Rath wrote: > Hello, > > I just discovered that on one of my systems journald only retains log > entries for about 10 days: > > # journalctl | head -1 > -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 09:28:19 > UTC. -- > > I do not understand what could cause this, because I have no retention > limit configured, and the logs take up way less space than I have > reserved: > > # journalctl --disk-usage > Archived and active journals take up 320.0M in the file system. > > # journalctl > alllogs > # ls -lh alllogs > -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > > > Can someone help me understand where the log entries have gone? > > # journalctl --version > systemd 241 (241) > +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP > +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN > -PCRE2 default-hierarchy=hybrid > > # grep -vE '^#' /etc/systemd/journald.conf > > [Journal] > SystemMaxUse=300M > > One thing to consider is journald allocates space per-file in 8MiB increments. On my laptop for example, there are 27 user journals, 8MiB each, where the last object offset is around 2MiB. This alone burns ~162MiB in allocated but unused space. We should probably have some lower level tooling for scrutinizing the journal files and reporting how much of the space is actually used vs. fallocated. Regards, Vito Caputo ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, Nov 14, 2020, 20:17 Mantas Mikulėnas wrote: > On Sat, Nov 14, 2020 at 11:31 AM Nikolaus Rath wrote: > >> Hello, >> >> I just discovered that on one of my systems journald only retains log >> entries for about 10 days: >> >> # journalctl | head -1 >> -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 >> 09:28:19 UTC. -- >> >> I do not understand what could cause this, because I have no retention >> limit configured, and the logs take up way less space than I have >> reserved: >> >> # journalctl --disk-usage >> Archived and active journals take up 320.0M in the file system. >> >> # journalctl > alllogs >> # ls -lh alllogs >> -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs >> > > That just shows the 'MESSAGE' field -- it does not show any other fields > that each entry will have stored, such as the unit name which generated the > message; the program's command line; and apparently even the original > unparsed packet that was received through /dev/log. Try `journalctl -o > export` to get a closer idea of what the messages in systemd-journal look > like. > > For example, on one of my servers, a plain `journalctl -a` outputs 260 MB > of data, but `journalctl -o export` is 1.9 GB. (Which is still not quite > the same as 2.4 GB of *.journal files, but there's always going to be some > discrepancy due to how a binary database allocates space.) > One specific reason is that journal files are indexed – you can search them by any field value, without needing to do a full linear grep. (This is how "systemctl status" shows just one unit's logs, for example.) The indexes are stored along with the data, so the .journal files will be larger than the corresponding "-o export" dump. > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Nov 14 2020, Uoti Urpala wrote: > On Sat, 2020-11-14 at 09:31 +, Nikolaus Rath wrote: >> # journalctl --disk-usage >> Archived and active journals take up 320.0M in the file system. >> >> # journalctl > alllogs >> # ls -lh alllogs >> -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > > The journal stores a lot of metadata for each log entry, so the > "alllogs" size is not a good indicator of disk space requirements. > Use "journalctl -o verbose" to see all information that is actually > stored. # journalctl -o verbose -a > alllogs # ls -lh alllogs -rw-r--r-- 1 root root 223M Nov 14 19:19 alllogs Wow, that is a lot of overhead. However, this still isn't 320 MB, and it doesn't take into account that journalctl is storing the data compressed, and much of the output (the field names) most likely isn't stored in this form. I remain unconvinced... Could someone else post an example of how many log entries they get per MB of journal size on their system? Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Nov 14 2020, Andrei Borzenkov wrote: > 14.11.2020 14:32, Nikolaus Rath пишет: > ... # grep -vE '^#' /etc/systemd/journald.conf [Journal] SystemMaxUse=300M >>> >>> The number shown by disk usage (320 MB) is higher than 300 MB. Maybe also >>> check the files >>> in `/var/log/journal`. >> >> It's a bit bigger on disk too: >> >> # du -hs /var/log/journal >> 321M /var/log/journal >> >> journalctl --verify does not find any errors. >> >> >> Could that be related to the short retention, or is this an unrelated >> problem? >> > > It is not a "problem". You told journald to keep 300M of data and it > does exactly that. Hu? As far as I can tell, I told it to keep 300 MB and it's using 320 MB. Furthermore, in these 320 MB it seemingly only stored 27 MB worth of log data. Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, Nov 14, 2020 at 11:31 AM Nikolaus Rath wrote: > Hello, > > I just discovered that on one of my systems journald only retains log > entries for about 10 days: > > # journalctl | head -1 > -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 > 09:28:19 UTC. -- > > I do not understand what could cause this, because I have no retention > limit configured, and the logs take up way less space than I have > reserved: > > # journalctl --disk-usage > Archived and active journals take up 320.0M in the file system. > > # journalctl > alllogs > # ls -lh alllogs > -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > That just shows the 'MESSAGE' field -- it does not show any other fields that each entry will have stored, such as the unit name which generated the message; the program's command line; and apparently even the original unparsed packet that was received through /dev/log. Try `journalctl -o export` to get a closer idea of what the messages in systemd-journal look like. For example, on one of my servers, a plain `journalctl -a` outputs 260 MB of data, but `journalctl -o export` is 1.9 GB. (Which is still not quite the same as 2.4 GB of *.journal files, but there's always going to be some discrepancy due to how a binary database allocates space.) -- Mantas Mikulėnas ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
On Sat, 2020-11-14 at 09:31 +, Nikolaus Rath wrote: > # journalctl --disk-usage > Archived and active journals take up 320.0M in the file system. > > # journalctl > alllogs > # ls -lh alllogs > -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs The journal stores a lot of metadata for each log entry, so the "alllogs" size is not a good indicator of disk space requirements. Use "journalctl -o verbose" to see all information that is actually stored. So basically I believe this is (at least mostly) a case of the configured space not being enough to store more logs, given all the metadata space requirements. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
14.11.2020 14:32, Nikolaus Rath пишет: ... >>> >>> # grep -vE '^#' /etc/systemd/journald.conf >>> >>> [Journal] >>> SystemMaxUse=300M >> >> The number shown by disk usage (320 MB) is higher than 300 MB. Maybe also >> check the files >> in `/var/log/journal`. > > It's a bit bigger on disk too: > > # du -hs /var/log/journal > 321M /var/log/journal > > journalctl --verify does not find any errors. > > > Could that be related to the short retention, or is this an unrelated problem? > It is not a "problem". You told journald to keep 300M of data and it does exactly that. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [SPECIFICATION RFC] The firmware and bootloader log specification
Hi Daniel, I think this is a good idea. Alas, as I hear for the first time about it, I lack any context of prior discussions / context. So bear with me, if I ask things that have already been answered. On 14.11.20 00:52, Daniel Kiper wrote: > The goal is to pass all logs produced by various boot components to the > running OS. The OS kernel should expose these logs to the user space > and/or process them internally if needed. The content of these logs > should be human readable. However, they should also contain the > information which allows admins to do e.g. boot time analysis. > > The log specification should be as much as possible platform agnostic > and self contained. The final version of this spec should be merged into > existing specifications, e.g. UEFI, ACPI, Multiboot2, or be a standalone > spec, e.g. as a part of OASIS Standards. The former seems better but is > not perfect too... > > Here is the description (pseudocode) of the structures which will be > used to store the log data. I guess using C syntax for your "pseudocode" isn't a good choice as it can confuse people and might lead to (unportable) implementations that try to copy this definition to C. IMHO, it's much better for a specifi- cation to provide exact bit/byte offsets. The protocol tool [P], for instance, can be used to draw things in ASCII. A portable C implemen- tation could then use these offsets for proper (de)serialization with- out structs that try to mimic the representation in memory. > The members of struct bf_log: > - version: the firmware and bootloader log format version number, 1 for now, > - producer: the producer/firmware/bootloader/... type; the length > allows ASCII UUID storage if somebody needs that functionality, So, is this always supposed to be a string? > - flags: it can be used to store information about log state, e.g. > it was truncated or not (does it make sense to have an information > about the number of lost messages?), Truncation is an interesting point as I see no length for the available space specified. I assume most implementations would want a field for this. Otherwise they would have to track it separately. In coreboot, we use a ring-buffer for messages as it seems more useful to keep the most recent messages, it's also extended across reboots and suspend/resume cycles. For this, it would need an additional pointer where the oldest message resides, iow. where to start reading messages. > - next_bf_log_addr: address of next bf_log struct; none if zero Do I understand this correctly that a later-stage boot component would use this to add its own `bf_log` to the chain? e.g. if I start initia- lizing hardware with coreboot and then use GRUB2 to boot, each of them would set up its own ` bf_log` and GRUB2 would set this pointer if possible? > (I think > newer spec versions should not change anything in first 5 bf_log members; > this way older log parsers will be able to traverse/copy all logs > regardless > of version used in one log or another), Good point, which brings me to another good practice regarding such data formats: A length field for the header. In this case the length from the start of `bf_log` to the start of `msgs`. This would give us backwards compatibility in case additional fields are added in the future. And would also allow the various implementation to add custom fields (not for communication with log parser but for their own use). > - next_msg_off: the offset, in bytes, from the beginning of the bf_log > struct, > of the next byte after the last log message in the msgs[]; i.e. the offset > of the next available log message slot; it is equal to the total size of > the log buffer including the bf_log struct, > - msgs: the array of log messages, > - should we add CRC or hash or signatures here? > > The members of struct bf_log_msg: > - size: total size of bf_log_msg struct, Does this include the actual message string? > - ts_nsec: timestamp expressed in nanoseconds starting from 0, But what is 0? In coreboot, we log timestamps relative to the last reset. Which, if applied to our log ring-buffer, might make things confusing because it can contain messages from multiple boots. > - level: similar to syslog meaning; can be used to differentiate normal > messages > from debug messages; the exact interpretation depends on the current > producer > type specified in the bf_log.producer, > - facility: similar to syslog meaning; can be used to differentiate the > sources of > the messages, e.g. message produced by networking module; the exact > interpretation > depends on the current producer type specified in the bf_log.producer, > - msg_off: the log message offset in strings[], > - strings[0]: the beginning of log message type, similar to the facility > member but > NUL terminated string instead of integer; this will be used by, e.g., the > GRUB2 > for messages printed using grub_
Re: [systemd-devel] Journald retaining logs for only 10 days
On Nov 14 2020, Paul Menzel wrote: >> I just discovered that on one of my systems journald only retains log >> entries for about 10 days: >> >> # journalctl | head -1 >> -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 09:28:19 >> UTC. -- >> >> I do not understand what could cause this, because I have no retention >> limit configured, and the logs take up way less space than I have >> reserved: >> >> # journalctl --disk-usage >> Archived and active journals take up 320.0M in the file system. >> >> # journalctl > alllogs >> # ls -lh alllogs >> -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs > > What size does adding the switch `-a` result in? From journalctl(1): No significant difference: # journalctl -a > alllogs # ls -lh alllogs -rw-r--r-- 1 root root 28M Nov 14 11:31 alllogs >> Can someone help me understand where the log entries have gone? >> >> # journalctl --version >> systemd 241 (241) >> +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP >> +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN >> -PCRE2 default-hierarchy=hybrid >> >> # grep -vE '^#' /etc/systemd/journald.conf >> >> [Journal] >> SystemMaxUse=300M > > The number shown by disk usage (320 MB) is higher than 300 MB. Maybe also > check the files > in `/var/log/journal`. It's a bit bigger on disk too: # du -hs /var/log/journal 321M/var/log/journal journalctl --verify does not find any errors. Could that be related to the short retention, or is this an unrelated problem? Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Journald retaining logs for only 10 days
Dear Nikolaus, Am 14.11.20 um 10:31 schrieb Nikolaus Rath: I just discovered that on one of my systems journald only retains log entries for about 10 days: # journalctl | head -1 -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 09:28:19 UTC. -- I do not understand what could cause this, because I have no retention limit configured, and the logs take up way less space than I have reserved: # journalctl --disk-usage Archived and active journals take up 320.0M in the file system. # journalctl > alllogs # ls -lh alllogs -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs What size does adding the switch `-a` result in? From journalctl(1): -a, --all Show all fields in full, even if they include unprintable characters or are very long. By default, fields with unprintable characters are abbreviated as "blob data". (Note that the pager may escape unprintable characters again.) Can someone help me understand where the log entries have gone? # journalctl --version systemd 241 (241) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid # grep -vE '^#' /etc/systemd/journald.conf [Journal] SystemMaxUse=300M The number shown by disk usage (320 MB) is higher than 300 MB. Maybe also check the files in `/var/log/journal`. Kind regards, Paul ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Journald retaining logs for only 10 days
Hello, I just discovered that on one of my systems journald only retains log entries for about 10 days: # journalctl | head -1 -- Logs begin at Wed 2020-11-04 15:57:13 UTC, end at Sat 2020-11-14 09:28:19 UTC. -- I do not understand what could cause this, because I have no retention limit configured, and the logs take up way less space than I have reserved: # journalctl --disk-usage Archived and active journals take up 320.0M in the file system. # journalctl > alllogs # ls -lh alllogs -rw-r--r-- 1 root root 27M Nov 14 09:24 alllogs Can someone help me understand where the log entries have gone? # journalctl --version systemd 241 (241) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid # grep -vE '^#' /etc/systemd/journald.conf [Journal] SystemMaxUse=300M Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.« ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] log_target=console and multiple console devices
Quoting https://freedesktop.org/wiki/Software/systemd/Debugging/ console= can be specified multiple times, systemd will output to all of them. That's wrong. systemd will write to /dev/console and according to kernel documentation (and my experience :) ) kernel will forward it only to the last listed console. From https://www.kernel.org/doc/html/latest/admin-guide/serial-console.html: You can specify multiple console= options on the kernel command line. Output will appear on all of them. The last device will be used when you open /dev/console. So *kernel* output will indeed be duplicated to all of them, but not output written to /dev/console. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Man page description for systemd.log_target & Co is completely lost.
man systemd(1): $SYSTEMD_LOG_TARGET === systemd reads the log target from this environment variable. This can be overridden with --log-target=. --log-target= = Set log target. See systemd.log_target above. systemd.log_color, systemd.log_level=, systemd.log_location, systemd.log_target=, systemd.log_time, systemd.log_tid Controls log output, with the same effect as the $SYSTEMD_LOG_COLOR, $SYSTEMD_LOG_LEVEL, $SYSTEMD_LOG_LOCATION, $SYSTEMD_LOG_TARGET, $SYSTEMD_LOG_TIME, and $SYSTEMD_LOG_TID environment variables described above. systemd.log_color, systemd.log_location, systemd.log_time, and systemd.log_tid= can be specified without an argument, with the same effect as a positive boolean. Oops. Déjà Vu ... Sepulka – pl: sepulki, a prominent element of the civilization of Ardrites from the planet of Enteropia; see "Sepulkaria" Sepulkaria – sing: sepulkarium, establishments used for sepuling; see "Sepuling" Sepuling – an activity of Ardrites from the planet of Enteropia; see "Sepulka" Looking in history ... commit c035f3766c7808955a7d2e42316b3f008a236fcc Author: Zbigniew Jędrzejewski-Szmek Date: Fri Nov 15 13:30:02 2019 +0100 man: significantly downgrade the Options section in systemd(1) This structure of the man page originates from the time when systemd was installed on top of sysvinit systems, and users had an actual chance to interact with the systemd binary directly. Nowadays it is almost never called directly, so let's properly explain this in the overview. The Options section is moved down below the kernel command line, those options are only needed in special circumstances. Let's refer the reader to the description of the kernel command line options, and not duplicate the descriptions (which makes the text longer than necessary and increases chances for discrepancies). Well, I do not see any reference to "kernel command line" in description above. There is cross reference to kernel-command-line(7) in See also ... let's try it. systemd.unit=, rd.systemd.unit=, systemd.dump_core, systemd.early_core_pattern=, systemd.crash_chvt, systemd.crash_shell, systemd.crash_reboot, systemd.confirm_spawn, systemd.service_watchdogs, systemd.show_status, systemd.status_unit_format=, systemd.log_target=, systemd.log_level=, systemd.log_location=, systemd.log_color, systemd.default_standard_output=, systemd.default_standard_error=, systemd.setenv=, systemd.machine_id=, systemd.unified_cgroup_hierarchy, systemd.legacy_systemd_cgroup_controller = Parameters understood by the system and service manager to control system behavior. For details, see systemd(1). We are back to square one. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
