Package: logcheck
Version: 1.4.3
Severity: normal
Tags: patch
X-Debbugs-Cc: tom+deb...@ankh.fr.eu.org
Dear maintainers,
The systemd journal is checked by default, in addition to rsyslog files,
starting with logcheck version 1.4.1. But the format of timestamps are
different by default for journal (uses the old-school format: "Jul 20
11:51:03") and rsyslog >= 8.2210.0-3 (uses an RFC3339-compatible format:
"2023-07-20T11:51:03.046581+0200"). So logcheck cannot compare them
correctly.
Also, journalctl can emit multi-line logs, whereas rsyslog cannot.
The attached patch adds a new ISO_TIMESTAMPS config variable to the logcheck
script, allowing logcheck to:
- call journalctl with "-o short-iso-precise" in order to get a
nearly-RFC3339-compatible timestamp
- add the missing ':' between the timezone hours and minutes to convert the
journalctl shot-iso-precise timestamp to the exact RFC3339 timestamp
format used by rsyslog
- join continuous lines (lines starting with whitespace) from
journalctl output, as rsyslog only logs them as one line
- handle SORTUNIQ correctly even when using ISO_TIMESTAMPS
I have been using this locally since early March.
(NB in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032197 I also
suggested to make journalctl use the "correct" timestamp format; if/when
this actually gets done, the "second sed action" in my logcheck patch
can be dropped)
Regards,
Tom
-- System Information:
Debian Release: trixie/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 6.3.0-2-amd64 (SMP w/8 CPU threads; PREEMPT)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages logcheck depends on:
ii adduser 3.137
ii cron [cron-daemon] 3.0pl1-162
ii lockfile-progs 0.1.19
ii logtail 1.4.3
ii mime-construct 1.12+really1.11-1
ii postfix [mail-transport-agent] 3.8.1-2
Versions of packages logcheck recommends:
ii logcheck-database 1.4.3
Versions of packages logcheck suggests:
ii rsyslog [system-log-daemon] 8.2306.0-2
--
Thomas Parmelan
--- logcheck.1.4.3 2023-07-20 11:03:01.945539427 +0200
+++ logcheck 2023-07-20 11:02:09.184636208 +0200
@@ -90,6 +90,7 @@
REBOOT=0
FQDN=0
SORTUNIQ=0
+ISO_TIMESTAMPS=1
SUPPORT_CRACKING_IGNORE=0
SYSLOGSUMMARY=0
LOCKDIR=/run/lock/logcheck
@@ -480,6 +481,10 @@
local JOURNALCTL_OPTS=() OPTS=()
local offsetfile offsettime
+ if [ "$ISO_TIMESTAMPS" -eq 1 ]; then
+ JOURNALCTL_OPTS+=("-o" "short-iso-precise")
+ fi
+
# There are some problems with this section.
debug "logoutput called with file: $file"
if [ -f "$file" ]; then
@@ -508,7 +513,13 @@
offsettime="--since=-5h"
fi
debug "Running $JOURNALCTL ${JOURNALCTL_OPTS[*]} -q $offsettime"
+ # first sed action: join lines starting with whitespace
+ # (source: https://www.gnu.org/software/sed/manual/html_node/Joining-lines.html)
+ # second sed action: convert ISO8601 timestamps (as generated by journalctl -o short-iso-precise)
+ # to RFC3339 timestamps (as generated by rsyslog)
"$JOURNALCTL" "${JOURNALCTL_OPTS[@]}" --quiet "$offsettime" \
+ | sed -E ':a ; $!N ; s/\n\s+/ / ; ta ; P ; D' \
+ | sed -E 's/^([0-9T:.+-]{29})/\1:/' \
>> "$TMPDIR/logoutput/$file" 2>&1 \
|| error "Could not run journalctl or save output"
touch "$offsetfile"
@@ -602,13 +613,6 @@
INTRO=1
fi
-# Use sort -u or -k 1,3 -s
-if [ "$SORTUNIQ" -eq 1 ];then
- SORT="sort -u"
-else
- SORT="sort -k 1,3 -s"
-fi
-
# Set the mime encoding if required (blank means auto-detected)
if [ -n "$MIMEENCODING" ]; then
ENCODING=(--encoding "$MIMEENCODING")
@@ -780,8 +784,20 @@
# First sort the logs to remove duplicate lines (including from different logfiles with
# the same lines) to reduce CPU and memory usage.
debug "Sorting logs"
-$SORT "$TMPDIR/logoutput"/* | sed -e 's/[[:space:]]\+$//' > "$TMPDIR/logoutput-sorted" \
+# Use "sort -u" or "sort ... | uniq"
+if [ "$SORTUNIQ" -eq 1 ];then
+ SORT_OPTS="-u"
+ sort $SORT_OPTS "$TMPDIR/logoutput"/* | sed -e 's/[[:space:]]\+$//' > "$TMPDIR/logoutput-sorted" \
+ || error "Could not save sorted log content to $TMPDIR/logoutput-sorted"
+else
+ if [ "$ISO_TIMESTAMPS" -eq 1 ]; then
+ SORT_OPTS="-k 1 -s"
+ else
+ SORT_OPTS="-k 1,3 -s"
+ fi
+ sort $SORT_OPTS "$TMPDIR/logoutput"/* | sed -e 's/[[:space:]]\+$//' | uniq > "$TMPDIR/logoutput-sorted" \
|| error "Could not save sorted log content to $TMPDIR/logoutput-sorted"
+fi
debug "After sorting, we have the following log entries to check" "$TMPDIR/logoutput-sorted"
# See if the logoutput-sorted file actually has data to check,
