Re: [systemd-devel] Fastest way to dump last X Mo of logs from the journal ?

2024-04-24 Thread Barry Scott


> On 24 Apr 2024, at 19:48, Etienne Champetier  
> wrote:
> 
> Anyone have other ideas to do fast exports without having all logs in
> memory or twice on disk ?

Maybe base on time not size? Then you can use the --since option.

Barry



[systemd-devel] Fastest way to dump last X Mo of logs from the journal ?

2024-04-24 Thread Etienne Champetier
Hi all,

sos report includes the last X Mo of logs, sometimes filtered, sometimes not
right now it's doing the equivalent of "journalctl | tail -cXm", which
reads / format all logs, which can be extremely slow

The fastest way I found so far is:
journalctl --reverse | head -c Xm | tac
This still has the drawback of having all logs in memory, or if using
a temporary file, needing 2*X of disk space.

I've tried to play with journalctl cursor to find the start and then
output starting from the cursor

1) this doesn't work / CURSOR is only created when using -n
journalctl --reverse --cursor-file=CURSOR | head -c Xm > /dev/null

2) this ends up being ~2 times slower than just using reverse | head | tac
```
#!/bin/bash
cursor=$(mktemp cursor.XX)
logsize=0
while [ "$logsize" -lt 104857600 ]
do
  prevcursor="$(<$cursor)"
  ((logsize+=$(journalctl --reverse --cursor-file=$cursor -n 1000 | wc -c)))
  [ "$prevcursor" == "$(<$cursor)" ] && break
done
journalctl --cursor-file=$cursor
rm -f cursor
```

Anyone have other ideas to do fast exports without having all logs in
memory or twice on disk ?

sos report ticket: https://github.com/sosreport/sos/issues/3615