Re: Journald's qualities

[Anssi Saari]

Mariusz Gronczewski  writes:

> ... in what way? You need to resolve DNS first before you know which
> interface the traffic is going out of.

You can specify a domain via resolvectl domain for an interface. In my
simple case I specify just ~. as the domain for my VPN and for my local
domain it's lan. So everything not going to .lan goes to VPN DNS.

Re: Journald's qualities

[Jeffrey Walton]

On Mon, Feb 26, 2024 at 10:42 AM Mariusz Gronczewski  wrote:
>
> Dnia 2024-02-26, o godz. 13:10:43
> Anssi Saari  napisał(a):
>
> > Mariusz Gronczewski  writes:
> >
> > > Offtopic but since Debian switched to systemd for DNS management on
> > > VPNs and suc I need to restart it sometimes multiple times to just
> > > get "right" DNS servers, because there appears to be no notion of
> > > priority:
> > >
> > > https://github.com/systemd/systemd/issues/27543
> > >
> > > so now any time I connect to work (just openvpn tunnel, nothing
> > > fancy) I need to spam
> > >
> > > systemclt restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf
> > >
> > > few times till the dice rolls the right order of DNS servers...
> >
> > Interesting. I leaped on systemd-networkd and -resolved when I read
> > years ago it added interface specific DNS support. So now my local DNS
> > (dnsmasq in the router) handles my home network and what goes out via
> > the VPN (i.e. tun0 or wg0 these days) uses the VPN's DNS.
>
> ... in what way? You need to resolve DNS first before you know which
> interface the traffic is going out of.

I _think_ that depends on the configuration. You can use local DNS for
name resolution, or remote (VPN) DNS for name resolution.

Sometimes both are used at the same time. I think that's called "split
DNS" or "split brain DNS."

> > Or if the
> > VPN is off, the local DNS forwards queries to DHCP assigned DNS. I
> > see no issues although I don't have the kind of VPN where some
> > external traffic goes through it only but might work for that too.
> > For me the default was that systemd-resolved dutifully spammed all
> > DNS queries to all DNS servers through all interfaces.
> >
> > This interface specific DNS was a little hard to setup as I
> > recall. Easier with WG than OpenVPN.
> >
>
> Our case is basically that:
>
> * some of the records exist only on VPN DNS server (private domains
>   pointing to private IPs)
> * some of the records exist on outside but the VPN DNS returns private
>   range IP addresses for it (so-called split-horizon DNS).
>
> So the only right way is to ask the first server on the list. That
> worked before systemd-resolved came as Debian scripts just put the
> VPN's DNS servers in the front. Now it is throw of the dice any time
> the daemon is restarted.
>
> The proper way would be either to:
>
> * ask in order, with components registering the DNS server specifying
>   that priority so the daemon can result the sorted list
> * have a way to do per-domain exception and do "if domain is
>   *.internal.example.com, ask VPN server's DNS"
>
> The second is possible in dnsmasq but not (AFAIK) in systemd. And
> currently neither "make systemd a DNS resolver" nor "use
> systemd-resolved provided DNS config" work reliably.

Jeff

Re: Journald's qualities

[Mariusz Gronczewski]

Dnia 2024-02-26, o godz. 13:10:43
Anssi Saari  napisał(a):

> Mariusz Gronczewski  writes:
> 
> > Offtopic but since Debian switched to systemd for DNS management on
> > VPNs and suc I need to restart it sometimes multiple times to just
> > get "right" DNS servers, because there appears to be no notion of
> > priority:
> >
> > https://github.com/systemd/systemd/issues/27543
> >
> > so now any time I connect to work (just openvpn tunnel, nothing
> > fancy) I need to spam
> >
> > systemclt restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf
> >
> > few times till the dice rolls the right order of DNS servers...  
> 
> Interesting. I leaped on systemd-networkd and -resolved when I read
> years ago it added interface specific DNS support. So now my local DNS
> (dnsmasq in the router) handles my home network and what goes out via
> the VPN (i.e. tun0 or wg0 these days) uses the VPN's DNS. 

... in what way? You need to resolve DNS first before you know which
interface the traffic is going out of.

> Or if the
> VPN is off, the local DNS forwards queries to DHCP assigned DNS. I
> see no issues although I don't have the kind of VPN where some
> external traffic goes through it only but might work for that too.
> For me the default was that systemd-resolved dutifully spammed all
> DNS queries to all DNS servers through all interfaces.
> 
> This interface specific DNS was a little hard to setup as I
> recall. Easier with WG than OpenVPN.
> 

Our case is basically that:

* some of the records exist only on VPN DNS server (private domains
  pointing to private IPs)
* some of the records exist on outside but the VPN DNS returns private
  range IP addresses for it (so-called split-horizon DNS). 

So the only right way is to ask the first server on the list. That
worked before systemd-resolved came as Debian scripts just put the
VPN's DNS servers in the front. Now it is throw of the dice any time
the daemon is restarted.

The proper way would be either to:

* ask in order, with components registering the DNS server specifying
  that priority so the daemon can result the sorted list
* have a way to do per-domain exception and do "if domain is
  *.internal.example.com, ask VPN server's DNS"

The second is possible in dnsmasq but not (AFAIK) in systemd. And
currently neither "make systemd a DNS resolver" nor "use
systemd-resolved provided DNS config" work reliably.

-- 
Mariusz Gronczewski (XANi) 
GnuPG: 0xEA8ACE64
https://devrandom.eu

Re: Journald's qualities

[Anssi Saari]

Mariusz Gronczewski  writes:

> Offtopic but since Debian switched to systemd for DNS management on
> VPNs and suc I need to restart it sometimes multiple times to just get
> "right" DNS servers, because there appears to be no notion of priority:
>
> https://github.com/systemd/systemd/issues/27543
>
> so now any time I connect to work (just openvpn tunnel, nothing fancy)
> I need to spam
>
> systemclt restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf
>
> few times till the dice rolls the right order of DNS servers...

Interesting. I leaped on systemd-networkd and -resolved when I read
years ago it added interface specific DNS support. So now my local DNS
(dnsmasq in the router) handles my home network and what goes out via
the VPN (i.e. tun0 or wg0 these days) uses the VPN's DNS. Or if the VPN
is off, the local DNS forwards queries to DHCP assigned DNS. I see no
issues although I don't have the kind of VPN where some external traffic
goes through it only but might work for that too. For me the default was
that systemd-resolved dutifully spammed all DNS queries to all DNS
servers through all interfaces.

This interface specific DNS was a little hard to setup as I
recall. Easier with WG than OpenVPN.

Re: Journald's qualities

[Kamil Jońca]

Mariusz Gronczewski  writes:

[...]
>
> Offtopic but since Debian switched to systemd for DNS management on
> VPNs and suc I need to restart it sometimes multiple times to just get
> "right" DNS servers, because there appears to be no notion of priority:

Well, I am using  openresolv (earlier resolvconf) for that + dnsmasq, as
a forwarder. 

I have turned off systemd resolver.
Similary I do not like system-networkd (I am using ifupdown) - it also
pretend to be more wise than me.

KJ

Re: Journald's qualities (was: Selective rotation of journald logs)

[Mariusz Gronczewski]

Dnia 2024-02-23, o godz. 15:05:52
Nicholas Geovanis  napisał(a):

> On Fri, Feb 23, 2024, 2:57 PM Dan Ritter  wrote:
> 
> > Stefan Monnier wrote:  
> > > Makes one wonder why they don't use naive append-only "plain
> > > text" logs (tho with appropriate delimiters (maybe some kind of
> > > CSV) to make searches more reliable than with old-style plain
> > > text logs)?
> > >
> > > What are the advantages of journald's representation?
> > > I mean, to justify the slow search and large disk space usage,
> > > there is presumably some upside for some use cases.  I can see
> > > some weak argument against Sqlite based on the size of Sqlite,
> > > but what are the advantages of journald's representation compared
> > > to a naive one?  
> >
> >
> > systemd's design philosophy, observed from the outside, goes
> > like this:
> >  
> 
> bunch trimmed.
> 
> Exactly correct in my view. Systemd's use-case is the desktop, not the
> server in the datacenter. They will be using log-aggregation software
> in the datacenter anyway so no use for systemd logging. We don't
> install desktop software on servers either, no X Windows, no gnome,
> etc. Network connections are stable, no roaming :-)
> 
> Long-term logs are for servers, so systemd doesn't want them.

Right but it would be nice if it could at least forward them upstream
then! Your choices are

* use rsyslog, which does everything better than journald, including
  writing to many different databases directly, just to forward to
  remote host.
* setup super special listener for super special journald remote log
  sending method that nothing but journald supports

If it just supported standard, common protocols like RFC5424 flavour
(that adds structured data to entries), or hell, just "send a JSONed
log to that address", I could just have a bunch of my "small" devices
run journald-only and send it to central logging server. But no dice,
even my router needs to run extra daemon just to forward logs.

Actually, having pluggable "backends" would solve a lot of that, dumb
little IoT device could just have "a text file" or remote output
implemented as write method and "grep the text file" as query method.
Or DBI plugin that could write to SQLite and query it back. Or just
remote log plugin that returned "sorry, your logs are in another
castle" on query. But that's just building rsyslog but worse at that
point...

I wish I disliked C less than I miss those features...
-- 
Mariusz Gronczewski (XANi) 
GnuPG: 0xEA8ACE64
https://devrandom.eu

Re: Journald's qualities

[Mariusz Gronczewski]

Dnia 2024-02-23, o godz. 23:02:49
Jeffrey Walton  napisał(a):

> 
> Systemd also provides tamper-resistant logs. The property is often
> desirable in the enterprise. See Forward Secure Sealing,
> .
> 
> Jeff
> 

I had mentioned that feature. I haven't it seen in a wild or as an
requirement, ever, and we work with few of the local banks deploying
apps on their infrastructure.

It's *basically* always "send the logs to audit server". rm -rf
/var/log does not care about tamper proofing either.

IMO it's a feature that should be a separate plugin that the people
that need it can just load and use. There is no reason to have it in
default logging format or carry the burden of code for it in core.

Now, tamper-proof *wire* format, that could be useful (if enough other
software supported it). Rsyslog have RELP
(https://en.wikipedia.org/wiki/Reliable_Event_Logging_Protocol) that we
use as it fix few of the issues with sending logs via TCP/TLS
(interrupted connection can lose up to buffer's worth of logs), having
on top of that information "hey, some of the blocks of that log were
lost before being sent" would be useful. For that all it would be
needed is to FSS send queue of logger (which wouldn't be queried so it
could be nice and compressed), not entire on-disk format. 

Then again journald can't even send(AFAIK) using normal syslog protocol
as author decided to XKCD#927 that too...

-- 
Mariusz Gronczewski (XANi) 
GnuPG: 0xEA8ACE64
https://devrandom.eu

Re: Journald's qualities (was: Selective rotation of journald logs)

[Mariusz Gronczewski]

Dnia 2024-02-23, o godz. 12:34:34
Dan Ritter  napisał(a):

> Stefan Monnier wrote: 
> > Makes one wonder why they don't use naive append-only "plain text"
> > logs (tho with appropriate delimiters (maybe some kind of CSV) to
> > make searches more reliable than with old-style plain text logs)?
> > 
> > What are the advantages of journald's representation?
> > I mean, to justify the slow search and large disk space usage,
> > there is presumably some upside for some use cases.  I can see some
> > weak argument against Sqlite based on the size of Sqlite, but what
> > are the advantages of journald's representation compared to a naive
> > one?  
> 
> 
> systemd's design philosophy, observed from the outside, goes
> like this:
> 
> - assume that the machine is a laptop or mobile device that is
> always changing: moves from network to network, plugs and
> unplugs devices, goes to sleep and is woken up

I had thought similar thoughts many times when using it. But some of
the stuff that derived from that is pretty useful; for example we had a
service tht downloaded encryption keys from central server to unlock
the hard drive and sysV version was a bit of a mess (if service didn't
ran on boot there was no unlocked volume to mount), while in systemd
the dependencies allow it to automatically bring it up when mount is
requested, and the mount will also be automatically done if service
using it is started (if it has correct RequiresMountsFor entries). So
some actually useful stuff for more "constant config" machines had
flown from that

> - disk space is limited, but cpu time is free

In which way ? It doesn't care about wasting space, but it does assume
you have blazing fast storage or else journal related ops are sluggish
(and waste disk cache as it just reads hundreds of MBs to find last few
lines of logfile)

> - the network knows better than local config

Offtopic but since Debian switched to systemd for DNS management on
VPNs and suc I need to restart it sometimes multiple times to just get
"right" DNS servers, because there appears to be no notion of priority:

https://github.com/systemd/systemd/issues/27543

so now any time I connect to work (just openvpn tunnel, nothing fancy)
I need to spam

systemclt restart systemd-resolved ; sleep 1 ; cat /etc/resolv.conf

few times till the dice rolls the right order of DNS servers...

XANi



-- 
Mariusz Gronczewski (XANi) 
GnuPG: 0xEA8ACE64
https://devrandom.eu

Re: Journald's qualities

[Stefan Monnier]

 but what are the advantages of journald's representation compared
 to a naive one?
>>> 
>>> in short: querability without text parsing. That's about it.
>> 
>> They have to parse the binary format, so that's not in and of itself
>> an upside compared to parsing CSV.
>> 
>> I've made my share of bad design decisions that don't pan out. But
>> there's always an upside to my decision (even when it turns out it 
>> speeds up only those cases which can never occur, because of some
>> other aspect of the system).
>> 
>> AFAICT the format is *not* just a plain sequence of log entries, so 
>> there's some additional structure which is intended to speed up some
>> operations.
>> 
>> IOW, even if contrived, there should be *some* use case where it
>> does better than CSV, no?
>
> I can think of two possibilities, just offhand, in no particular order:
>
> * No need to parse the timestamps, et cetera, and take the risk that
> someone put in one that's in a format you don't expect; the times are
> stored internally in a consistent guaranteed format, so you can just use
> internal reader functions (paired with, and updated alongside, the
> internal writer functions) and be done with it.

Can't think of any reason why the same wouldn't apply to CSV: if someone
messes up the timestamps by hand, they're on their own.

> * No need to worry about handling log entries that *contain* commas, or
> whatever other element was chosen as the separator.

That's just a very minor convenience issue and it does not require
a structure any more complex than a plain sequence of log entries.

Same for FSS, it doesn't seem to require the more complex structure used
by journald.  There must have been some other use-case they had in mind
where they thought they could avoid the linear-time scan or something in
a way that they expected would be algorithmically beneficial.
I just can't see what it is they had in mind.


Stefan

Re: Journald's qualities

[Jeffrey Walton]

On Fri, Feb 23, 2024 at 10:17 PM The Wanderer  wrote:
>
> On 2024-02-23 at 15:35, Stefan Monnier wrote:
>
> >>> but what are the advantages of journald's representation compared
> >>> to a naive one?
> >>
> >> in short: querability without text parsing. That's about it.
> >
> > They have to parse the binary format, so that's not in and of itself
> > an upside compared to parsing CSV.
> >
> > I've made my share of bad design decisions that don't pan out. But
> > there's always an upside to my decision (even when it turns out it
> > speeds up only those cases which can never occur, because of some
> > other aspect of the system).
> >
> > AFAICT the format is *not* just a plain sequence of log entries, so
> > there's some additional structure which is intended to speed up some
> > operations.
> >
> > IOW, even if contrived, there should be *some* use case where it
> > does better than CSV, no?
>
> I can think of two possibilities, just offhand, in no particular order:
>
> * No need to parse the timestamps, et cetera, and take the risk that
> someone put in one that's in a format you don't expect; the times are
> stored internally in a consistent guaranteed format, so you can just use
> internal reader functions (paired with, and updated alongside, the
> internal writer functions) and be done with it.
>
> * No need to worry about handling log entries that *contain* commas, or
> whatever other element was chosen as the separator.

Systemd also provides tamper-resistant logs. The property is often
desirable in the enterprise. See Forward Secure Sealing,
.

Jeff

Re: Journald's qualities

[The Wanderer]

On 2024-02-23 at 15:35, Stefan Monnier wrote:

>>> but what are the advantages of journald's representation compared
>>> to a naive one?
>> 
>> in short: querability without text parsing. That's about it.
> 
> They have to parse the binary format, so that's not in and of itself
> an upside compared to parsing CSV.
> 
> I've made my share of bad design decisions that don't pan out. But
> there's always an upside to my decision (even when it turns out it 
> speeds up only those cases which can never occur, because of some
> other aspect of the system).
> 
> AFAICT the format is *not* just a plain sequence of log entries, so 
> there's some additional structure which is intended to speed up some
> operations.
> 
> IOW, even if contrived, there should be *some* use case where it
> does better than CSV, no?

I can think of two possibilities, just offhand, in no particular order:

* No need to parse the timestamps, et cetera, and take the risk that
someone put in one that's in a format you don't expect; the times are
stored internally in a consistent guaranteed format, so you can just use
internal reader functions (paired with, and updated alongside, the
internal writer functions) and be done with it.

* No need to worry about handling log entries that *contain* commas, or
whatever other element was chosen as the separator.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw



signature.asc
Description: OpenPGP digital signature

Re: Journald's qualities (was: Selective rotation of journald logs)

[Nicholas Geovanis]

On Fri, Feb 23, 2024, 2:57 PM Dan Ritter  wrote:

> Stefan Monnier wrote:
> > Makes one wonder why they don't use naive append-only "plain text" logs
> > (tho with appropriate delimiters (maybe some kind of CSV) to make
> > searches more reliable than with old-style plain text logs)?
> >
> > What are the advantages of journald's representation?
> > I mean, to justify the slow search and large disk space usage, there is
> > presumably some upside for some use cases.  I can see some weak argument
> > against Sqlite based on the size of Sqlite, but what are the advantages
> > of journald's representation compared to a naive one?
>
>
> systemd's design philosophy, observed from the outside, goes
> like this:
>

bunch trimmed.

Exactly correct in my view. Systemd's use-case is the desktop, not the
server in the datacenter. They will be using log-aggregation software in
the datacenter anyway so no use for systemd logging. We don't install
desktop software on servers either, no X Windows, no gnome, etc. Network
connections are stable, no roaming :-)

Long-term logs are for servers, so systemd doesn't want them.
> systemd thinks logs are for finding out what just happened
> recently. If you wanted long-term logs, obviously you would
> configure a central repository on some other machine and ship
> them across the network.
>
>
> I have nothing but praise for the Debian maintainers of rsyslog,
> who have arranged it so that installing rsyslog immediately does
> appropriate things to pull data out of systemd.
>
> -dsr-
>
>

Re: Journald's qualities

[Stefan Monnier]

> systemd's design philosophy, observed from the outside, goes
> like this:

Let's try and stick to the subject of the log representation in
`journald`, because we all know about the varied opinions about SystemD.

Being an "old-hand", I'm not in love of SystemD, but the thing does have
its upsides.  In the case of the format of `journald` logs, I don't
see them.


Stefan

Re: Journald's qualities

[Stefan Monnier]

>> Oh, that bug report is quite interesting, thanks.
>> Makes one wonder why they don't use naive append-only "plain text"
>> logs (tho with appropriate delimiters (maybe some kind of CSV) to make
>> searches more reliable than with old-style plain text logs)?
>> 
>> What are the advantages of journald's representation?
>> I mean, to justify the slow search and large disk space usage, there
>> is presumably some upside for some use cases.  
>
> The advantage of binary format is that you have separate fields for
> program, level etc and it is easier to filter by exactly what you want. 
> So "systemctl status servicename" just asks log subsystem for exactly
> what it needs.

That's why I said "with appropriate delimiters (maybe some kind of
CSV)".  Really plain text is what we had before, but it's not rocket
science to have a "plain text" format with a reliably delimited set
of fields, which is what I'd assume as the obvious naive choice.

>> argument against Sqlite based on the size of Sqlite, 
> I'd argue that's not really technically relevant

I did say "weak" 🙂

>> but what are the advantages of journald's representation compared to
>> a naive one?
> in short: querability without text parsing. That's about it.

They have to parse the binary format, so that's not in and of itself an
upside compared to parsing CSV.

I've made my share of bad design decisions that don't pan out.
But there's always an upside to my decision (even when it turns out it
speeds up only those cases which can never occur, because of some other
aspect of the system).

AFAICT the format is *not* just a plain sequence of log entries, so
there's some additional structure which is intended to speed up
some operations.

IOW, even if contrived, there should be *some* use case where it does
better than CSV, no?

> It does have some features around FSS (signing logs) so for folks that
> need it there isn't many alternatives as it would be hard to make
> similar feature on database.

That should be just as easy to add to a CSV-style representation, tho.

> But on flipside inability to separate log into log rotation groups is
> *nasty*. You basically can't have verbose service in the system as that
> will rotate away any old logs of other services away. And looking for
> those logs is also the worst case:

It shouldn't be difficult to add a filtering step to the log rotation
(just extract all the log entries, filter out the unwanted ones, and
create a new log file with the result).
Beside the obvious usual issues of atomicity, the only real problem
I can imagine there is FSS for which you'd need to seal separately the
groups that share differently filtering rules.

> I guess another one being appendable log is that in case of crash it's
> easier to recover than embedded database...

AFAICT, CSV shares this property just as well.

> One-service-per-file approach is honestly good enough for most stuff.
> PITA to get chronological order though, every approach really have
> some drawbacks and benefits.

If you use a reasonable format with precise enough timestamps, you
should be able to "weave" entries from separate files back into
a coherent single chronology (assuming all those entries got their
timestamps from the same journald in the first place).


Stefan

Re: Journald's qualities (was: Selective rotation of journald logs)

[Dan Ritter]

Stefan Monnier wrote: 
> Makes one wonder why they don't use naive append-only "plain text" logs
> (tho with appropriate delimiters (maybe some kind of CSV) to make
> searches more reliable than with old-style plain text logs)?
> 
> What are the advantages of journald's representation?
> I mean, to justify the slow search and large disk space usage, there is
> presumably some upside for some use cases.  I can see some weak argument
> against Sqlite based on the size of Sqlite, but what are the advantages
> of journald's representation compared to a naive one?


systemd's design philosophy, observed from the outside, goes
like this:

- assume that the machine is a laptop or mobile device that is
always changing: moves from network to network, plugs and
unplugs devices, goes to sleep and is woken up

- disk space is limited, but cpu time is free

- the network knows better than local config

- all services serve the local user's desktop, primarily

- the local user doesn't know anything but can call on a remote
  sysadmin

- systemd is the best at doing anything, so it should do
everything.

If all of these assumptions match up with your particular use
case, systemd is happy.


Long-term logs are for servers, so systemd doesn't want them.
systemd thinks logs are for finding out what just happened
recently. If you wanted long-term logs, obviously you would
configure a central repository on some other machine and ship
them across the network.


I have nothing but praise for the Debian maintainers of rsyslog,
who have arranged it so that installing rsyslog immediately does
appropriate things to pull data out of systemd.

-dsr-

Re: Journald's qualities (was: Selective rotation of journald logs)

[Mariusz Gronczewski]

Dnia 2024-02-23, o godz. 10:26:49
Stefan Monnier  napisał(a):

> > [13:37:48]cthulhu:/var/log/journal☠ journalctl |dd of=/dev/zero
> > bs=1M 0+15442 records in
> > 0+15442 records out
> > 63643115 bytes (64 MB, 61 MiB) copied, 5,47791 s, 11,6 MB/s
> > du -h /var/log/journal/
> > 337M/var/log/journal/44cf6f547971fc33309d1e9e02e7
> > 337M/var/log/journal/
> >
> > (I've raised a bug 8 years ago about it
> >  https://github.com/systemd/systemd/issues/2460 )  
> 
> Oh, that bug report is quite interesting, thanks.
> Makes one wonder why they don't use naive append-only "plain text"
> logs (tho with appropriate delimiters (maybe some kind of CSV) to make
> searches more reliable than with old-style plain text logs)?
> 
> What are the advantages of journald's representation?
> I mean, to justify the slow search and large disk space usage, there
> is presumably some upside for some use cases.  

The advantage of binary format is that you have separate fields for
program, level etc and it is easier to filter by exactly what you want. 
So "systemctl status servicename" just asks log subsystem for exactly
what it needs.

But journald logs aren't build for querying speed, there is for example
no index (in memory or on disk) that tells you where last log entries
from apps are so you get pathological cases like in ticket.

The problem is that you kinda do 2 types of querying on logs most often:

* get me all/most of what happened in last x lines/hours
* get me everything recent (or just everything period) that this
  particular service returned.

Which kinda exclude the "simple" tricks" like "just hash over service
name so at worst you have to search only 1/64th of files" as they would
make the "get me everything" case slower. But maybe not by all that
much.

The size issue... it's appendable binary format which means that any
metadata systemd adds can't be deduplicated by just having join on
table with service info, it needs to be repeated every single line and
I'm guessing that's where the bloat comes from.

I rambled about SQLite as it is kinda closest comparison for embedded
binary format that can be searched and I think that being able to run
SQL queries (sqlite now even have full text search) against your local
logs with no database daemons running would be killer feature on some of
the smaller devices.


> argument against Sqlite based on the size of Sqlite, 

I'd argue that's not really technically relevant as in most cases you'd
have other app in system using sqlite for *something* and will have
installed the library anyway. And space savings gets wasted just on how
verbose journald format is.

> but what are the
> advantages of journald's representation compared to a naive one?

in short: querability without text parsing. That's about it.

Which *is* a great feature, do not get me wrong, I use it all the time,
but the slowdown caused by its unfortunate implementation detail is
also showing up when I use it pretty often as the most common use case
is "something broke, log in on server that has been running on
automation for last 3 months and see what is happening"

It does have some features around FSS (signing logs) so for folks that
need it there isn't many alternatives as it would be hard to make
similar feature on database.

But on flipside inability to separate log into log rotation groups is
*nasty*. You basically can't have verbose service in the system as that
will rotate away any old logs of other services away. And looking for
those logs is also the worst case:

[17:57:56]cthulhu:~☠ echo 3 > /proc/sys/vm/drop_caches
[17:58:14]cthulhu:~☠ time systemctl status nginx |grep Notice
Notice: journal has been rotated since unit was started, output may be 
incomplete.

real0m1,645s
user0m0,000s
sys 0m0,094s
[17:58:19]cthulhu:~☠ echo 3 > /proc/sys/vm/drop_caches
[17:58:28]cthulhu:~☠ time sqlite /tmp/log_indexed.sqlite 'select * from log 
where source = "nginx";'  >/dev/null

real0m0,374s
user0m0,000s
sys 0m0,009s


I guess another one being appendable log is that in case of crash it's easier 
to recover than embedded database...

...or so would I say if I didn't saw journald throwing away binary logs with 
some errors after crash.


> 
> 
> Stefan
> 
> 
> PS: FWIW, the only time I thought about the design of a log format,
> I was thinking along the lines of reducing redundancy, i.e. keeping
> a kind of "suffix tree" of past messages so as to replace the many
> repeated chunks of text with references to past log entries.
> Then I realized that I could get the same result much more simply by
> gzipping the files, which would naturally be done as part of
> log rotation 🙂
> 

One-service-per-file approach is honestly good enough for most stuff.
PITA to get chronological order thought, every approach really have
some drawbacks and benefits.

But honestly ? Send it to central logger and worry there if you have a
chance, especially if you do some parsing there can be

Journald's qualities (was: Selective rotation of journald logs)

[Stefan Monnier]

> [13:37:48]cthulhu:/var/log/journal☠ journalctl |dd of=/dev/zero bs=1M
> 0+15442 records in
> 0+15442 records out
> 63643115 bytes (64 MB, 61 MiB) copied, 5,47791 s, 11,6 MB/s
> du -h /var/log/journal/
> 337M  /var/log/journal/44cf6f547971fc33309d1e9e02e7
> 337M  /var/log/journal/
>
> (I've raised a bug 8 years ago about it
>  https://github.com/systemd/systemd/issues/2460 )

Oh, that bug report is quite interesting, thanks.
Makes one wonder why they don't use naive append-only "plain text" logs
(tho with appropriate delimiters (maybe some kind of CSV) to make
searches more reliable than with old-style plain text logs)?

What are the advantages of journald's representation?
I mean, to justify the slow search and large disk space usage, there is
presumably some upside for some use cases.  I can see some weak argument
against Sqlite based on the size of Sqlite, but what are the advantages
of journald's representation compared to a naive one?


Stefan


PS: FWIW, the only time I thought about the design of a log format,
I was thinking along the lines of reducing redundancy, i.e. keeping
a kind of "suffix tree" of past messages so as to replace the many
repeated chunks of text with references to past log entries.
Then I realized that I could get the same result much more simply by
gzipping the files, which would naturally be done as part of
log rotation 🙂