Re: Heads up: persistent journal has been enabled in systemd

[Dominique Dumont]

On Wednesday, 5 February 2020 22:40:29 CET Dmitry Smirnov wrote:
> There are log readers like "lnav" and "multitail" that will become useless
> without traditional log files. "lnav" tails multiple logs by default and
> IMHO provides a very useful interface.

multitail provides -l option to read logs from a command. For instance:

 SYSTEMD_COLORS=0 multitail -l 'journalctl -f'

lnav has a similar feature:

 SYSTEMD_COLORS=0 journalctl -f | lnav

Hope this helps

Re: Heads up: persistent journal has been enabled in systemd

[Guillem Jover]

Hi!

On Tue, 2020-02-11 at 08:19:46 +0100, Helmut Grohne wrote:
> I am an early adopter (at a time when you had to pass init= to use
> systemd) and I also enabled the persistent journal on practically all of
> my systems. I find myself liking the filtering that is enabled by
> journalctl, but it seems to come at a cost: performance.

> On multiple systems (embedded, VMs, desktops, servers) I observe that
> journalctl takes longer to display the initial batch than I am willing
> to wait. Unfortunately, this also affects systemctl status. I admit that
> my patience is quite limited here. Having to wait 3 seconds for
> systemctl status someservice is already more than I am willing to wait.
> As such, I find myself resorting to plaintext logs more often than not
> to avoid the annoying delay. It gets way worse on a busy system where
> you'd need the journal most to figure out what's wrong.
> 
> Do you happen to know more about the performance aspects?
>  * Known discussions?
>-> https://github.com/systemd/systemd/issues/2460
>  * Workarounds / tricks?
>-> I already apply a size limit.
> 
> Memory consumption by journald itself is also worth a mention. For
> servers, I usually don't care, but for embedded systems it is sometimes
> difficult to afford. rsyslog comes at around a fifth of what journald
> needs.
> 
> As such, I question whether the journal is ready for production while at
> the same time wanting it to be. I believe that the github issue above
> should be fixed before enabling the persistent journal by default.

Unfortunately this all matches our experience too. At work we also tried
to enable persistent journal, to potentially also remove rsyslogd, and
to get the enhanced logging combined with service status, etc. But had
not only to revert that as the run-time performance degraded significantly
(not just when using systemctl or journalctl, but during normal system
operation too), we even had to set:

  Storage=none
  RateLimitIntervalSec=0s
  RateLimitBurst=0

to cope. :/ The default does not affect us in any way, we'd really like
to switch fully, and we've not tried yet with newer systemd versions,
but just mentioning as a word of warning of what this might imply.

Thanks,
Guillem

Re: Heads up: persistent journal has been enabled in systemd

[Anthony DeRobertis]

On February 11, 2020 6:28:08 PM UTC, Ansgar  wrote:
>
>The downside is that magic like [rsyslog disabling persistent journal] might 
>not be easily discoverable
>and confuse people who for some reason want a persistent journal and
>syslog.

A lot of my machines are configured like that, mainly because of how much 
faster things like tail -f /var/log/syslog are vs. journalctl -f. So I keep a 
short amount of logs in traditional syslog format (quick access), and much 
longer history in journald (better search/filter). I get the best of both 
worlds.

It'd be very surprising if rsyslog disabled the persistent journal on upgrade. 
Or when installing it.

I think it'd be fine for it to give me the option, default do not disable, via 
debconf, not sure if that'd be low or medium.

There's the obvious point that enabling the persistent journal could be 
surprising, too. But I think the harm there is relatively minimal (increased 
disk usage, but journald will not fill a filesystem) and is easily fixed with 
rm. The lack of a persistent journal isn't necessarily noticed until you try to 
check the logs, and at that point may be unfixable: the data is gone.

I would more worry about the harm on systems where there are intentionally no 
persistent logs, neither journal nor syslog. There could be privacy 
implications. But hopefully the release notes cover that use case.

Re: Heads up: persistent journal has been enabled in systemd

[Ansgar]

Hi,

On Sat, 2020-02-01 at 13:36 +, Steve McIntyre wrote:
> Michael Biebl wrote:
> > with today's upload of systemd 244.1-2 I finally enabled persistent
> > journal by default [1]. It has been a long requested feature.
> > 
> > The package will create a directory /var/log/journal on upgrades and new
> > installs, which enables persistent journal in so called auto mode.
> 
> Fine for new installations, but please *don't* do this for
> upgrades. Those people with existing logging setups will be surprised
> by this.

I generally prefer upgraded systems to behave more like newly installed
ones, but I wonder if here syslog daemons like rsyslog should ship a
journald.conf dropin in lib/systemd/journald.conf.d/00-rsyslog.conf
setting `Storage=volatile`. If you have installed a syslog daemon, you
likely don't want a persistent journal as well.

Upgraded systems that have rsyslog installed won't get a persistent
journal this way.

The downside is that magic like this might not be easily discoverable
and confuse people who for some reason want a persistent journal and
syslog.

Ansgar

Re: Heads up: persistent journal has been enabled in systemd

[Helmut Grohne]

Hi Michael,

On Sat, Feb 01, 2020 at 04:05:55AM +0100, Michael Biebl wrote:
> with today's upload of systemd 244.1-2 I finally enabled persistent
> journal by default [1]. It has been a long requested feature.

Thank you.

> Users that prefer text logs can of course still install rsyslog by
> default (or their syslogger of choice).

I am an early adopter (at a time when you had to pass init= to use
systemd) and I also enabled the persistent journal on practically all of
my systems. I find myself liking the filtering that is enabled by
journalctl, but it seems to come at a cost: performance.

On multiple systems (embedded, VMs, desktops, servers) I observe that
journalctl takes longer to display the initial batch than I am willing
to wait. Unfortunately, this also affects systemctl status. I admit that
my patience is quite limited here. Having to wait 3 seconds for
systemctl status someservice is already more than I am willing to wait.
As such, I find myself resorting to plaintext logs more often than not
to avoid the annoying delay. It gets way worse on a busy system where
you'd need the journal most to figure out what's wrong.

Do you happen to know more about the performance aspects?
 * Known discussions?
   -> https://github.com/systemd/systemd/issues/2460
 * Workarounds / tricks?
   -> I already apply a size limit.

Memory consumption by journald itself is also worth a mention. For
servers, I usually don't care, but for embedded systems it is sometimes
difficult to afford. rsyslog comes at around a fifth of what journald
needs.

As such, I question whether the journal is ready for production while at
the same time wanting it to be. I believe that the github issue above
should be fixed before enabling the persistent journal by default.

Helmut

Re: Heads up: persistent journal has been enabled in systemd

[Thomas Goirand]

On 2/4/20 8:30 PM, Russ Allbery wrote:
> Dmitry Smirnov  writes:
>> On Saturday, 1 February 2020 2:05:55 PM AEDT Michael Biebl wrote:
> 
>>> Depending on how it goes, I might ask the ftp-masters to lower the
>>> priority of rsyslog from important to optional, so it would no longer
>>> be installed by default on new bullseye installations.
> 
>> I have a mixed feelings about that. I think that replacing rsyslog with 
>> journald is two steps back in regards to functionality and flexibility.
> 
>> Rsyslog in unparalleled with its ability to process and filter messages
>> (rainerscript), to transform messages (liblognorm), to forward messages
>> to Elasticsearch or centralise Rsyslog instance, _reliably_ (relp), to
>> buffer message queue on disk in case communication is disrupted, etc.
> 
> I completely agree with this assessment of the quality of rsyslog
> features, but I'm not sure that's the right criteria to be considering for
> the choice of the *default*.  I'm fairly certain that 95% or more of
> installed Debian systems never used any of those features, as nice as they
> are.
> 
> The goal of the default is not to provide in latent form every excellent
> feature that anyone may want to use.  It's to provide a hopefully simple,
> reliable, functional, and light-weight implementation of a facility that
> serves as a reasonable default for most systems.  Anyone with other needs
> or preferences is very likely to replace or supplement that implementation
> with something else, similar to how I replace exim4 with postfix on all of
> my systems.

I'm ok with this reasoning, however, rsyslogd is also a reasonable
default, and I don't see why it wouldn't be good for the general use
also. The main problem with journald is that it can't keep-up with too
much syslog traffic and doesn't scale. I'd very much prefer if our
default was a solution that works on *all* cases, rather than just the
one for a specific target.

Cheers,

Thomas Goirand (zigo)

Re: Heads up: persistent journal has been enabled in systemd

[Hideki Yamane]

Hi,

 Thanks for your heads up.

On Sat, 1 Feb 2020 04:05:55 +0100
Michael Biebl  wrote:
> with today's upload of systemd 244.1-2 I finally enabled persistent
> journal by default [1]. It has been a long requested feature.

 I read this thread and other info, my thought is


Pros)

 Well, as I read upstream's documentation[1], I prefer persistent journal
 if they implemented features as they said. Especially journald logging
 feature seems to be more reliable (tamper-proof) than normal syslog's one.
 e.g. if someone hacked your system and you found it, then logs are
 NOT reliable information easily. However, if you've enabled persistent
 journal, it's hard to falsify it.


 This change is non-destructive, users can change its default setting to
 go back to rsyslog as Michael notes.


Cons)

 There are some regression at "reading" logs (e.g. we must specify syslog
 facilities by number)[2]

 Users should learn new "How to use journalctl" things.


Note)
 
 Ubuntu already did this change on upgrades and it is smooth.
 



 So, my conclusion is

 Use "reliable logging system" is good for our users, so I prefer it as
 the default. If you need more flexibility, then you can install rsyslog
 or something for your logging system - yes, you have a choice! :)


[1] 
https://docs.google.com/document/pub?id=1IC9yOXj7j6cdLLxWEBAGRL6wl97tFxgjLUEHIX3MSTs
[2] https://github.com/systemd/systemd/issues/9716



-- 
Regards,

 Hideki Yamane henrich @ debian.org/iijmio-mail.jp

Re: Heads up: persistent journal has been enabled in systemd

[Steve McIntyre]

Arg, typo in the list address. Sorry. :-(

On Sat, Feb 08, 2020 at 09:28:50PM +, Steve McIntyre wrote:
>Hey Michael!
>
>Sorry for leaving you hanging here. Really busy week...
>
>Michael Biebl wrote:
>>Am 01.02.20 um 14:36 schrieb Steve McIntyre:
>>> Michael Biebl wrote:

 with today's upload of systemd 244.1-2 I finally enabled persistent
 journal by default [1]. It has been a long requested feature.

 The package will create a directory /var/log/journal on upgrades and new
 installs, which enables persistent journal in so called auto mode.
>>> 
>>> Fine for new installations, but please *don't* do this for
>>> upgrades. Those people with existing logging setups will be surprised
>>> by this.
>>
>>I honestly don't like packages that behave differently depending on
>>whether they were upgraded or installed anew (given their default
>>configuration wasn't changed). I assume there would be equally as many
>>or even more surprised users that find their upgraded system behave
>>differently then their freshly installed system.
>>
>>The change here is basically just an update of the default behaviour of
>>journald. If you explicitly configured a different journald behavior via
>>Storage=, this is respected. If you already created a /var/log/journal
>>in the past, the change will be a nop.
>>
>>Existing sysloggers will continue to work after this change as before,
>>they are not directly affected by this change.
>>
>>Can you eloborate more on this what your concerns are here?
>
>This would suddenly change behaviour on existing systems with no
>persistent journal. That's my setup right now - I'm using rsyslog for
>logging, with journald just as a pass-through. After an upgrade, I'd
>get *both* persistent journal and rsyslog writing logs to disk. That's
>not ideal for anybody.
>
>>I think Didier's idea of documenting this in the release notes is a good
>>one, so I filed #950447. The NEWS entry is a good idea as well and I
>>thought about that, but decided to not add it (yet) but instead ask for
>>feedback on debian-devel first. Depending on the feedback, the content
>>of that NEWS entry might change. E.g. I was unsure whether I should
>>mention that a syslogger like rsyslog can now be uninstalled safely, as
>>the contents are now available in the journal as well. Feedback on this
>>welcome.
>
>If you're going to make a change like this, *definitely* it should be
>mentioned in NEWS, IMHO.
>
>-- 
>Steve McIntyre, Cambridge, UK.st...@einval.com
>  Armed with "Valor": "Centurion" represents quality of Discipline,
>  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
>  concord the digital world while feeling safe and proud.
>
-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
"I suspect most samba developers are already technically insane... Of
 course, since many of them are Australians, you can't tell." -- Linus Torvalds

Re: Heads up: persistent journal has been enabled in systemd

[Wouter Verhelst]

On Thu, Feb 06, 2020 at 10:45:05AM +0100, Svante Signell wrote:
> On Thu, 2020-02-06 at 07:58 +0100, Vincent Bernat wrote:
> >  ❦  6 février 2020 09:50 +11, Dmitry Smirnov :
> > 
> > > > and 2) continuing to use rsyslog isn't an option if the default changes.
> > > 
> > > No. I just don't want default to change. IMHO rationale for this is weak
> > > but everybody keeps arguing that it would not be a big deal. In time we 
> > > will
> > > see how that goes (what could possibly go wrong?) but why do the change in
> > > first place?
> > 
> > To not have logs duplicated in two places.
> 
> If this is your motivation for the change it is a _very_ weak one, right? Disk
> space is not a crucial problem anymore.

I have been using computers for three decades now. All that time, people
have been saying "disk space is cheap nowadays". Yet all that same time,
my disks have been hovering around ~95% full.

Disk space is, and will always remain, a "crucial problem". If we have
more disk space, that just means people will find novel ways of filling
it up to the brink.

-- 
To the thief who stole my anti-depressants: I hope you're happy

  -- seen somewhere on the Internet on a photo of a billboard

Re: Heads up: persistent journal has been enabled in systemd

[Andrei POPESCU]

On Jo, 06 feb 20, 22:43:50, Scott Kitterman wrote:
> 
> I have one specific request relative to this change that should be 
> pretty easy:
> 
> Please include in the bullseye release notes a description of this 
> change and tell users how to restore the buster logging configuration 
> if they prefer it.

As mentioned in the first mail, Michael already filed #950447.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Lisandro Damián Nicanor Pérez Meyer]

Hi!

El jue., 6 feb. 2020 20:39, Scott Kitterman  escribió:
[snip]

I went back this far in the thread to try to get back before it went off
the rails (IMO).

I get that the results of this discussion aren't going to be my
preference.  Ok.  Win some, lose some.  I think it's a bad decision for
Debian, but it's not my call.  Let's move on.


While I do not have a specific opinion on the correct default, knowing I
can easily switch it is already a great thing.


I have one specific request relative to this change that should be pretty
easy:

Please include in the bullseye release notes a description of this change
and tell users how to restore the buster logging configuration if they
prefer it.


I do totally agree here.

Also, thanks Michael for bringing this over. Whatever the result is the
fact that you put this on the table is definitely the way to go.

Cheers, Lisandro.

Re: Heads up: persistent journal has been enabled in systemd

[Dmitry Smirnov]

On Friday, 7 February 2020 4:22:49 AM AEDT Marco d'Itri wrote:
> If you want to argue that the adoption of popcon is not uniform among
> different types of systems then you need to prove it.
> Expectations and anedoctes are not statistics.

Come on, maybe in theory there is such thing as "spherical cow" but in
reality "popularity-contest" is not installed by default in Azure Debian 
images. So there is already a non-negligible number of Debian systems in 
Azure that do not report statistics. And that's just one example...

-- 
Cheers,
 Dmitry Smirnov.

---

There have been times throughout American history where what is right is
not the same as what is legal. Sometimes to do the right thing you have to
break the law.
-- Edward Snowden


signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Scott Kitterman]

On February 6, 2020 5:22:15 AM UTC, Scott Kitterman  
wrote:
>
>
>On February 5, 2020 8:43:43 AM UTC, Bastian Blank 
>wrote:
>>Hi Scott
>>
>>On Wed, Feb 05, 2020 at 07:44:25AM +, Scott Kitterman wrote:
>>> >> Of course the fact that I can't use all the tools available to
>>> >manipulate text 
>>> >> files to follow or analyze logs is problematic.  If I'm using
>>> >journalctl, how 
>>> >> do I replicate 'tail -f /var/log/mail.loog'?
>>> >
>>> >journalctl -f SYSLOG_IDENTIFIER=2
>>> >
>>> Do syslog facilities really have to be addressed by number rather
>>than name?  That seems like a horrible interface.
>>
>>That's because you asked a pretty specific question and you've been
>>told the exact replacement.
>>
>>However, you most likely don't actually want to tail this specific
>>file.
>>It is a solution for a more abstract problem, e.g. "I want to see logs
>>from Postfix".  And this question got a different answer.
>
>Yes, it's been demonstrated that what I'm after won't be impossible
>with journalctl, only far more difficult.  And yes, I quite frequently
>find myself tailing that specific file because it has exactly the
>information I need.  Please don't explain to me what I want like you
>understand it better than I do.
>
>I understand the premise that journalctl offers the promise of better
>interfaces to system logs.  I think it's a long way from that now and I
>don't think my uses of system logs are particularly niche.
>
>Scott K

I went back this far in the thread to try to get back before it went off the 
rails (IMO).  

I get that the results of this discussion aren't going to be my preference.  
Ok.  Win some, lose some.  I think it's a bad decision for Debian, but it's not 
my call.  Let's move on.

I have one specific request relative to this change that should be pretty easy:

Please include in the bullseye release notes a description of this change and 
tell users how to restore the buster logging configuration if they prefer it.

Scott K

Re: Heads up: persistent journal has been enabled in systemd

[Simon Richter]

Hi,

On Thu, Feb 06, 2020 at 07:24:28PM +0100, Ansgar wrote:

[popcon is not indicative of install base]

> That's likely given some libreoffice packages (unlikely to be installed
> on servers) are at ~40% popcon and I would expect significantly more
> server installations than desktop ones.

> So popcon might overestimate sysvinit usage and it might in reality be
> lower over the total installation base.

Also quite possible.

That's my point: we do not have useful data on deployments because desktops
are more likely to report in, container builders are more likely to
repeatedly download the same package in rapid succession, container users
will often not download an init system from the mirror network at all
because it is in the base images, datacenter operators will certainly run
their own mirrors, embedded systems are deployed via dd'ing images,
company-wide FAI rollouts almost certainly disable popcon, containers use
the service as the entry point and skip the init system, dependencies might
pull in packages that go unused, ...

That is what I'm saying: these numbers have more noise than signal in them,
especially for the init systems. The numbers are very useful for comparing
packages within a suite, e.g. GNOME Terminal vs GNOME Minesweeper, which is
why we use them to distribute packages over the installation media, but I
wouldn't even trust them to get the ratio of GNOME vs KDE installations
right.

If we wanted to get more accurate statistics, we'd have to at least
distinguish between bare metal, VMs, and containers on one axis, and
manual installation by the user, default installation by d-i or automated
installation through a configuration management tool, and then we'd still
have to correct for the fact that packages installed by default will not
usually be manually installed. We'd have slightly more accurate numbers
that would still give anyone without a relevant doctorate pretty much
nothing to work with, and it would be a lot of effort, so let's not.

tl;dr: statistics is hard, please do not base decisions on popcon or
downloads.

   Simon

Re: Heads up: persistent journal has been enabled in systemd

[Svante Signell]

On Thu, 2020-02-06 at 14:14 -0500, Michael Stone wrote:
> On Thu, Feb 06, 2020 at 08:09:13PM +0100, Svante Signell wrote:

> > > > On a Debian system _not_ running systemd:
> > > > 
> > > > du -sh /var/log
> > > > 74M /var/log

> > Of course it matters. It is about the _default_ setting, or have I
> > understood this thread wrongly?

Yes, maybe I have misunderstood. This is about systems with systemd
installed, but that is the Debian default right? No way to install
Debian without getting it as default. When is Debian ever to offer a
non-systemd alternative for installation?

Quoting the original posting from Michael Biebl:
Depending on how it goes, I might ask the ftp-masters to lower the
priority of rsyslog from important to optional, so it would no longer
be installed by default on new bullseye installations.
This would avoid, that we store log messages twice on disk.
Users that prefer text logs can of course still install rsyslog by
default (or their syslogger of choice). Alternative init systems might
consider adding a Recommends on a syslog implementation of their choice
or creating a task, which would pull in a syslogger.

> You seem to have misunderstood this thread wrongly in many ways.
> It's  not clear what you hope to accomplish by continuing. If you
> just want everyone to know that you dislike systemd, congratulations:
> you've  accomplished that. Now please move on.

So the above quote from Michael is for Debian and as a default you will
get the systemd journald binary logs. This is very much in the same
vein as the discussion with Thomas Goirand about systemd-
{users,tmpfiles}. If systemd is installed the default _has_ to be a
systemd-monolithic add-on, no discussion possible, I realize that now.
The GR had no effect on that.

Fine, but then give us a chance to create a systemd-free Debian
alternative, without blocking every attempt to do that :(

Do you want me to give examples of e.g. bugs/proposed solutions you
have blocked recently?

Re: Heads up: persistent journal has been enabled in systemd

[Ondřej Surý]

> On 6 Feb 2020, at 13:25, Ansgar  wrote:
> 
> I have no problem installing a different MTA than Debian's default
> (exim), my preferred shell, my preferred editor and so on either.

That. The first thing I do on a fresh Debian system is to install
postfix, configure it as smarthost and install emacs-nox. I don’t
care about the defaults as long as I have a choice.

So, I am perfectly fine with changing the default to journalctl.

P.S.: I absolutely love coredumpctl as a sofware developer
and I absolutely hate systemd-resolved as a DNS developer,
but honestly I would be ok with systemd-resolved installed
by default as it is improvement over doing nothing.

Ondrej
--
Ondřej Surý
ond...@sury.org

Re: Heads up: persistent journal has been enabled in systemd

[Eduard Bloch]

Hallo,
* Marvin Renich [Wed, Feb 05 2020, 08:27:02AM]:
> * Matt Zagrabelny  [200204 21:27]:
> > The contents of /var/log/journal will be binary files that journalctl
> > will read. IIRC.
>
> This is my objection to the systemd journal.  Binary log files are
> absolutely _horrible_ for the general user, but they are terrific for
> large data centers.

Who is your general user? A dude who reads log files all day long? That
dude would probably create scripts to help him anyway.

> In a large data center, the logs from many machines are being sent via
> network to a machine whose job it is to collect them, where sysadmins
> can monitor all the hosts.  The binary format is much more efficient.
>
> However, on an individual user's machine, you are now forced to use a
> specialized program to parse the logs.  If a journal file is damaged in

Apples and oranges. Binary is also more efficient for local storage, and
on the remote machine you also need to convert it for reading.

> When your machine has crashed or your hard disk is having intermittent
> failures are times when you most need to be able to read log files, and
> are times when dealing with a binary format may be the hardest.

When a machine has crashed that severe that even log files are damaged,
then are probably also full of junk anyway when they are plain ASCII.
When you loose the most recent data due to crash then it also makes
hardly any difference whether that filepart was text or binary.

> When choosing defaults for the Debian distribution, if one default is
> more appropriate for the general user and another is better for the
> experienced sysadmin, the maintainer should almost always opt in favor
> of the general user.

Exactly. Which means binary for general users, which is more appropriate
(reduces CPU cycle waste and power consumption, while a "general user"
hardly needs to read log files).

Best regards,
Eduard.

Re: Heads up: persistent journal has been enabled in systemd

[Michael Stone]

On Thu, Feb 06, 2020 at 08:09:13PM +0100, Svante Signell wrote:

On Thu, 2020-02-06 at 13:41 -0500, Michael Stone wrote:

On Thu, Feb 06, 2020 at 07:22:07PM +0100, Svante Signell wrote:
> On a Debian sytem _not_ running systemd:
>
> du -sh /var/log
> 74M/var/log
>
> And the binary logs from systemd would of course be much smaller
> since
> they are binary. Any numbers?

It looks like you just proved that this discussion doesn't matter
for
your use case. Can we stop this now?


Of course it matters. It is about the _default_ setting, or have Iunderstood 
this thread wrongly?


You seem to have misunderstood this thread wrongly in many ways. It's 
not clear what you hope to accomplish by continuing. If you just want 
everyone to know that you dislike systemd, congratulations: you've 
accomplished that. Now please move on.



Numbers for systemd installations, please!


No, you're wasting everyone's time with this irrelevant digression.

Re: Heads up: persistent journal has been enabled in systemd

[Svante Signell]

On Thu, 2020-02-06 at 13:41 -0500, Michael Stone wrote:
> On Thu, Feb 06, 2020 at 07:22:07PM +0100, Svante Signell wrote:
> > On a Debian sytem _not_ running systemd:
> > 
> > du -sh /var/log
> > 74M /var/log
> > 
> > And the binary logs from systemd would of course be much smaller
> > since
> > they are binary. Any numbers?
> 
> It looks like you just proved that this discussion doesn't matter
> for 
> your use case. Can we stop this now?

Of course it matters. It is about the _default_ setting, or have Iunderstood 
this thread wrongly? 

Numbers for systemd installations, please!

Re: Heads up: persistent journal has been enabled in systemd

[Martin Steigerwald]

Sam Hartman - 06.02.20, 17:11:20 CET:
> > "Martin" == Martin Steigerwald  writes:
> Martin> Well, that is *exactly* why I thought the GR is not going
> to Martin> be helpful.
> 
> Martin> Cause in *no way* it appeared to have *solved* the
> conflict Martin> underneath it.
> 
> No the GR did not somehow magically cause people to agree.
> It did decide what Debian is going to do.
> That's already had significant positive impacts even on this thread.

I replied in private. This is the public part of my reply – I apologize 
for anything in there that someone may receive as other than non-
violent, I tried the best to write from my point of view, but this is 
going to be emotional:

Your reply exactly shows the kind of toxic way to communicate that the 
GR invited to the project. A way to communicate that lacks empathy and 
compassion which is necessary for splits in communities to heal.

From what I can see you are doing a huge disservice to the Debian 
community with an reply like this – deepening the split that is already 
there and has never been healed so far –, but there is no point in 
bringing that up in detail here at this moment.

I continue to stay unsubscribed as I unsubscribed for a reason. I can 
let go putting up with the toxic way to communicate that I have seen and 
even experienced on this list more than once and that in that way I 
never ever saw or experienced on mailing lists of communities like KDE 
or Devuan.

I love myself and I am not putting up with that kind of toxic 
communication anymore.

Please refrain from answering to me and (!) to the mailing list like you 
did in your reply. I unsubscribed from this list for a reason and at 
least for now do not intend to read what people write after I left. You 
can rant all the way you want about what I wrote, but please leave me 
out of it.

So this is the public part of it.

Again, I respectfully ask you to leave me alone if you have nothing 
constructive or supportive to tell me – and what you wrote me, Sam, 
clearly was neither supportive nor constructive for me. At the very 
least, do reply privately to me if you feel you must do that. I my 
delete any further mail on this without notification. I just replied to 
this one in public as well as you wrote it to the mailing list *and* to 
me, even tough I told I am going to unsubscribe, and I did not want to 
let it sit there without an reply.

Best,
-- 
Martin

signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Sam Hartman]

> "Simon" == Simon Richter  writes:


Simon> Adoption of systemd on machines with popcon installed and
Simon> active, which are largely desktop and laptop installations,
Simon> i.e. those cases where systemd provides a tangible benefit.

Simon, I've seen your analysis of why systemd doesn't provide
significant benefits to server installations.
And yet, people also disagree with you and believe that systemd is
valuable for servers.
I've actually received multiple mails from people claiming that by
moving to systemd we are focusing on enterprise or server deployments at
the expense of the sorts of environments where you claim systemd has
value.

I understand that there's something you get at when you claim that
systemd has no value on servers.
And yet, phrasing things that way leaves no room for those who disagree
with you.
I'd appreciate it if you could come up with a phrasing that didn't state
that as fact.
"where I don't see the benefit of systemd," would be one simple solution
if that works for you.

Re: Heads up: persistent journal has been enabled in systemd

[Michael Stone]

On Thu, Feb 06, 2020 at 07:22:07PM +0100, Svante Signell wrote:

On a Debian sytem _not_ running systemd:

du -sh /var/log
74M /var/log

And the binary logs from systemd would of course be much smaller since
they are binary. Any numbers?


It looks like you just proved that this discussion doesn't matter for 
your use case. Can we stop this now?

Re: Heads up: persistent journal has been enabled in systemd

[Michael Stone]

On Thu, Feb 06, 2020 at 04:49:31PM +0100, Simon Richter wrote:

I'd expect servers and embedded systems to be vastly underrepresented in
both of these statistics, but that doesn't mean these use cases are in any
way uninteresting to the project.


Please stop beating the dead horse of whether debian is going to use 
systemd. FWIW, I find it even more useful on servers than on single-user 
machines and have it installed on orders of magnitude more servers than 
single-user machines. IOW, you're simply projecting your own prejudices 
and continuing to derail this thread into another pointless discussion 
about the merits of systemd.

Re: Heads up: persistent journal has been enabled in systemd

[Ansgar]

Marco d'Itri writes:
> On Feb 06, Simon Richter  wrote:
>
>> I'd expect servers and embedded systems to be vastly underrepresented in
>> both of these statistics, but that doesn't mean these use cases are in any
>> way uninteresting to the project.
> If you want to argue that the adoption of popcon is not uniform among 
> different types of systems then you need to prove it.
> Expectations and anedoctes are not statistics.

That's likely given some libreoffice packages (unlikely to be installed
on servers) are at ~40% popcon and I would expect significantly more
server installations than desktop ones.

So popcon might overestimate sysvinit usage and it might in reality be
lower over the total installation base.

Ansgar

Re: Heads up: persistent journal has been enabled in systemd

[Svante Signell]

On Thu, 2020-02-06 at 16:09 +, Philip Hands wrote:
> > > I solved this by removing Systemd from my systems.
> > > 
> > > And now what?
> Well, since we're apparently meant to be obsessed about what it is
> like to accept every default, then let's assume for a moment that you
> are asking about rsyslog with the maintainer supplied rsyslog.conf.
> 
> Having just installed this laptop, I can run this:
> 
> # grep -r '5590629f1f60.*delivery successful' /var/log
> /var/log/mail.log:Feb  6 14:51:14 rummy dma[164f55.5590629f1f60]:
> delivery successful
> /var/log/mail.log:Feb  6 14:51:14 rummy dma[164f55.5590629f1f60]:
> delivery successful
> /var/log/mail.info:Feb  6 14:51:14 rummy dma[164f55.5590629f1f60]:
> delivery successful
> /var/log/syslog:Feb  6 14:51:14 rummy dma[164f55.5590629f1f60]:
> delivery successful
> which you'll note is actually triplicating log messages by default.

On a Debian sytem _not_ running systemd:

du -sh /var/log
74M /var/log

And the binary logs from systemd would of course be much smaller since
they are binary. Any numbers?

HTH!

Re: Heads up: persistent journal has been enabled in systemd

[Svante Signell]

On Thu, 2020-02-06 at 15:08 +0100, Marco d'Itri wrote:
> On Feb 06, Svante Signell  wrote:
> 
> > There are still a large number of
> > Debian users opting away from using systemd (and still use Debian,
> > not
> > derivatives). And what about non-linux systems?
> This is not true: adoption of systemd in buster is larger than 99%.
> Other systems will have different defaults.

I see that you are trying to heal the wounds Sam is talking about. Good
luck :D 

Re: Heads up: persistent journal has been enabled in systemd

[Marco d'Itri]

On Feb 06, Simon Richter  wrote:

> I'd expect servers and embedded systems to be vastly underrepresented in
> both of these statistics, but that doesn't mean these use cases are in any
> way uninteresting to the project.
If you want to argue that the adoption of popcon is not uniform among 
different types of systems then you need to prove it.
Expectations and anedoctes are not statistics.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Matt Zagrabelny]

On Thu, Feb 6, 2020 at 11:03 AM Matthias Klumpp  wrote:
>
> >From personal experience, all that's needed to switch to the journal
> for an admin is to re-learn a couple of commands and be open to a bit
> of change. I so far found nothing that I could do with rsyslog to be
> impossible with the journal.

Additionally you don't have to guess where logs are being sent for a
particular service:

systemd: journalctl --since=-1h -u puppet

non-systemd: grep puppet /var/log/*
though that is inefficient and would possibly miss some data or give
false positives.

-m

Re: Heads up: persistent journal has been enabled in systemd

[Anthony DeRobertis]

On 2/6/20 6:26 AM, Martin Steigerwald wrote:

I solved this by removing Systemd from my systems.

And now what?


Then you're not running the default configuration, which is of course 
perfectly fine. As part of switching from systemd to your init system of 
choice, you'd also install your favorite non-journald logger.


I, personally, think the systemd maintainers here have made the right 
choice to enable the persistent journal by default. And that it really 
doesn't make sense to have two copies of the logs written to disk, 
increase the attack surface with another daemon, etc., so that yes, 
eventually rsyslog shouldn't be in the default install, even though I'll 
probably keep it installed on most of my machines for a while. I'm 
confident they're working hard to improve Debian, even and especially 
when I personally disagree with a decision, and thank them for that.


Finally, I believe you want Debian to continue to be a distro where 
admins have choice as to what runs on their systems. I believe the best 
way you could help to achieve this is to present easy technical 
solutions to make that choice possible. For example:


   If rsyslog is no longer installed by default, non-systemd init
   systems should add a Recommends: system-log-daemon, so that
   switching to one of them would (by default) pull in a syslogd.

Maybe that's not quite the right technical solution, but it'd be a much 
more pleasant and productive conversation to iterate on that than this 
conversation has been.

Re: Heads up: persistent journal has been enabled in systemd

[Matthias Klumpp]

Am Do., 6. Feb. 2020 um 17:12 Uhr schrieb Simon Richter :
>
> Hi Marco,
>
> On Thu, Feb 06, 2020 at 03:08:28PM +0100, Marco d'Itri wrote:
>
> > > There are still a large number of
> > > Debian users opting away from using systemd (and still use Debian, not
> > > derivatives). And what about non-linux systems?
>
> > This is not true: adoption of systemd in buster is larger than 99%.
> > Other systems will have different defaults.
>
> Adoption of systemd on machines with popcon installed and active, which are
> largely desktop and laptop installations, i.e. those cases where systemd
> provides a tangible benefit.
>
> Popcon is useful for determining what goes on the first installation DVD,
> but neither popcon nor mirror download statistics can measure the impact a
> particular package has.
>
> I'd expect servers and embedded systems to be vastly underrepresented in
> both of these statistics, but that doesn't mean these use cases are in any
> way uninteresting to the project.

Those are also the usecases where defaults matter the least though. We
can certainly expect administrators of a server to be able to `apt
install rsyslog` if they want to use it. On embedded systems I
personally actually found the systemd journal to be very nice to use,
but embedded systems are the most customized Debian installations out
there, so we can't choose a default that works for all of them anyway.

>From personal experience, all that's needed to switch to the journal
for an admin is to re-learn a couple of commands and be open to a bit
of change. I so far found nothing that I could do with rsyslog to be
impossible with the journal.

Cheers,
Matthias

-- 
I welcome VSRE emails. See http://vsre.info/

Re: Heads up: persistent journal has been enabled in systemd

[Sam Hartman]

> "Martin" == Martin Steigerwald  writes:

Martin> Well, that is *exactly* why I thought the GR is not going to
Martin> be helpful.

Martin> Cause in *no way* it appeared to have *solved* the conflict
Martin> underneath it.

No the GR did not somehow magically cause people to agree.
It did decide what Debian is going to do.
That's already had significant positive impacts even on this thread.

The systemd maintainers felt comfortable announcing the persistent
journal change and engaging in a discussion of whether syslogd should be
installed by default.

That is, we actually moved forward enough that the systemd maintainers
felt comfortable enough engaging with the community.
I've seen the same thing on a number of other fronts: people feel like
they have enough of an answer that they pull their heads out of the hole
they have been hiding in for years and *engage with the community*.

That's amazing; that's wonderful.

We actually did decide that we're focusing our default on systemd
systems for Debian.
Yes, we maintain the technologies to explore and build alternatives.
But in terms of defaults and where we focus our defaults, we actually
did decide.

If you find that Debian's focus is no longer aligned with yours, I wish
you the best.
If you still feel like you want to be involved in packaging, I hope you
do that.

--Sam

Re: Heads up: persistent journal has been enabled in systemd

[Philip Hands]

Martin Steigerwald  writes:

> Martin Steigerwald - 06.02.20, 12:26:32 CET:
>> Vincent Bernat - 06.02.20, 07:58:32 CET:
>> >  ❦  6 février 2020 09:50 +11, Dmitry Smirnov :
>> > >> and 2) continuing to use rsyslog isn't an option if the default
>> > >> changes.>
>> > >
>> > > No. I just don't want default to change. IMHO rationale for this
>> > > is
>> > > weak but everybody keeps arguing that it would not be a big deal.
>> > > In time we will see how that goes (what could possibly go wrong?)
>> > > but why do the change in first place?
>> >
>> > To not have logs duplicated in two places.
>>
>> I solved this by removing Systemd from my systems.
>>
>> And now what?
>
> And before that I had Systemd only log to /run.
>
> Especially as I found that I did not use journalctl in my daily practice
> anyway.
>
> So why would removing rsyslog from the default install the only viable
> approach to solve duplicate logging?

Well, since we're apparently meant to be obsessed about what it is like
to accept every default, then let's assume for a moment that you are
asking about rsyslog with the maintainer supplied rsyslog.conf.

Having just installed this laptop, I can run this:

# grep -r '5590629f1f60.*delivery successful' /var/log
/var/log/mail.log:Feb  6 14:51:14 rummy dma[164f55.5590629f1f60]: delivery 
successful
/var/log/mail.info:Feb  6 14:51:14 rummy dma[164f55.5590629f1f60]: delivery 
successful
/var/log/syslog:Feb  6 14:51:14 rummy dma[164f55.5590629f1f60]: delivery 
successful

which you'll note is actually triplicating log messages by default.

HTH

Cheers, Phil.
--
|)|  Philip Hands  [+44 (0)20 8530 9560]  HANDS.COM Ltd.
|-|  http://www.hands.com/http://ftp.uk.debian.org/
|(|  Hugo-Klemm-Strasse 34,   21075 Hamburg,GERMANY


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Simon Richter]

Hi Marco,

On Thu, Feb 06, 2020 at 03:08:28PM +0100, Marco d'Itri wrote:

> > There are still a large number of
> > Debian users opting away from using systemd (and still use Debian, not
> > derivatives). And what about non-linux systems?

> This is not true: adoption of systemd in buster is larger than 99%.
> Other systems will have different defaults.

Adoption of systemd on machines with popcon installed and active, which are
largely desktop and laptop installations, i.e. those cases where systemd
provides a tangible benefit.

Popcon is useful for determining what goes on the first installation DVD,
but neither popcon nor mirror download statistics can measure the impact a
particular package has.

I'd expect servers and embedded systems to be vastly underrepresented in
both of these statistics, but that doesn't mean these use cases are in any
way uninteresting to the project.

   Simon

Re: Heads up: persistent journal has been enabled in systemd

[Simon Richter]

Hi,

On Thu, Feb 06, 2020 at 02:32:57PM +0100, Martin Steigerwald wrote:

> Also it does not (yet) mean that I abandoned my packaging efforts for it.

Same here. My packages don't care about systemd either way, so there is no
reason not to package them at the root of the distribution tree.

> But of course, if you like to get rid of contributors, feel free to 
> pursue this kind of "go away already!" attitude.

Agreed. It wasn't helpful during the GR debate, and it's not helpful now.

   Simon

amen (was Re: Heads up: persistent journal has been enabled in systemd)

[Holger Levsen]

On Wed, Feb 05, 2020 at 09:28:27PM -0500, Michael Stone wrote:
> > No. I just don't want default to change.
> Nothing is worse for the debian project than for every single suggestion
> that something change be met with opposition that boils down to opposition
> to change. Things have changed before, they will change again, and honestly
> the impact of most of those changes is very small (for either better or
> worse) and certainly they are not things that should be met with the level
> of emotional response that they are. If anything has changed for the worse
> in the project it's this sense we aren't allowed to change things anymore.

amen to that.

this thread is (mostly) really awful (and the proposed change just fine, IMO).


-- 
cheers,
Holger

---
   holger@(debian|reproducible-builds|layer-acht).org
   PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C

There are no jobs on a dead planet. (Also many other things but people mostly
seem to care about jobs.)


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Marco d'Itri]

On Feb 06, Svante Signell  wrote:

> There are still a large number of
> Debian users opting away from using systemd (and still use Debian, not
> derivatives). And what about non-linux systems?
This is not true: adoption of systemd in buster is larger than 99%.
Other systems will have different defaults.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Stephan Seitz]

On Do, Feb 06, 2020 at 13:25:06 +0100, Ansgar wrote:

Given you wrote earlier that you moved all but one of your machines
away from Debian, whatever Debian installs by default doesn't affect
you anyway.


Well, I still use Debian. In Testing you have elogind now as a complete 
systemd replacement. So I’m certainly not interested in too much 
activated systemd features, that could make things more difficult



I have no problem installing a different MTA than Debian's default
(exim), my preferred shell, my preferred editor and so on either.


As long as you can do it, it’s fine. But you’ll certainly find more MTA 
or editors in the Debian archive than init systems…


Shade and sweet water!

Stephan

--
|If your life was a horse, you'd have to shoot it.|

Re: Heads up: persistent journal has been enabled in systemd

[Martin Steigerwald]

Martin Steigerwald - 06.02.20, 14:32:57 CET:
> Ansgar - 06.02.20, 13:25:06 CET:
> > On Thu, 2020-02-06 at 12:30 +0100, Martin Steigerwald wrote:
> > > Especially as I found that I did not use journalctl in my daily
> > > practice anyway.
> > 
> > Given you wrote earlier that you moved all but one of your machines
> > away from Debian, whatever Debian installs by default doesn't affect
> > you anyway.
> > 
> > I don't write on mailing lists of distributions I don't use (or only
> > use a bit) arguing about what is included in their default
> > installation, though one could argue that emacs should be installed
> > everywhere by default.
> 
> That I do not use Debian anymore on all except one of my system does
> not mean that I still feel bonds with the project, especially some
> teams of it, like the Debian Qt KDE team.
> 
> Also it does not (yet) mean that I abandoned my packaging efforts for
> it.
> 
> But of course, if you like to get rid of contributors, feel free to
> pursue this kind of "go away already!" attitude.
> 
> To be very clear: After your comment I feel *unwelcome* here.

Okay, it is time to draw a conclusion out of this:

I choose not to repeat the same pattern as with previous discussions.

I have seen how people have been made unwelcome on debian-devel just for 
*sharing* their concerns and points of view quite some times here.

So I leave, for now at least.

I feel unwelcome here.

Now I welcome and accept that feeling.

If you have nothing constructive or supportive about this please refrain 
from contacting me. I may just delete any non-supportive mail without 
any notification.

I am at a loss how Debian can resolve the underlying conflict at the 
moment. I see how the GR certainly did not achieve this.

Whether I will leave Debian altogether and/or abandon my packaging work 
is not up for decision today for me. I do not make such a decision 
lightly, especially after the personal bonds and relationships I have 
with some of the contributors to this project.

But for now participating in discussions on debian-devel does not appear 
to be a healthy use of my time. So I stop doing that.

For now at least.

Best of success for moving through to a better outcome for everyone 
involved. For now, I am out.

Best,
-- 
Martin

signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Martin Steigerwald]

Ansgar - 06.02.20, 13:25:06 CET:
> On Thu, 2020-02-06 at 12:30 +0100, Martin Steigerwald wrote:
> > Especially as I found that I did not use journalctl in my daily
> > practice anyway.
> 
> Given you wrote earlier that you moved all but one of your machines
> away from Debian, whatever Debian installs by default doesn't affect
> you anyway.
> 
> I don't write on mailing lists of distributions I don't use (or only
> use a bit) arguing about what is included in their default
> installation, though one could argue that emacs should be installed
> everywhere by default.

That I do not use Debian anymore on all except one of my system does not 
mean that I still feel bonds with the project, especially some teams of 
it, like the Debian Qt KDE team.

Also it does not (yet) mean that I abandoned my packaging efforts for it.

But of course, if you like to get rid of contributors, feel free to 
pursue this kind of "go away already!" attitude.

To be very clear: After your comment I feel *unwelcome* here.

If Debian is to become a Systemd only distribution, by all means, then I 
bet it is time for me to leave. *sigh*

If there is still willingness to resolve the underlying conflict in a way 
that everyone can win, I still like to be in and help to move that along 
as best as I can.

> > So why would removing rsyslog from the default install the only
> > viable approach to solve duplicate logging?
> 
> If the rsyslog maintainer is also fine with this (which I'll assume he
> is), I think there's no real problem if that would be done.  I might

Can we for starters agree on not *assuming* things? Especially when it 
comes to issues people feel strongly about?

> install rsyslog by hand on a few systems then, but that's not a real
> problem given I already have to install other packages anyway...
> 
> I have no problem installing a different MTA than Debian's default
> (exim), my preferred shell, my preferred editor and so on either.

I usually install Postfix everywhere, but… does that somehow remove my 
right to voice my concern about removing rsyslog from the default 
install of Debian here?

Is this mailing list a place where people are not allowed to state their 
concern and their preferences anymore? Where they receive "shut up" 
messages when they try to do so?

I have been contacted off list in a supportive way already. From someone 
who apparently does not like to raise his voice on this list anymore.

Ciao,
-- 
Martin

Re: Heads up: persistent journal has been enabled in systemd

[Ansgar]

On Thu, 2020-02-06 at 12:30 +0100, Martin Steigerwald wrote:
> Especially as I found that I did not use journalctl in my daily
> practice anyway.

Given you wrote earlier that you moved all but one of your machines
away from Debian, whatever Debian installs by default doesn't affect
you anyway.

I don't write on mailing lists of distributions I don't use (or only
use a bit) arguing about what is included in their default
installation, though one could argue that emacs should be installed
everywhere by default.

> So why would removing rsyslog from the default install the only
> viable approach to solve duplicate logging?

If the rsyslog maintainer is also fine with this (which I'll assume he
is), I think there's no real problem if that would be done.  I might
install rsyslog by hand on a few systems then, but that's not a real
problem given I already have to install other packages anyway...

I have no problem installing a different MTA than Debian's default
(exim), my preferred shell, my preferred editor and so on either.

Ansgar

Re: Heads up: persistent journal has been enabled in systemd

[Martin Steigerwald]

Martin Steigerwald - 06.02.20, 12:26:32 CET:
> Vincent Bernat - 06.02.20, 07:58:32 CET:
> >  ❦  6 février 2020 09:50 +11, Dmitry Smirnov :
> > >> and 2) continuing to use rsyslog isn't an option if the default
> > >> changes.>
> > > 
> > > No. I just don't want default to change. IMHO rationale for this
> > > is
> > > weak but everybody keeps arguing that it would not be a big deal.
> > > In time we will see how that goes (what could possibly go wrong?)
> > > but why do the change in first place?
> > 
> > To not have logs duplicated in two places.
> 
> I solved this by removing Systemd from my systems.
> 
> And now what?

And before that I had Systemd only log to /run.

Especially as I found that I did not use journalctl in my daily practice 
anyway.

So why would removing rsyslog from the default install the only viable 
approach to solve duplicate logging?

-- 
Martin

Re: Heads up: persistent journal has been enabled in systemd

[Martin Steigerwald]

Vincent Bernat - 06.02.20, 07:58:32 CET:
>  ❦  6 février 2020 09:50 +11, Dmitry Smirnov :
> >> and 2) continuing to use rsyslog isn't an option if the default
> >> changes.> 
> > No. I just don't want default to change. IMHO rationale for this is
> > weak but everybody keeps arguing that it would not be a big deal.
> > In time we will see how that goes (what could possibly go wrong?)
> > but why do the change in first place?
> 
> To not have logs duplicated in two places.

I solved this by removing Systemd from my systems.

And now what?

-- 
Martin

Re: Heads up: persistent journal has been enabled in systemd

[Martin Steigerwald]

Scott Kitterman - 06.02.20, 06:27:44 CET:
> >Are you suggesting that voters fully understood the implications?
> >Is this OK now to replace everything with systemd counterparts?
> >
> >I certainly voted with considerations for _init_ system.
> >
> >If I recall correctly, no GR option suggested to "use as much systemd
> >components as possible" or I think the outcome of GR could have been
> >different...
> >
> >Anyway the big disadvantage of changing default is that now random
> >Debian
> >systems will have no traditional logging interface (rsyslog) and
> >we're all
> >will be forced to adapt to the new interface in the absence of old
> >one on
> >some systems...
> 
> I think you need to go reread the option that won.
> 
> I make no claims about what others understood or didn't, but to me
> this is exactly the kind of thing that the winning option makes
> likely.  That doesn't mean I like it, but we had a GR, now we're
> stuck with it.

Well, that is *exactly* why I thought the GR is not going to be helpful. 

Cause in *no way* it appeared to have *solved* the conflict underneath 
it.

I'd still love to see otherwise. But right now I see the same "I am 
right and you are wrong" pattern arise in this discussion as before. And 
also the same meta discussion as in "you are not discussing it in the 
right way or proper way" code of conduct related thing. As if the GR did 
not even have concluded. The GR was still within a win-loose setting and 
here it shows.

The GR did *not* change any of the involved people and their behavior. 
And my bet is, that it won't do that in the future anyway.

That written: I do not care at all whether Systemd by default stores the 
journal persistently or not. Especially as I am not even using it on my 
own systems anymore.

But I do not like rsyslog to be removed from the default install. It 
might be a nice idea to improve its default configuration here and there, 
but only with good justification. So for example SUSE has stopped using 
those insufficient time stamps with SLE 12 already. I know there is a 
compatibility bug somewhere… but if log reading software has not yet 
adapted to modern standard conform timestamps, I'd not let that hold 
back progress. I am considering to comment out that 
RSYSLOG_TraditionalFileFormat line, cause really let that old insufficient 
time stamps be gone already. I say this with my log analysis trainer hat 
on. With the same hat on I say… I still like to use rsyslog. I can 
process that with standard tools I can use with any other text file 
instead of having to learn a ton of options of journalctl. Simplicity as 
a goal. I love to see that come back to software development! And 
neither Logstash nor Fluentd or Graylog have issues dealing with those 
text files and manage to dig out the relevant information out of them 
just fine.

Mind you, as far as I am aware neither SUSE nor Red Hat dared to remove 
rsyslog from their enterprise offerings so far. Just imagine the uproar 
this would cause! I know Fedora has it removed, but CentOS 7 certainly 
has not and neither SLE 12 and 15. I am not completely sure about CentOS 
8 cause I did not consciously check, but I believe I have seen rsyslog 
running there as well.

Best,
-- 
Martin

Re: Heads up: persistent journal has been enabled in systemd

[Vincent Bernat]

 ❦  6 février 2020 10:45 +01, Svante Signell :

>> To not have logs duplicated in two places.
>
> If this is your motivation for the change it is a _very_ weak one, right? Disk
> space is not a crucial problem anymore. Additionally, what would be the 
> defaults
> for non-systemd systems running GNU/Linux? There are still a large number of
> Debian users opting away from using systemd (and still use Debian, not
> derivatives). And what about non-linux systems?

For individual systems, not running an additional daemon and not wasting
space on disk doesn't seem to be that a weak motivation. On my
workstations, I am disabling the syslog daemon since a long time since I
prefer browsing logs through journalctl and its superior abilities for
filtering (grep, date ranges, services).

People can still install the syslog daemon on their choices. On servers,
I prefer syslog-ng to rsyslog and I configure it to use journald as its
source for local logs as it gives me access to structured logs and more
information.

On non-Linux systems, a arch-specific package could just pull back
rsyslog.
-- 
A man was reading The Canterbury Tales one Saturday morning, when his
wife asked "What have you got there?"  Replied he, "Just my cup and
Chaucer."


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Svante Signell]

On Thu, 2020-02-06 at 07:58 +0100, Vincent Bernat wrote:
>  ❦  6 février 2020 09:50 +11, Dmitry Smirnov :
> 
> > > and 2) continuing to use rsyslog isn't an option if the default changes.
> > 
> > No. I just don't want default to change. IMHO rationale for this is weak
> > but everybody keeps arguing that it would not be a big deal. In time we will
> > see how that goes (what could possibly go wrong?) but why do the change in
> > first place?
> 
> To not have logs duplicated in two places.

If this is your motivation for the change it is a _very_ weak one, right? Disk
space is not a crucial problem anymore. Additionally, what would be the defaults
for non-systemd systems running GNU/Linux? There are still a large number of
Debian users opting away from using systemd (and still use Debian, not
derivatives). And what about non-linux systems?

Thank you for your time! 

Re: Heads up: persistent journal has been enabled in systemd

[Vincent Bernat]

 ❦  6 février 2020 09:50 +11, Dmitry Smirnov :

>> and 2) continuing to use rsyslog isn't an option if the default changes.
>
> No. I just don't want default to change. IMHO rationale for this is weak but 
> everybody keeps arguing that it would not be a big deal. In time we will see 
> how that goes (what could possibly go wrong?) but why do the change in first 
> place?

To not have logs duplicated in two places.
-- 
Use statement labels that mean something.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Scott Kitterman]

On February 5, 2020 8:44:43 PM UTC, Dmitry Smirnov  wrote:
>On Wednesday, 5 February 2020 11:01:08 AM AEDT Scott Kitterman wrote:
>> We just had a GR where the project voted it was just fine to systemd
>all
>> the things, so this sort of thing is to be expected.
>
>Are you suggesting that voters fully understood the implications?
>Is this OK now to replace everything with systemd counterparts?
>
>I certainly voted with considerations for _init_ system.
>
>If I recall correctly, no GR option suggested to "use as much systemd 
>components as possible" or I think the outcome of GR could have been 
>different...
>
>Anyway the big disadvantage of changing default is that now random
>Debian 
>systems will have no traditional logging interface (rsyslog) and we're
>all 
>will be forced to adapt to the new interface in the absence of old one
>on 
>some systems...

I think you need to go reread the option that won.

I make no claims about what others understood or didn't, but to me this is 
exactly the kind of thing that the winning option makes likely.  That doesn't 
mean I like it, but we had a GR, now we're stuck with it.

Scott K

Re: Heads up: persistent journal has been enabled in systemd

[Scott Kitterman]

On February 5, 2020 8:43:43 AM UTC, Bastian Blank  wrote:
>Hi Scott
>
>On Wed, Feb 05, 2020 at 07:44:25AM +, Scott Kitterman wrote:
>> >> Of course the fact that I can't use all the tools available to
>> >manipulate text 
>> >> files to follow or analyze logs is problematic.  If I'm using
>> >journalctl, how 
>> >> do I replicate 'tail -f /var/log/mail.loog'?
>> >
>> >journalctl -f SYSLOG_IDENTIFIER=2
>> >
>> Do syslog facilities really have to be addressed by number rather
>than name?  That seems like a horrible interface.
>
>That's because you asked a pretty specific question and you've been
>told the exact replacement.
>
>However, you most likely don't actually want to tail this specific
>file.
>It is a solution for a more abstract problem, e.g. "I want to see logs
>from Postfix".  And this question got a different answer.

Yes, it's been demonstrated that what I'm after won't be impossible with 
journalctl, only far more difficult.  And yes, I quite frequently find myself 
tailing that specific file because it has exactly the information I need.  
Please don't explain to me what I want like you understand it better than I do.

I understand the premise that journalctl offers the promise of better 
interfaces to system logs.  I think it's a long way from that now and I don't 
think my uses of system logs are particularly niche.

Scott K

Re: Heads up: persistent journal has been enabled in systemd

[Michael Stone]

On Thu, Feb 06, 2020 at 09:50:36AM +1100, Dmitry Smirnov wrote:

On Thursday, 6 February 2020 9:04:33 AM AEDT Michael Stone wrote:

Nobody said "exclusively" except you!


It was suggested that default will change and I'm concerned about that.


And yet you said "exclusively" and generally argued in a confrontational 
manner that seems unwilling to accept a minor change because you just 
don't want to consider use cases other than your own habits.



Either option has benefits and
disadvantages, but for some reason you're arguing as though 1) only your
particular use cases matter


Nonsense. It is not "my case" but established default that remained stable
for as long as I can remember. That would be for about a decade.


Well, my memory goes back more than twice as long and I remember it 
changing before. I remember not being particularly happy with rsyslog 
when it became the default, but it didn't really matter that much so it 
didn't particularly bother me either. rsyslog was (and is) one of 
*several* alternatives, and has advantages and disadvantages compared to 
each of them.



and 2) continuing to use rsyslog isn't an option if the default changes.


No. I just don't want default to change.


Nothing is worse for the debian project than for every single suggestion 
that something change be met with opposition that boils down to 
opposition to change. Things have changed before, they will change 
again, and honestly the impact of most of those changes is very small 
(for either better or worse) and certainly they are not things that 
should be met with the level of emotional response that they are. If 
anything has changed for the worse in the project it's this sense we 
aren't allowed to change things anymore.

Re: Heads up: persistent journal has been enabled in systemd

[Dmitry Smirnov]

On Thursday, 6 February 2020 9:04:33 AM AEDT Michael Stone wrote:
> Nobody said "exclusively" except you!

It was suggested that default will change and I'm concerned about that.


> Either option has benefits and
> disadvantages, but for some reason you're arguing as though 1) only your
> particular use cases matter

Nonsense. It is not "my case" but established default that remained stable 
for as long as I can remember. That would be for about a decade.


> and 2) continuing to use rsyslog isn't an option if the default changes.

No. I just don't want default to change. IMHO rationale for this is weak but 
everybody keeps arguing that it would not be a big deal. In time we will see 
how that goes (what could possibly go wrong?) but why do the change in first 
place?

I'm resting the case as I have nothing left to add...

-- 
Regards,
 Dmitry Smirnov.

---

A wise man proportions his belief to the evidence.
-- David Hume, "An Inquiry Concerning Human Understanding"


signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Martin Steigerwald]

Andrey Rahmatullin - 05.02.20, 22:17:14 CET:
> On Thu, Feb 06, 2020 at 07:44:43AM +1100, Dmitry Smirnov wrote:
> > > We just had a GR where the project voted it was just fine to
> > > systemd all the things, so this sort of thing is to be expected.
> > 
> > Are you suggesting that voters fully understood the implications?
> > Is this OK now to replace everything with systemd counterparts?
> 
> Yes.
> 
> > I certainly voted with considerations for _init_ system.
> > 
> > If I recall correctly, no GR option suggested to "use as much
> > systemd
> > components as possible" or I think the outcome of GR could have been
> > different...
> 
> The winning option says "Packages may use any systemd facility at the
> package maintainer's discretion, provided that this is consistent with
> other Policy requirements and the normal expectation that packages
> shouldn't depend on experimental or unsupported (in Debian) features
> of other packages".
> You can read it at https://www.debian.org/vote/2019/vote_002#textb

Hmmm, for me this still reads different than: "Ok, let's change the 
default logging system."

Cause if I understood above wording correctly, then it means a package 
may use a Systemd feature. But this would only affect this one package. 
However here a system wide default of the system is changed.

Thanks,
-- 
Martin

Re: Heads up: persistent journal has been enabled in systemd

[Michael Stone]

On Thu, Feb 06, 2020 at 08:40:29AM +1100, Dmitry Smirnov wrote:

On Thursday, 6 February 2020 6:59:38 AM AEDT Nikolaus Rath wrote:

I would venture that for every user who is grateful that
/var/log/mail.log collects all the various mail-related logs, there is
another user that curses about non being able to separate (out of the
box) the logs from all the programs that consider themselves
mail-related, and another user who struggles to remember in which
logfile $DIFFICULT_TO_CLASSIFY_PROGRAM might be writing its logs.


There are log readers like "lnav" and "multitail" that will become useless
without traditional log files. "lnav" tails multiple logs by default and IMHO
provides a very useful interface.

Exclusively relying on systemd logging facilities limits ways in which we can
analyse logs.


Nobody said "exclusively" except you! Either option has benefits and 
disadvantages, but for some reason you're arguing as though 1) only your 
particular use cases matter and 2) continuing to use rsyslog isn't an 
option if the default changes.

Re: Heads up: persistent journal has been enabled in systemd

[Russ Allbery]

Dmitry Smirnov  writes:

> There are log readers like "lnav" and "multitail" that will become
> useless without traditional log files. "lnav" tails multiple logs by
> default and IMHO provides a very useful interface.

> Exclusively relying on systemd logging facilities limits ways in which
> we can analyse logs.

apt-get install rsyslog

Debian, of all projects, should be more relaxed about defaults, given that
we're the OS of choice for people who like to customize things.  I've been
using Debian for some 20 years and don't believe I have *ever* accepted
all of the defaults or left a system in a purely default mode for longer
than about five minutes.

No one is going to prevent you from using rsyslog everywhere that you want
to use rsyslog.

-- 
Russ Allbery (r...@debian.org)  

Re: Heads up: persistent journal has been enabled in systemd

[Dmitry Smirnov]

On Thursday, 6 February 2020 6:59:38 AM AEDT Nikolaus Rath wrote:
> I would venture that for every user who is grateful that
> /var/log/mail.log collects all the various mail-related logs, there is
> another user that curses about non being able to separate (out of the
> box) the logs from all the programs that consider themselves
> mail-related, and another user who struggles to remember in which
> logfile $DIFFICULT_TO_CLASSIFY_PROGRAM might be writing its logs.

There are log readers like "lnav" and "multitail" that will become useless 
without traditional log files. "lnav" tails multiple logs by default and IMHO 
provides a very useful interface.

Exclusively relying on systemd logging facilities limits ways in which we can 
analyse logs.

-- 
All the best,
 Dmitry Smirnov.

---

That which has always been accepted by everyone, everywhere, is almost
certain to be false.
-- Paul Valery


signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Didier 'OdyX' Raboud]

Le mercredi, 5 février 2020, 21.44:43 h CET Dmitry Smirnov a écrit :
> On Wednesday, 5 February 2020 11:01:08 AM AEDT Scott Kitterman wrote:
> > We just had a GR where the project voted it was just fine to systemd all
> > the things, so this sort of thing is to be expected.
> 
> Are you suggesting that voters fully understood the implications?

Let me suggest (out of reading what they wrote in other parts of the thread), 
that I read Scott's message as a mix of despair and sarcasm.

> Is this OK now to replace everything with systemd counterparts?
> 
> I certainly voted with considerations for _init_ system.

The vote was named "init systems and systemd".

> If I recall correctly, no GR option suggested to "use as much systemd
> components as possible" or I think the outcome of GR could have been
> different...

This was option "Choice 1: F: Focus on systemd".

The option that won, "Choice 2: B: Systemd but we support exploring 
alternatives" [0] said:
> Packages may use any systemd facility at the package maintainer's
> discretion, provided that this is consistent with other Policy requirements
> and the normal expectation that packages shouldn't depend on experimental or
> unsupported (in Debian) features of other packages.

Of course, it's ironic to read that under this project's opinion; "src:systemd 
may use any systemd facility at systemd maintainer's discretion".

But, irony aside, we _did_ decide that we were to allow systemd facilities' 
usage, and moving ahead with journal's persistent storage seems to me very 
much inline with the project's latest GR-backed opinion.

I'm not saying the project decided "journald should replace rsyslog" though. 
I'm saying that activating the persistent storage in journald feels very much 
inline with the project's expressed opinion. Also, it is orthogonal to whether 
we should be demoting rsyslog, or to what would happen to it upon upgrade.

OdyX

[0] https://www.debian.org/vote/2019/vote_002#textb

Re: Heads up: persistent journal has been enabled in systemd

[Matt Zagrabelny]

On Wed, Feb 5, 2020 at 3:03 PM Dmitry Smirnov  wrote:

>
> Anyway the big disadvantage of changing default is that now random Debian
> systems will have no traditional logging interface (rsyslog) and we're all
> will be forced to adapt to the new interface in the absence of old one on
> some systems...
>

>From Michael's initial email:

"Depending on how it goes, I might ask the ftp-masters to lower the
priority of rsyslog from important to optional, so it would no longer be
installed by default on new bullseye installations."

As of right now, the only change is to enable persistent journal logs.

Text logs will still be there with the current systemd upload.

-m

Re: Heads up: persistent journal has been enabled in systemd

[Andrey Rahmatullin]

On Thu, Feb 06, 2020 at 07:44:43AM +1100, Dmitry Smirnov wrote:
> > We just had a GR where the project voted it was just fine to systemd all
> > the things, so this sort of thing is to be expected.
> 
> Are you suggesting that voters fully understood the implications?
> Is this OK now to replace everything with systemd counterparts?
Yes.

> I certainly voted with considerations for _init_ system.
> 
> If I recall correctly, no GR option suggested to "use as much systemd 
> components as possible" or I think the outcome of GR could have been 
> different...
The winning option says "Packages may use any systemd facility at the
package maintainer's discretion, provided that this is consistent with
other Policy requirements and the normal expectation that packages
shouldn't depend on experimental or unsupported (in Debian) features of
other packages".
You can read it at https://www.debian.org/vote/2019/vote_002#textb

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Dmitry Smirnov]

On Wednesday, 5 February 2020 11:01:08 AM AEDT Scott Kitterman wrote:
> We just had a GR where the project voted it was just fine to systemd all
> the things, so this sort of thing is to be expected.

Are you suggesting that voters fully understood the implications?
Is this OK now to replace everything with systemd counterparts?

I certainly voted with considerations for _init_ system.

If I recall correctly, no GR option suggested to "use as much systemd 
components as possible" or I think the outcome of GR could have been 
different...

Anyway the big disadvantage of changing default is that now random Debian 
systems will have no traditional logging interface (rsyslog) and we're all 
will be forced to adapt to the new interface in the absence of old one on 
some systems...

-- 
Cheers,
 Dmitry Smirnov.

---

Believing is easier than thinking. Hence so many more believers than
thinkers.
-- Bruce Calvert


signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Nikolaus Rath]

On Feb 05 2020, Scott Kitterman  wrote:
> On February 5, 2020 12:35:45 PM UTC, Ansgar  wrote:
>>On Wed, 2020-02-05 at 07:44 +, Scott Kitterman wrote:
>>> Do syslog facilities really have to be addressed by number rather
>>than name?  That seems like a horrible interface.
>>
>>Currently yes.  There is an upstream bug asking for a way to specify
>>them by name[1], but nobody implemented it yet.
>>
>>I think a `--facility` option should be fairly easy to implement.  Just
>>adapt some code from the existing `--identifier` and `--priority`
>>options, there is already a method to translate facility names to
>>numbers (see calls to `log_facility_unshifted_from_string`).
>>
>>Ansgar
>>
>>  [1]: https://github.com/systemd/systemd/issues/9716
>
> Thanks.  That seems like a pretty basic feature.
>
> My impression so far is that the journalctl interface is a regression from 
> what we have
> now in every way I care about.

I would venture that for every user who is grateful that
/var/log/mail.log collects all the various mail-related logs, there is
another user that curses about non being able to separate (out of the
box) the logs from all the programs that consider themselves
mail-related, and another user who struggles to remember in which
logfile $DIFFICULT_TO_CLASSIFY_PROGRAM might be writing its logs. The
former use-case is well served by having the current set of logfiles,
the latter will be much happier with being able to use journalctl.

So this doesn't strike me as a strong argument in either direction.

Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«

Re: Heads up: persistent journal has been enabled in systemd

[Andrey Rahmatullin]

On Wed, Feb 05, 2020 at 01:25:58PM +0100, Didier 'OdyX' Raboud wrote:
> Le mercredi, 5 février 2020, 07.01:44 h CET Scott Kitterman a écrit :
> > Not particularly useful IMO.  In /var/log/mail.log I can see log entries
> > from all the programs configured to log to the mail facility.
> 
> Unless I'm mistaken, exim4 logs nothing to /var/log/mail.log by default, only 
> to /var/log/exim4/{mainlog,paniclog}
This is correct.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Ansgar]

On Wed, 2020-02-05 at 13:32 +, Scott Kitterman wrote:
> > Currently yes.  There is an upstream bug asking for a way to
> > specify
> > them by name[1], but nobody implemented it yet.
> > 
> > I think a `--facility` option should be fairly easy to
> > implement.  Just
> > adapt some code from the existing `--identifier` and `--priority`
> > options, there is already a method to translate facility names to
> > numbers (see calls to `log_facility_unshifted_from_string`).
> > 
> > Ansgar
> > 
> >  [1]: https://github.com/systemd/systemd/issues/9716
> 
> Thanks.  That seems like a pretty basic feature.

It is easy to implement, but systemd upstream seems not to like the
rather inflexible syslog facilities which don't cover all services
people run.  Journalctl can display logs for abitrary groups of
services, e.g.

$ journalctl --since=-1h -u dovecot -u "postfix@*" -u dkimpy-milter -u 
dkimpy-milter-verify

which would give similar results as looking at mail.log.

Ansgar

Re: Heads up: persistent journal has been enabled in systemd

[Michael Stone]

On Wed, Feb 05, 2020 at 01:32:41PM +, Scott Kitterman wrote:

My impression so far is that the journalctl interface is a regression from what 
we have now in every way I care about.


Great! Good thing you can just keep using rsyslogd. 

Re: Heads up: persistent journal has been enabled in systemd

[Scott Kitterman]

On February 5, 2020 12:35:45 PM UTC, Ansgar  wrote:
>On Wed, 2020-02-05 at 07:44 +, Scott Kitterman wrote:
>> Do syslog facilities really have to be addressed by number rather
>than name?  That seems like a horrible interface.
>
>Currently yes.  There is an upstream bug asking for a way to specify
>them by name[1], but nobody implemented it yet.
>
>I think a `--facility` option should be fairly easy to implement.  Just
>adapt some code from the existing `--identifier` and `--priority`
>options, there is already a method to translate facility names to
>numbers (see calls to `log_facility_unshifted_from_string`).
>
>Ansgar
>
>  [1]: https://github.com/systemd/systemd/issues/9716

Thanks.  That seems like a pretty basic feature.

My impression so far is that the journalctl interface is a regression from what 
we have now in every way I care about.  I really don't think that, despite the 
undoubted architectural advantages, it's suitable to be Debian's default.

Scott K

Re: Heads up: persistent journal has been enabled in systemd

[Marvin Renich]

* Matt Zagrabelny  [200204 21:27]:
> The contents of /var/log/journal will be binary files that journalctl
> will read. IIRC.

This is my objection to the systemd journal.  Binary log files are
absolutely _horrible_ for the general user, but they are terrific for
large data centers.

In a large data center, the logs from many machines are being sent via
network to a machine whose job it is to collect them, where sysadmins
can monitor all the hosts.  The binary format is much more efficient.

However, on an individual user's machine, you are now forced to use a
specialized program to parse the logs.  If a journal file is damaged in
the middle, the remainder of the file is useless without expertise in
the binary format of the file and significantly more (usually
prohibitive) forensic effort.  If a machine crashes and the user needs
to examine the logs offline (e.g. booting from a live image, or copying
the logs to a different machine), now he/she needs to learn how to use
journalctl on external files; perhaps this is easy, but it is one more
thing to learn.  Anyone administering their own personal machine knows
how to look at text files, and there are a wide variety of tools
available to manipulate them.  Text log files can be copied to a Windows
machine, a BSD system, or even to a smart phone, and are just as usable
there; binary journal files are not.

When your machine has crashed or your hard disk is having intermittent
failures are times when you most need to be able to read log files, and
are times when dealing with a binary format may be the hardest.

When choosing defaults for the Debian distribution, if one default is
more appropriate for the general user and another is better for the
experienced sysadmin, the maintainer should almost always opt in favor
of the general user.

...Marvin

Re: Heads up: persistent journal has been enabled in systemd

[Ansgar]

On Wed, 2020-02-05 at 07:44 +, Scott Kitterman wrote:
> Do syslog facilities really have to be addressed by number rather than name?  
> That seems like a horrible interface.

Currently yes.  There is an upstream bug asking for a way to specify
them by name[1], but nobody implemented it yet.

I think a `--facility` option should be fairly easy to implement.  Just
adapt some code from the existing `--identifier` and `--priority`
options, there is already a method to translate facility names to
numbers (see calls to `log_facility_unshifted_from_string`).

Ansgar

  [1]: https://github.com/systemd/systemd/issues/9716

Re: Heads up: persistent journal has been enabled in systemd

[Didier 'OdyX' Raboud]

Le mercredi, 5 février 2020, 07.01:44 h CET Scott Kitterman a écrit :
> Not particularly useful IMO.  In /var/log/mail.log I can see log entries
> from all the programs configured to log to the mail facility.

Unless I'm mistaken, exim4 logs nothing to /var/log/mail.log by default, only 
to /var/log/exim4/{mainlog,paniclog}

-- 
OdyX

Re: Heads up: persistent journal has been enabled in systemd

[Simon Richter]

Hi,

On Wed, Feb 05, 2020 at 09:43:43AM +0100, Bastian Blank wrote:

> However, you most likely don't actually want to tail this specific file.
> It is a solution for a more abstract problem, e.g. "I want to see logs
> from Postfix".  And this question got a different answer.

The appeal of mail.log is that all mail related stuff is together, so you
can see the handover between the components.

   Simon

Re: Heads up: persistent journal has been enabled in systemd

[Philipp Kern]

On 2020-02-05 08:12, Michael Biebl wrote:

journald has matured of the years and journalctl has become more
sophisticated with dealing with broken journal files, so I would be 
very

much interested in your experience with more recent systemd versions.


Not quite the same issue, but I will note that journalctl's (and also 
systemctl status') performance reading journal files is still pretty 
awful on spinning rust[1]. At times this makes me go to text logs 
instead because slicing the files using tail and grep is much, much 
faster.


Kind regards
Philipp Kern

[1] I think this is pretty much 
https://github.com/systemd/systemd/issues/2460

Re: Heads up: persistent journal has been enabled in systemd

[Bastian Blank]

Hi Scott

On Wed, Feb 05, 2020 at 07:44:25AM +, Scott Kitterman wrote:
> >> Of course the fact that I can't use all the tools available to
> >manipulate text 
> >> files to follow or analyze logs is problematic.  If I'm using
> >journalctl, how 
> >> do I replicate 'tail -f /var/log/mail.loog'?
> >
> >journalctl -f SYSLOG_IDENTIFIER=2
> >
> Do syslog facilities really have to be addressed by number rather than name?  
> That seems like a horrible interface.

That's because you asked a pretty specific question and you've been
told the exact replacement.

However, you most likely don't actually want to tail this specific file.
It is a solution for a more abstract problem, e.g. "I want to see logs
from Postfix".  And this question got a different answer.

Bastian

-- 
War isn't a good life, but it's life.
-- Kirk, "A Private Little War", stardate 4211.8

Re: Heads up: persistent journal has been enabled in systemd

[Scott Kitterman]

On February 5, 2020 6:53:56 AM UTC, Michael Biebl  wrote:
>Am 05.02.20 um 07:01 schrieb Scott Kitterman:
>> Of course the fact that I can't use all the tools available to
>manipulate text 
>> files to follow or analyze logs is problematic.
>Fwiw, you can still run
>journalctl -f | grep bla | awk and what not
>which will filter / process incoming messages
>or journalctl | grep ... to filter all of the journal
>
>I might be misunderstanding what you are hinting at, so please
>elaborate
>if the above doesn't address your concerns.

That's only helpful to the extent journalctl is essentially bug free.  With 
text files there are many different tools/approaches one can use to extract 
information if there are problems.  Not so now.

Personally I spend very little time in /var/log/syslog.  I'm almost always 
using facility specific logs which I now understand aren't actually impossible 
to replicate with journalctl, the combination of these things makes it feel 
like a huge usability regression.

I feel like you're piling a lot of complexity onto end users for approximately 
no benefit to most.

Scott K

Re: Heads up: persistent journal has been enabled in systemd

[Scott Kitterman]

On February 5, 2020 6:49:07 AM UTC, Michael Biebl  wrote:
>Am 05.02.20 um 07:01 schrieb Scott Kitterman:
>
>> Not particularly useful IMO.  In /var/log/mail.log I can see log
>entries from 
>> all the programs configured to log to the mail facility.  That way I
>can see 
>> the interaction between them.  On a typical server that is for
>sending mail I 
>> often need to see log entries from postfix, clamsmtp, and
>dkimpy-milter 
>> together to understand how a message is (or isn't) making it through
>the 
>> system.
>> 
>> Of course the fact that I can't use all the tools available to
>manipulate text 
>> files to follow or analyze logs is problematic.  If I'm using
>journalctl, how 
>> do I replicate 'tail -f /var/log/mail.loog'?
>
>journalctl -f SYSLOG_IDENTIFIER=2
>
>You can also apply additional filters, like severity and much more.
>
>https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs
>has a quite well written introduction
>
>HTH,
>Michael

Not really.  Nothing in the link addresses my question.

Do syslog facilities really have to be addressed by number rather than name?  
That seems like a horrible interface.

Scott K

Re: Heads up: persistent journal has been enabled in systemd

[Michael Biebl]

Am 05.02.20 um 07:49 schrieb Michael Biebl:
> Am 05.02.20 um 07:01 schrieb Scott Kitterman:
> 
>> Not particularly useful IMO.  In /var/log/mail.log I can see log entries 
>> from 
>> all the programs configured to log to the mail facility.  That way I can see 
>> the interaction between them.  On a typical server that is for sending mail 
>> I 
>> often need to see log entries from postfix, clamsmtp, and dkimpy-milter 
>> together to understand how a message is (or isn't) making it through the 
>> system.
>>
>> Of course the fact that I can't use all the tools available to manipulate 
>> text 
>> files to follow or analyze logs is problematic.  If I'm using journalctl, 
>> how 
>> do I replicate 'tail -f /var/log/mail.loog'?
> 
> journalctl -f SYSLOG_IDENTIFIER=2
> 
> You can also apply additional filters, like severity and much more.

This also is an aspect I very much like about the journal.
Once rsyslog has written a log message to disk, most of the meta data is
lost and you can no longer filter for it.
E.g. say I only wanted to inspect messages with the cron facility, then
this is easy to do with journalctl. Or you want to filter out daemon
messages with certain severities or coming from certain processes.
While you can write rsyslog filter rules for that, you can't apply those
to existing log messages that have been written to disk.






signature.asc
Description: OpenPGP digital signature

Re: Heads up: persistent journal has been enabled in systemd

[Michael Biebl]

Am 05.02.20 um 06:08 schrieb Russell Stuart:
> journald has nits I mention below, but I was prepared to put up with
> them and drop rsyslog until one day a server stopped in a nasty way and
> journalctl refused to display what lead up to the crash because it's
> binary logs were corrupt.  As far as I was concerned this made journald
> unfit for use on production servers.  (rsyslog's logs also get 4k lumps
> of nulls and other garbage in them in similar situations, but they
> remain usable.)

journal files are append only. If journald notices a
corrupt/inconsistent journal file, it will rotate it away and start with
a fresh journal file. If you run journalctl, it will do its best to read
as much data from the unclean journal as possible.

I assume you had some kind of file system corruption. So depending on
where that corruption inside the journal file happened, it's possible
that journalctl couldn't read behind this corruption.
Typically, due to the append only nature of the journal, if a crash
happens, only the very last log messages should be lost.

journald has matured of the years and journalctl has become more
sophisticated with dealing with broken journal files, so I would be very
much interested in your experience with more recent systemd versions.

Michael






signature.asc
Description: OpenPGP digital signature

Re: Heads up: persistent journal has been enabled in systemd

[Michael Biebl]

Am 05.02.20 um 07:01 schrieb Scott Kitterman:
> Of course the fact that I can't use all the tools available to manipulate 
> text 
> files to follow or analyze logs is problematic.
Fwiw, you can still run
journalctl -f | grep bla | awk and what not
which will filter / process incoming messages
or journalctl | grep ... to filter all of the journal

I might be misunderstanding what you are hinting at, so please elaborate
if the above doesn't address your concerns.

Michael



signature.asc
Description: OpenPGP digital signature

Re: Heads up: persistent journal has been enabled in systemd

[Michael Biebl]

Am 05.02.20 um 07:01 schrieb Scott Kitterman:

> Not particularly useful IMO.  In /var/log/mail.log I can see log entries from 
> all the programs configured to log to the mail facility.  That way I can see 
> the interaction between them.  On a typical server that is for sending mail I 
> often need to see log entries from postfix, clamsmtp, and dkimpy-milter 
> together to understand how a message is (or isn't) making it through the 
> system.
> 
> Of course the fact that I can't use all the tools available to manipulate 
> text 
> files to follow or analyze logs is problematic.  If I'm using journalctl, how 
> do I replicate 'tail -f /var/log/mail.loog'?

journalctl -f SYSLOG_IDENTIFIER=2

You can also apply additional filters, like severity and much more.

https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs
has a quite well written introduction

HTH,
Michael




signature.asc
Description: OpenPGP digital signature

Re: Heads up: persistent journal has been enabled in systemd

[Bastian Blank]

On Wed, Feb 05, 2020 at 09:39:19AM +1100, Dmitry Smirnov wrote:
> For example, if a certain daemon manifested a condition when a message is 
> logged too often, then with Rsyslog I could suppress noise by something like 
> the following

This is a workaround for another problem.  Fix the real problem.

> This is just an example (probably not the best one) how feature can be handy 
> when it is needed. The point is that you'll only realise that you can't do 
> something any more is when you need it the most.

Please show that one Joey Random User who asked how to do that.

> Rsyslog is not broken to be replaced as default logging system.

No-one said it is broken.  But not broken does not mean a random user
needs it.  We try to build a distribution with good defaults that fit
most people, we don't try to build one that specificaly follows your
specific requirements or whishes.

> Finally there were no attempt to seek consensus, no survey, nothing?
> Just a decision of few maintainers to replace good default with their 
> preferences?

Sure, there was.  Please look at debian-vote.

> It is difficult to appreciate needless disruptive changes.

Which one?  Do you really consider persistent journal, which this thread
is about, a _disruptive_ change?  Please elaborate.

Bastian

-- 
"That unit is a woman."
"A mass of conflicting impulses."
-- Spock and Nomad, "The Changeling", stardate 3541.9

Re: Heads up: persistent journal has been enabled in systemd

[Vincent Bernat]

 ❦  5 février 2020 01:01 -05, Scott Kitterman :

> Not particularly useful IMO.  In /var/log/mail.log I can see log entries from 
> all the programs configured to log to the mail facility.  That way I can see 
> the interaction between them.  On a typical server that is for sending mail I 
> often need to see log entries from postfix, clamsmtp, and dkimpy-milter 
> together to understand how a message is (or isn't) making it through the 
> system.
>
> Of course the fact that I can't use all the tools available to manipulate 
> text 
> files to follow or analyze logs is problematic.  If I'm using journalctl, how 
> do I replicate 'tail -f /var/log/mail.loog'?

You can use:

journalctl -f SYSLOG_FACILITY=2
-- 
Its name is Public Opinion.  It is held in reverence.  It settles everything.
Some think it is the voice of God.
-- Mark Twain


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Scott Kitterman]

On Tuesday, February 4, 2020 9:01:55 PM EST Matt Zagrabelny wrote:
> On Tue, Feb 4, 2020 at 5:15 PM Scott Kitterman  wrote:
> > On Tuesday, February 4, 2020 5:22:15 PM EST Vincent Bernat wrote:
> > >  ❦  4 février 2020 11:30 -08, Russ Allbery :
> > > >> As a heavy user or Rsyslog features I feel that switching default
> > > >> logging system yields no benefits to say the least.
> > > > 
> > > > As a heavy user, perhaps you're not the target audience for a default?
> > > > You're going to install rsyslog no matter what, since you know it well
> > > > and
> > > > use it heavily.  The only effect of this change on you will be a
> > > > one-line
> > > > change to whatever you use for configuration management for new
> > > > systems.
> > > 
> > > rsyslog even knows how to directly pull logs from the journal, which
> > > gives you access to stuff not logged to syslog (stdout/stderr of service
> > > files, applications logging directly to journal), as well to structured
> > > logs (comm pid, user, unit and more when the service supports journald
> > > directly).
> > 
> > For those of us who aren't customizers of Debian's logging function, it'd
> > be nice to have a clearer understanding of what this changes means.
> > 
> > Today, when, for example, I want to investigate something email related, I
> > look in /var/log/mail.log.
> 
> Random email related journal commands:
> 
> journalctl -u postfix
> journalctl -f -u postfix
> journalctl -b -u postfix
> 
> the -u is for the unit name. the -b is for since boot. man journalctl
> for details.

Not particularly useful IMO.  In /var/log/mail.log I can see log entries from 
all the programs configured to log to the mail facility.  That way I can see 
the interaction between them.  On a typical server that is for sending mail I 
often need to see log entries from postfix, clamsmtp, and dkimpy-milter 
together to understand how a message is (or isn't) making it through the 
system.

Of course the fact that I can't use all the tools available to manipulate text 
files to follow or analyze logs is problematic.  If I'm using journalctl, how 
do I replicate 'tail -f /var/log/mail.loog'?

Note that I'm not asking about some specialized configuration that I've set up. 
 
All I want to know is how to make it work like Debian works out of the box 
today.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Russell Stuart]

On Tue, 2020-02-04 at 18:10 -0800, Russ Allbery wrote:
> It does take a bit of retraining to use journalctl instead (and I'm
> personally not horribly fond of its UI, although that's probably
> because I'm using it wrong), but it's a lot better at effectively
> narrowing log messages to the things of interest once you get used to
> it.

journald has nits I mention below, but I was prepared to put up with
them and drop rsyslog until one day a server stopped in a nasty way and
journalctl refused to display what lead up to the crash because it's
binary logs were corrupt.  As far as I was concerned this made journald
unfit for use on production servers.  (rsyslog's logs also get 4k lumps
of nulls and other garbage in them in similar situations, but they
remain usable.)

That was a long time ago, and it may well be fixed now.  But if it
isn't IMO turning off rsyslog by default is a bad idea.  My view is the
main reason Debian exists is to serve as a reliable base for production
machines.  Debian Desktop is what I use on my personal machine and yes,
dropping rsyslog hardly matters there, but I wouldn't be using Debian
Desktop if I wasn't using Debian in production.   

Another journald anti-feature (which is probably an unfair attribution
as it is almost certainly a consequence of systemd's design), is a
manually started service doesn't print the reason it refused to start
to stderr.  Having to fire up journald and wade through it's crappy UI
to get something sysV used to put under my nose is a step backwards.

Finally, it may be I just don't know how to use it well, but looking
for a needle in a haystack of logs is slower with journalctl that it is
with grep, and not by a small margin.  Journald making the thing you
spend most time doing with logs slower doesn't help it in the
slightest.  But I don't spend a lot of time searching logs, so it
wouldn't stop me from dropping rsyslog.

Get rid of those problems, and dropping rsyslog becomes a no-brainer
for me.


signature.asc
Description: This is a digitally signed message part

Re: Heads up: persistent journal has been enabled in systemd

[Matt Zagrabelny]

On Tue, Feb 4, 2020 at 5:15 PM Scott Kitterman  wrote:
>
> On Tuesday, February 4, 2020 5:22:15 PM EST Vincent Bernat wrote:
> >  ❦  4 février 2020 11:30 -08, Russ Allbery :
> > >> As a heavy user or Rsyslog features I feel that switching default
> > >> logging system yields no benefits to say the least.
> > >
> > > As a heavy user, perhaps you're not the target audience for a default?
> > > You're going to install rsyslog no matter what, since you know it well and
> > > use it heavily.  The only effect of this change on you will be a one-line
> > > change to whatever you use for configuration management for new
> > > systems.
> >
> > rsyslog even knows how to directly pull logs from the journal, which
> > gives you access to stuff not logged to syslog (stdout/stderr of service
> > files, applications logging directly to journal), as well to structured
> > logs (comm pid, user, unit and more when the service supports journald
> > directly).
>
> For those of us who aren't customizers of Debian's logging function, it'd be
> nice to have a clearer understanding of what this changes means.
>
> Today, when, for example, I want to investigate something email related, I
> look in /var/log/mail.log.

Random email related journal commands:

journalctl -u postfix
journalctl -f -u postfix
journalctl -b -u postfix

the -u is for the unit name. the -b is for since boot. man journalctl
for details.

  Other specialized log files for their special
> purposes.  For data not covered by a specialized log, I look in /var/log/
> syslog.
>
> Will the specialized log files still be there?

I suppose it depends. Do the processes write to /dev/log or manually
write to files?

  Will the net effect be that I
> just need to look in /var/log/journal (or something similar) instead of in /
> var/log/syslog?

The contents of /var/log/journal will be binary files that journalctl
will read. IIRC.

  Is the persistent journal a text file or will I need
> specialized tools to interact with it?

specialized: journalctl.

-m




>
> Scott K

Re: Heads up: persistent journal has been enabled in systemd

[Russ Allbery]

Dmitry Smirnov  writes:
> On Wednesday, 5 February 2020 6:30:03 AM AEDT Russ Allbery wrote:

>> The primary benefit that I can see is one fewer daemon running on a
>> default installation, one fewer thing to have security vulnerabilities
>> or some other problems, one fewer thing to keep up to date, and a
>> smaller base installation.  To be clear, these benefits are fairly
>> minor, but they do exist.

> The question is whether those benefits are enough to justify replacing a
> very solid and reliable logging system.

I'm confused.  Currently, we have both the systemd journal and, in
addition, rsyslog.  The proposal is that we would stop running the
separate rsyslog daemon by default and have only the systemd journal.

I believe the journal is already upstream of all traffic that rsyslog sees
in a default installation given that systemd controls /dev/syslog.  We
currently are running two logging infrastructures in the default
installation, and the proposal is to run only one.  What's being replaced?
Am I misunderstanding something?

I think the strongest argument against running rsyslog by default is that
currently rsyslog is responsible for receiving forwarded messages from the
journal and writing a second copy of them in the conventional text log
files.  I believe without rsyslog one would have to use journalctl,
systemctl, or similar programs to read messages, which might be a
confusing change.  I'm personally not sure whether avoiding that change is
worth double-writing log messages and running another daemon in the
default install.

It does take a bit of retraining to use journalctl instead (and I'm
personally not horribly fond of its UI, although that's probably because
I'm using it wrong), but it's a lot better at effectively narrowing log
messages to the things of interest once you get used to it.  (And anyone
who doesn't like that can just apt-get install rsyslog and be done.)

In a lot of environments, I'd probably install rsyslog anyway for various
reasons, but that's of course trivial to do.

-- 
Russ Allbery (r...@debian.org)  

Re: Heads up: persistent journal has been enabled in systemd

[Scott Kitterman]

On Tuesday, February 4, 2020 5:39:19 PM EST Dmitry Smirnov wrote:
> On Wednesday, 5 February 2020 6:30:03 AM AEDT Russ Allbery wrote:
> > The primary benefit that I can see is one fewer daemon running on a
> > default installation, one fewer thing to have security vulnerabilities or
> > some other problems, one fewer thing to keep up to date, and a smaller
> > base installation.  To be clear, these benefits are fairly minor, but they
> > do exist.
> 
> The question is whether those benefits are enough to justify replacing a
> very solid and reliable logging system.
> 
> It is probably correct that most users don't use Rsyslog features. But that
> doesn't mean that those features should be taken away from default
> installation.
> 
> For example, if a certain daemon manifested a condition when a message is
> logged too often, then with Rsyslog I could suppress noise by something like
> the following
> 
> 
> if ($programname == "noisydaemon") then {
>   if ($msg contains "frequently repeated noise") then {
> stop
>   }
> }
> 
> 
> This is just an example (probably not the best one) how feature can be handy
> when it is needed. The point is that you'll only realise that you can't do
> something any more is when you need it the most.
> 
> Also the cost of learning. I'm sure that more people are more familiar with
> Rsyslog. So there is inconvenience too. Disruptive changing of good default
> for weak reasons is not nice.
> 
> Rsyslog is not broken to be replaced as default logging system.
> 
> Finally there were no attempt to seek consensus, no survey, nothing?
> Just a decision of few maintainers to replace good default with their
> preferences?
> 
> It is difficult to appreciate needless disruptive changes.

We just had a GR where the project voted it was just fine to systemd all the 
things, so this sort of thing is to be expected.

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Scott Kitterman]

On Tuesday, February 4, 2020 5:22:15 PM EST Vincent Bernat wrote:
>  ❦  4 février 2020 11:30 -08, Russ Allbery :
> >> As a heavy user or Rsyslog features I feel that switching default
> >> logging system yields no benefits to say the least.
> > 
> > As a heavy user, perhaps you're not the target audience for a default?
> > You're going to install rsyslog no matter what, since you know it well and
> > use it heavily.  The only effect of this change on you will be a one-line
> > change to whatever you use for configuration management for new
> > systems.
> 
> rsyslog even knows how to directly pull logs from the journal, which
> gives you access to stuff not logged to syslog (stdout/stderr of service
> files, applications logging directly to journal), as well to structured
> logs (comm pid, user, unit and more when the service supports journald
> directly).

For those of us who aren't customizers of Debian's logging function, it'd be 
nice to have a clearer understanding of what this changes means.

Today, when, for example, I want to investigate something email related, I 
look in /var/log/mail.log.  Other specialized log files for their special 
purposes.  For data not covered by a specialized log, I look in /var/log/
syslog.

Will the specialized log files still be there?  Will the net effect be that I 
just need to look in /var/log/journal (or something similar) instead of in /
var/log/syslog?  Is the persistent journal a text file or will I need 
specialized tools to interact with it?

Scott K


signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Dmitry Smirnov]

On Wednesday, 5 February 2020 6:30:03 AM AEDT Russ Allbery wrote:
> The primary benefit that I can see is one fewer daemon running on a
> default installation, one fewer thing to have security vulnerabilities or
> some other problems, one fewer thing to keep up to date, and a smaller
> base installation.  To be clear, these benefits are fairly minor, but they
> do exist.

The question is whether those benefits are enough to justify replacing a very 
solid and reliable logging system.

It is probably correct that most users don't use Rsyslog features. But that 
doesn't mean that those features should be taken away from default 
installation.

For example, if a certain daemon manifested a condition when a message is 
logged too often, then with Rsyslog I could suppress noise by something like 
the following


if ($programname == "noisydaemon") then {
  if ($msg contains "frequently repeated noise") then {
stop
  }
}


This is just an example (probably not the best one) how feature can be handy 
when it is needed. The point is that you'll only realise that you can't do 
something any more is when you need it the most.

Also the cost of learning. I'm sure that more people are more familiar with 
Rsyslog. So there is inconvenience too. Disruptive changing of good default 
for weak reasons is not nice.

Rsyslog is not broken to be replaced as default logging system.

Finally there were no attempt to seek consensus, no survey, nothing?
Just a decision of few maintainers to replace good default with their 
preferences?

It is difficult to appreciate needless disruptive changes.

-- 
Best wishes,
 Dmitry Smirnov.

---

It is impossible to imagine Goethe or Beethoven being good at billiards
or golf.
-- H. L. Mencken


signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Vincent Bernat]

 ❦  4 février 2020 11:30 -08, Russ Allbery :

>> As a heavy user or Rsyslog features I feel that switching default
>> logging system yields no benefits to say the least.
>
> As a heavy user, perhaps you're not the target audience for a default?
> You're going to install rsyslog no matter what, since you know it well and
> use it heavily.  The only effect of this change on you will be a one-line
> change to whatever you use for configuration management for new
> systems.

rsyslog even knows how to directly pull logs from the journal, which
gives you access to stuff not logged to syslog (stdout/stderr of service
files, applications logging directly to journal), as well to structured
logs (comm pid, user, unit and more when the service supports journald
directly).
-- 
Use the good features of a language; avoid the bad ones.
- The Elements of Programming Style (Kernighan & Plauger)


signature.asc
Description: PGP signature

Re: Heads up: persistent journal has been enabled in systemd

[Russ Allbery]

Dmitry Smirnov  writes:
> On Saturday, 1 February 2020 2:05:55 PM AEDT Michael Biebl wrote:

>> Depending on how it goes, I might ask the ftp-masters to lower the
>> priority of rsyslog from important to optional, so it would no longer
>> be installed by default on new bullseye installations.

> I have a mixed feelings about that. I think that replacing rsyslog with 
> journald is two steps back in regards to functionality and flexibility.

> Rsyslog in unparalleled with its ability to process and filter messages
> (rainerscript), to transform messages (liblognorm), to forward messages
> to Elasticsearch or centralise Rsyslog instance, _reliably_ (relp), to
> buffer message queue on disk in case communication is disrupted, etc.

I completely agree with this assessment of the quality of rsyslog
features, but I'm not sure that's the right criteria to be considering for
the choice of the *default*.  I'm fairly certain that 95% or more of
installed Debian systems never used any of those features, as nice as they
are.

The goal of the default is not to provide in latent form every excellent
feature that anyone may want to use.  It's to provide a hopefully simple,
reliable, functional, and light-weight implementation of a facility that
serves as a reasonable default for most systems.  Anyone with other needs
or preferences is very likely to replace or supplement that implementation
with something else, similar to how I replace exim4 with postfix on all of
my systems.

> Am I correct that journald is nowhere near that functionality?

Yes.

> As a heavy user or Rsyslog features I feel that switching default
> logging system yields no benefits to say the least.

As a heavy user, perhaps you're not the target audience for a default?
You're going to install rsyslog no matter what, since you know it well and
use it heavily.  The only effect of this change on you will be a one-line
change to whatever you use for configuration management for new systems.

The primary benefit that I can see is one fewer daemon running on a
default installation, one fewer thing to have security vulnerabilities or
some other problems, one fewer thing to keep up to date, and a smaller
base installation.  To be clear, these benefits are fairly minor, but they
do exist.

-- 
Russ Allbery (r...@debian.org)  

Re: Heads up: persistent journal has been enabled in systemd

[Dmitry Smirnov]

On Saturday, 1 February 2020 2:05:55 PM AEDT Michael Biebl wrote:
> Depending on how it goes, I might ask the ftp-masters to lower the
> priority of rsyslog from important to optional, so it would no longer be
> installed by default on new bullseye installations.

I have a mixed feelings about that. I think that replacing rsyslog with 
journald is two steps back in regards to functionality and flexibility.

Rsyslog in unparalleled with its ability to process and filter messages 
(rainerscript), to transform messages (liblognorm), to forward messages to 
Elasticsearch or centralise Rsyslog instance, _reliably_ (relp), to buffer 
message queue on disk in case communication is disrupted, etc.

Am I correct that journald is nowhere near that functionality?
Also maturity/stability -- Rsyslog has been around much longer and it never 
let me down...

As a heavy user or Rsyslog features I feel that switching default logging 
system yields no benefits to say the least.

-- 
Best wishes,
 Dmitry Smirnov

---

Men become civilized, not in proportion to their willingness to believe,
but in proportion to their readiness to doubt.
-- H. L. Mencken


signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Michael Stone]

On Sun, Feb 02, 2020 at 02:35:19PM +, Anthony DeRobertis wrote:

On February 2, 2020 12:02:33 PM UTC, Simon khng  
wrote:

Why was rsyslog used as the persistent storage instead of journald for
previous Debian distribution?


rsyslog has been the default Debian log storage since before switching to 
systemd, possibly since before systemd existed (it was syslogd, but would have 
to check if it was rsyslog or some other implementation).


Way back at the beginning there was syslogd, that got combined early on 
with klogd into the sysklogd package, rsyslogd replaced it on default 
installs in lenny (released 2009). So yes, the reason is that it 
predates systemd. :)

Re: Heads up: persistent journal has been enabled in systemd

[Anthony DeRobertis]

On February 2, 2020 12:02:33 PM UTC, Simon khng  
wrote:

>Why was rsyslog used as the persistent storage instead of journald for
>previous Debian distribution?

rsyslog has been the default Debian log storage since before switching to 
systemd, possibly since before systemd existed (it was syslogd, but would have 
to check if it was rsyslog or some other implementation).

Re: Heads up: persistent journal has been enabled in systemd

[Simon khng]

Hi,

Not sure if stated in any documentation, but I will ask here since it is
easier to get clarification.

Why was rsyslog used as the persistent storage instead of journald for
previous Debian distribution?

I did not check but is the current journald temp session log storing data
in binary format maybe? Since it was stated in the email that users can use
other log package if text format is preferable.

On that topic, why use the network manager package instead of the networkd?
This is actually a similar situation for current stable of Debian.

Simon

On Sat, Feb 1, 2020, 11:24 AM Michael Biebl  wrote:

> Hi,
>
> with today's upload of systemd 244.1-2 I finally enabled persistent
> journal by default [1]. It has been a long requested feature.
>
> The package will create a directory /var/log/journal on upgrades and new
> installs, which enables persistent journal in so called auto mode.
>
> If you decide, that you want to disable the persistent journal again,
> you can run:
> journalctl --relinquish-var; rm -rf /var/log/journal
>
> Future package updates will respect this choice and not re-create the
> directory. You can, of course, also configure this explicitly via the
> Storage= option in journald.conf.
>
> Depending on how it goes, I might ask the ftp-masters to lower the
> priority of rsyslog from important to optional, so it would no longer be
> installed by default on new bullseye installations.
> This would avoid, that we store log messages twice on disk.
> Users that prefer text logs can of course still install rsyslog by
> default (or their syslogger of choice).
> Alternative init systems might consider adding a Recommends on a syslog
> implementation of their choice or creating a task, which would pull in a
> syslogger.
>
> Here are some resources that you might find useful:
> - man journald.conf
>   https://www.freedesktop.org/software/systemd/man/journald.conf.html#
> - man journalctl
>   https://www.freedesktop.org/software/systemd/man/journalctl.html#
> - man systemd-journald.service
>   https://www.freedesktop.org/software/systemd/man/systemd-journald.html#
>
> Regards,
> Michael
>
>
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717388
>
>

Re: Heads up: persistent journal has been enabled in systemd

[Dimitri John Ledkov]

On Sat, 1 Feb 2020 at 22:01, Michael Biebl  wrote:
>
> Hi Steve
>
> Am 01.02.20 um 14:36 schrieb Steve McIntyre:
> > Michael Biebl wrote:
> >>
> >> with today's upload of systemd 244.1-2 I finally enabled persistent
> >> journal by default [1]. It has been a long requested feature.
> >>
> >> The package will create a directory /var/log/journal on upgrades and new
> >> installs, which enables persistent journal in so called auto mode.
> >
> > Fine for new installations, but please *don't* do this for
> > upgrades. Those people with existing logging setups will be surprised
> > by this.
>
> I honestly don't like packages that behave differently depending on
> whether they were upgraded or installed anew (given their default
> configuration wasn't changed). I assume there would be equally as many
> or even more surprised users that find their upgraded system behave
> differently then their freshly installed system.
>
> The change here is basically just an update of the default behaviour of
> journald. If you explicitly configured a different journald behavior via
> Storage=, this is respected. If you already created a /var/log/journal
> in the past, the change will be a nop.
>
> Existing sysloggers will continue to work after this change as before,
> they are not directly affected by this change.
>
> Can you eloborate more on this what your concerns are here?
>
> I think Didier's idea of documenting this in the release notes is a good
> one, so I filed #950447. The NEWS entry is a good idea as well and I
> thought about that, but decided to not add it (yet) but instead ask for
> feedback on debian-devel first. Depending on the feedback, the content
> of that NEWS entry might change. E.g. I was unsure whether I should
> mention that a syslogger like rsyslog can now be uninstalled safely, as
> the contents are now available in the journal as well. Feedback on this
> welcome.
>
> I would also like to mention that there is prior art. Ubuntu made the
> switch to a persistent journal a while back and they did it on upgrades
> and new installations [1] as well.
> I've CCed Dimitri, just in case he wants to chime in here. Maybe he has
> some feedback on this transition in Ubuntu.
>

In Ubuntu, we have enabed persistant journal by default on fresh
installs and upgrades, but only once.

# Enable persistent journal, in auto-mode, by default on new installs
installs and upgrades
if dpkg --compare-versions "$2" lt "235-3ubuntu3~"; then
mkdir -p /var/log/journal
# create tmpfiles only when running systemd, otherwise %b substitution fails
if [ -d /run/systemd/system ]; then
systemd-tmpfiles --create --prefix /var/log/journal
fi
fi

Meaning, post-install/upgrade if one removes /var/log/journal it gets
disabled and doesn't come back again.

Overall, the feedback was extremely positive. A lot of different
classes of users were expecting journal to be there and were very
unhappy to find out retroactively that it was not there on first boot
/ last boot / broken boot. Especially people with existing logging
setups were pissed off about missing logs that only appeared in the
small ephemeral runtime journal and never made it to the persistent
journal on disk - for example early boot logs; failing to boot;
shutdown logs; and whenever remote logging services failed.

The fact that we leave it "auto" mode and simply removing
/var/log/journal gets rid of the persistent journaling is very
intuitive too, and grumpy people who dislike it manage to
disable/remove it, without any complaints (i.e. there are no bug
reports "why is journal back").

We have a recurring class of "bugs" filed by people who claim "OMG
journal is growing to unlimited size", and every single time so far
that turned out to be not true, and the caps that journald calculates
are actually enforced.
$ journalctl -u systemd-journald -b | head
System Journal (/var/log/journal/1b8df0fa27039f0163586c6756a6d401) is
3.9G, max 4.0G, 23.7M free.

After pointing out that caps are actually enfoced and rotated off, and
that one can set bigger / smaller caps for the persistent journal,
people seemed to agree that it doesn't in fact grow to unlimited
sizes.

This is the experience we have had in Ubuntu, and I do hope that in
Debian too it will be enabled on new installs and upgrades, but only
once and keep it in auto mode - meaning that removal of
/var/log/journal disables persistent journaling.

The change was done in December 2017 and thus systems installed with /
upgraded to 18.04 LTS overwhelmingly have persistent journal enabled
by default across the board.

-- 
Regards,

Dimitri.

Re: Heads up: persistent journal has been enabled in systemd

[Michael Biebl]

Hi Steve

Am 01.02.20 um 14:36 schrieb Steve McIntyre:
> Michael Biebl wrote:
>>
>> with today's upload of systemd 244.1-2 I finally enabled persistent
>> journal by default [1]. It has been a long requested feature.
>>
>> The package will create a directory /var/log/journal on upgrades and new
>> installs, which enables persistent journal in so called auto mode.
> 
> Fine for new installations, but please *don't* do this for
> upgrades. Those people with existing logging setups will be surprised
> by this.

I honestly don't like packages that behave differently depending on
whether they were upgraded or installed anew (given their default
configuration wasn't changed). I assume there would be equally as many
or even more surprised users that find their upgraded system behave
differently then their freshly installed system.

The change here is basically just an update of the default behaviour of
journald. If you explicitly configured a different journald behavior via
Storage=, this is respected. If you already created a /var/log/journal
in the past, the change will be a nop.

Existing sysloggers will continue to work after this change as before,
they are not directly affected by this change.

Can you eloborate more on this what your concerns are here?

I think Didier's idea of documenting this in the release notes is a good
one, so I filed #950447. The NEWS entry is a good idea as well and I
thought about that, but decided to not add it (yet) but instead ask for
feedback on debian-devel first. Depending on the feedback, the content
of that NEWS entry might change. E.g. I was unsure whether I should
mention that a syslogger like rsyslog can now be uninstalled safely, as
the contents are now available in the journal as well. Feedback on this
welcome.

I would also like to mention that there is prior art. Ubuntu made the
switch to a persistent journal a while back and they did it on upgrades
and new installations [1] as well.
I've CCed Dimitri, just in case he wants to chime in here. Maybe he has
some feedback on this transition in Ubuntu.

Regards,
Michael


[1]
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f94f18d9dbc085b6a9ff33c141a6e542142f85b5




signature.asc
Description: OpenPGP digital signature

Re: Heads up: persistent journal has been enabled in systemd

[Bernd Zeimetz]

On 2/1/20 2:36 PM, Steve McIntyre wrote:
> Michael Biebl wrote:
>>
>> with today's upload of systemd 244.1-2 I finally enabled persistent
>> journal by default [1]. It has been a long requested feature.
>>
>> The package will create a directory /var/log/journal on upgrades and new
>> installs, which enables persistent journal in so called auto mode.
> 
> Fine for new installations, but please *don't* do this for
> upgrades. Those people with existing logging setups will be surprised
> by this.


Yes, Steve is right here, please don't mess with existing
configurations. In most cases the result would not be what people wanted
to have.


-- 
 Bernd ZeimetzDebian GNU/Linux Developer
 http://bzed.dehttp://www.debian.org
 GPG Fingerprint: ECA1 E3F2 8E11 2432 D485  DD95 EB36 171A 6FF9 435F

Re: Heads up: persistent journal has been enabled in systemd

[Didier 'OdyX' Raboud]

Le samedi, 1 février 2020, 14.36:20 h CET Steve McIntyre a écrit :
> Michael Biebl wrote:
> >with today's upload of systemd 244.1-2 I finally enabled persistent
> >journal by default [1]. It has been a long requested feature.
> >
> >The package will create a directory /var/log/journal on upgrades and new
> >installs, which enables persistent journal in so called auto mode.
> 
> Fine for new installations, but please *don't* do this for
> upgrades. Those people with existing logging setups will be surprised
> by this.

I disagree. Please _do_ this for upgrades (it's great functionality!), but 
please do make sure that it is documented in NEWS.Debian (and release notes) 
so that it gets mentioned to host admins upon upgrade.

-- 
OdyX

signature.asc
Description: This is a digitally signed message part.

Re: Heads up: persistent journal has been enabled in systemd

[Steve McIntyre]

Michael Biebl wrote:
>
>with today's upload of systemd 244.1-2 I finally enabled persistent
>journal by default [1]. It has been a long requested feature.
>
>The package will create a directory /var/log/journal on upgrades and new
>installs, which enables persistent journal in so called auto mode.

Fine for new installations, but please *don't* do this for
upgrades. Those people with existing logging setups will be surprised
by this.

-- 
Steve McIntyre, Cambridge, UK.st...@einval.com
  Armed with "Valor": "Centurion" represents quality of Discipline,
  Honor, Integrity and Loyalty. Now you don't have to be a Caesar to
  concord the digital world while feeling safe and proud.

Heads up: persistent journal has been enabled in systemd

[Michael Biebl]

Hi,

with today's upload of systemd 244.1-2 I finally enabled persistent
journal by default [1]. It has been a long requested feature.

The package will create a directory /var/log/journal on upgrades and new
installs, which enables persistent journal in so called auto mode.

If you decide, that you want to disable the persistent journal again,
you can run:
journalctl --relinquish-var; rm -rf /var/log/journal

Future package updates will respect this choice and not re-create the
directory. You can, of course, also configure this explicitly via the
Storage= option in journald.conf.

Depending on how it goes, I might ask the ftp-masters to lower the
priority of rsyslog from important to optional, so it would no longer be
installed by default on new bullseye installations.
This would avoid, that we store log messages twice on disk.
Users that prefer text logs can of course still install rsyslog by
default (or their syslogger of choice).
Alternative init systems might consider adding a Recommends on a syslog
implementation of their choice or creating a task, which would pull in a
syslogger.

Here are some resources that you might find useful:
- man journald.conf
  https://www.freedesktop.org/software/systemd/man/journald.conf.html#
- man journalctl
  https://www.freedesktop.org/software/systemd/man/journalctl.html#
- man systemd-journald.service
  https://www.freedesktop.org/software/systemd/man/systemd-journald.html#

Regards,
Michael


[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717388



signature.asc
Description: OpenPGP digital signature