[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Examples to distinguish Before=/After= and Wants=/Requires=/BindsTo=

[Ulrich Windl]

>>> "Pagliari, Vivenzio (Nokia - DE/Ulm)"  schrieb
am
17.09.2021 um 09:52 in Nachricht


> Hello Manuel,
> 
> my experience is that it is helpful for people new to systemd to "raise the

> right question" that is answered by the corresponding relationship between 
> systemd units.
> 
> The question "*if* a unit shall be started/enqueued" is answered 
> Wants=/Requires=/BindsTo relation. I call this the "dependency relation". To

> know "*when* a unit shall be started/enqueued, you need to consider the 
> Before=/After= relation. I call this is the "time order relation". It is 
> pretty obvious, that a time order relation only makes sense, of you have a 
> dependency relation as well. The other way round it is different: it may
make 
> sense that some unit A wants some other unit B, but there is no time
ordering 
> between them.
> 
> If you have some "mathematically inclined" people whom you want to describe

> it: Given a set A of all systemd units on your computer, you have the
special 
> unit default.target (or the one passed on the kernel command line with the 
> systemd.unit parameter) which forms the root of a "dependency tree", given
by 
> the Requires/Wants relations. This tree forms the subset B of systemd units

> on your computer to be started/enqueued at boot time. This set B of systemd

> units is a partially ordered set: Some (but not all) of the elements have a

> (time) order relation, by which they can be (topologically) sorted. This is

> then the order in which the units are enqueued. Note that this order is not

> "fix", there can be different "total orderings" of a partially ordered set.

Yes, what makes things more complicated is that "After X" actually means
"NotBefore X" (i.e. "at some later time" and not "immediately after X"), and
"Before" means "NotAfter".
Combined with the swapped roles on system startup/shutdown, it can be rather
tricky sometimes to get things right.

Regards,
Ulrich

> 
> HTH
> Vivenzio
> 
> ‑Original Message‑
> From: systemd‑devel  On Behalf
Of 
> Manuel Wagesreither
> Sent: Wednesday, September 15, 2021 5:16 PM
> To: SystemD Devel 
> Subject: [systemd‑devel] Examples to distinguish Before=/After= and 
> Wants=/Requires=/BindsTo=
> 
> Hello all,
> 
> I'm onboarding some collegues who don't have much experience with systemd. 
> One thing I would like to focus on is the difference between Before=/After=

> and Wants=/Requires=/BindsTo in systemd units.
> 
> I think it would get immediately clear if could provide them an example 
> where we want one but not the other. Unfortunately I've got problems coming

> up with such an example. In my use cases, whenever I needed an After= I 
> needed an Wants= as well.
> 
> Can you come up with something good?
> 
> Thanks, regards,
> Manuel

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Examples to distinguish Before=/After= and Wants=/Requires=/BindsTo=

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 17.09.2021 um 11:49
in
Nachricht :
> On Mi, 15.09.21 17:15, Manuel Wagesreither (man...@fastmail.fm) wrote:
> 
>> Hello all,
>>
>> I'm onboarding some collegues who don't have much experience with
>> systemd. One thing I would like to focus on is the difference
>> between Before=/After= and Wants=/Requires=/BindsTo in systemd
>> units.
>>
>> I think it would get immediately clear if could provide them an
>> example where we want one but not the other. Unfortunately I've got
>> problems coming up with such an example. In my use cases, whenever I
>> needed an After= I needed an Wants= as well.
>>
>> Can you come up with something good?
> 
> Whenever you have a conceptually "weak" dependency between to
> packages, i.e. both services can work alone, but they can also wor
> together, and if so, one should be ordered after the other.
> 
> more specific example: you can use apache without mysql, and you can
> use mysql without apache, but quite often they are used together, and
> if so you likely want to start mysql first, and apache second, since
> it likely consumes services of mysql, and not the other way
> round. Hence in this example, you'd place an ordering dep, but not
> requirement dep.

And another aspect: While you could add the dependencies in mysql as well as
apache, you'd probably do it in apache here (as it is the "consumer"), right?
Or would you place it outside of both?

> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] Re: Prefix for direct logging

[Ulrich Windl]

>>> Arjun D R  schrieb am 29.09.2021 um 08:44 in
Nachricht
:
> Hi Lennart,
> 
> That's a good idea but still I would like to have the prefix as it is in
> the journal . I understand it is impossible to bypass the journal and

Why not write and use your own log-output function based on vsnprintf() (for
example) (adding the prefixes you want)?


> expect the direct logging to be the same as journal entries. We can achieve
> it through socket but still we cannot have the luxurious prefix as in
> journal.
> 
> I would better like to have a timer unit and corresponding logger unit.
> Based on each service logging capability, let me decide the time.
> 
> So it's like:
> 
> Timer unit - To trigger the logger service for every n seconds (we can set
> it as per the service logging range).
> Logger Service - Run journalctl -u  and the service is designed to
> pass the stdout to the file (using StandardOutput)
> 
> By this way, I can avoid the script running for every few seconds which
> runs journalctl for almost 30+ services and redirect it to the respective
> log files one by one at a moment which is impacting the load.
> 
> Thanks for the help.
> 
> --
> Arjun
> 
> On Tue, Sep 28, 2021 at 7:06 PM Lennart Poettering 
> wrote:
> 
>> On Mo, 27.09.21 15:40, Arjun D R (drarju...@gmail.com) wrote:
>>
>> > Hi Folks,
>> >
>> > Currently we are using systemd-journald for service logging. We run
>> > journalctl for a bunch of services and redirect those to the custom log
>> > files for every few seconds. This takes up the CPU for that particular
>> > time period since we have lot of IO operations as well. We came to know
>> > that systemd version v236+ supports direct logging
>> > (StandardOutput:file:) to the custom log file by the service.
I
>> > would like to use that facility but we don't get the prefix that we used
>> to
>> > get when using the journal.
>> >
>> > Is there a way to prepare a custom patch locally to add the necessary
>> > prefix to the stdout before writing to the custom log file? Is that a
>> good
>> > idea? Any other suggestions?
>>
>>
>> You might define a socket unit 'prefixlogger@.socket' like this:
>>
>>   [Unit]
>>   StopWhenUnneeded=yes
>>
>>   [Socket]
>>   ListenFIFO=/run/prefixlogger.fifo.%i
>>   Service=prefixlogger@%i.service
>>
>> And then a matching service 'prefixlogger@.service':
>>
>>   [Service]
>>   StandardInput=socket
>>   StandardOutput=file:/var/log/foo.log.%i
>>   ExecStart=sed -e 's/^/foo:/'
>>
>> And then in the services that shall run with this:
>>
>>   [Unit]
>>   Wants=prefixlogger@%N.socket
>>   After=prefixlogger@%N.socket
>>
>>   [Service]
>>   ExecStart=/my/service/binary
>>   StandardOutput=file:/run/prefixlogger/fifo.%N
>>
>> (This is all untested, might need some minor changes to actually work,
>> but you get the idea).
>>
>> So what this does is this: first we define a little socket service
>> that can be initialized easily a bunch of times: it listens on a FIFO
>> in the fs and everything it reads from it it writes to some log
>> file. The service is just an invocation of "sed" with standard input
>> being the fifo and standard output being the log file to write to.
>>
>> You then use it by using StandrdOutput=… in your main unit, to connect
>> its stdout/stderr to that fifo. Also, you add deps so that each time a
>> service that tneeds this starts the log prefix service socket for it
>> starts too.
>>
>> Lennart
>>
>> --
>> Lennart Poettering, Berlin
>>

[systemd-devel] Q: write error, watchdog, journald core dump, ordering of entries

[Ulrich Windl]

Hi!

We had a stuck networkc card on a server that seems to have caused the RAID 
controller with two SSDs to be stuck on write as well.
Anyway journald dumped core with this stack:
Oct 05 20:13:25 h19 systemd-coredump[26759]: Process 3321 (systemd-journal) of 
user 0 dumped core.
Oct 05 20:13:25 h19 systemd-coredump[26759]: Coredump diverted to 
/var/lib/systemd/coredump/core.systemd-journal.0.a4eb19afcc314d99936cbdd5542e4fed.3321.163345758500.lz4
Oct 05 20:13:25 h19 systemd-coredump[26759]: Stack trace of thread 3321:
Oct 05 20:13:25 h19 systemd-coredump[26759]: #0  0x7f913492d0c2 
journal_file_append_object (libsystemd-shared-234.so)
Oct 05 20:13:25 h19 systemd-coredump[26759]: #1  0x7f913492dba3 n/a 
(libsystemd-shared-234.so)
Oct 05 20:13:25 h19 systemd-coredump[26759]: #2  0x7f913492fc79 
journal_file_append_entry (libsystemd-shared-234.so)
Oct 05 20:13:25 h19 systemd-coredump[26759]: #3  0x557fe532908d n/a 
(systemd-journald)
Oct 05 20:13:25 h19 systemd-coredump[26759]: #4  0x557fe532b15f n/a 
(systemd-journald)
Oct 05 20:13:25 h19 systemd-coredump[26759]: #5  0x557fe5324664 n/a 
(systemd-journald)
Oct 05 20:13:25 h19 systemd-coredump[26759]: #6  0x557fe5326a80 n/a 
(systemd-journald)
Oct 05 20:13:25 h19 kernel: printk: systemd-coredum: 6 output lines suppressed 
due to ratelimiting

(systemd-234-24.90.1.x86_64 of SLES15 SP2 on x86_64)

journald seems to have restarted later, but I wonder about the ordering of the 
entries following:
Oct 05 20:13:25 h19 systemd-journald[26760]: Journal started
Oct 05 20:13:25 h19 systemd-journald[26760]: System journal 
(/var/log/journal/8695c89eb080463dad2ca9f9aaedf162) is 928.0M, max 4.0G, 3.0G 
free.

Oct 05 20:12:52 h19 systemd[1]: systemd-journald.service: Watchdog timeout 
(limit 3min)!
Oct 05 20:12:52 h19 systemd[1]: systemd-journald.service: Killing process 3321 
(systemd-journal) with signal SIGABRT.
Oct 05 20:13:25 h19 systemd[1]: Starting Flush Journal to Persistent Storage...
Oct 05 20:13:25 h19 systemd[1]: Started Flush Journal to Persistent Storage.

I don't understand why the core dump is logged before the signal being sent and 
the watchdog timeout.

(a journalctl -f running on a terminal produced the entried in the same odd 
order)

Can someone explain?

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Q: write error, watchdog, journald core dump, ordering of entries

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 11.10.2021 um 09:54
in
Nachricht :
> On Mi, 06.10.21 10:29, Ulrich Windl (ulrich.wi...@rz.uni‑regensburg.de)
wrote:
> 
>> Hi!
>>
>> We had a stuck networkc card on a server that seems to have caused the RAID

> controller with two SSDs to be stuck on write as well.
>> Anyway journald dumped core with this stack:
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: Process 3321 (systemd‑journal)
of 
> user 0 dumped core.
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: Coredump diverted to 
>
/var/lib/systemd/coredump/core.systemd‑journal.0.a4eb19afcc314d99936cbdd5542e4
> fed.3321.163345758500.lz4
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: Stack trace of thread 3321:
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: #0  0x7f913492d0c2 
> journal_file_append_object (libsystemd‑shared‑234.so)
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: #1  0x7f913492dba3 n/a 
> (libsystemd‑shared‑234.so)
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: #2  0x7f913492fc79 
> journal_file_append_entry (libsystemd‑shared‑234.so)
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: #3  0x557fe532908d n/a 
> (systemd‑journald)
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: #4  0x557fe532b15f n/a 
> (systemd‑journald)
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: #5  0x557fe5324664 n/a 
> (systemd‑journald)
>> Oct 05 20:13:25 h19 systemd‑coredump[26759]: #6  0x557fe5326a80 n/a 
> (systemd‑journald)
>> Oct 05 20:13:25 h19 kernel: printk: systemd‑coredum: 6 output lines 
> suppressed due to ratelimiting
>>
>> (systemd‑234‑24.90.1.x86_64 of SLES15 SP2 on x86_64)
>>
>> journald seems to have restarted later, but I wonder about the ordering of

> the entries following:
>> Oct 05 20:13:25 h19 systemd‑journald[26760]: Journal started
>> Oct 05 20:13:25 h19 systemd‑journald[26760]: System journal 
> (/var/log/journal/8695c89eb080463dad2ca9f9aaedf162) is 928.0M, max 4.0G,
3.0G 
> free.
>>
>> Oct 05 20:12:52 h19 systemd[1]: systemd‑journald.service: Watchdog timeout

> (limit 3min)!
>> Oct 05 20:12:52 h19 systemd[1]: systemd‑journald.service: Killing process 
> 3321 (systemd‑journal) with signal SIGABRT.
>> Oct 05 20:13:25 h19 systemd[1]: Starting Flush Journal to Persistent 
> Storage...
>> Oct 05 20:13:25 h19 systemd[1]: Started Flush Journal to Persistent
Storage.
>>
>> I don't understand why the core dump is logged before the signal
>> being sent and the watchdog timeout.
> 
> PID 1 logs to journald. PID 1 also runs and supervises
> journald. That's quite a special relationship: PID1 both is client to
> journald and manages it.
> 
> Now when journald hangs due to some underlying IO issue, then it might
> miss the watchdog deadline, and PID 1 might then kill it to get it
> back up. It will log about this to the journal, but given tha tthe
> journal is hanging/being killed it's not going to write the messages
> to disk, the mesages will remain queued in the logging socket for a
> bit. Until eventually journald starts up again, and resumes processing
> log messages. it will then process the messages already queued in the
> sockets from when it was hanging, and thus the order might be
> surprising.

Hi!

Thanks for explaining.
Don't you have some OOB-logging, that is: Log a message before processing the
queue logs.
When killing the journal process it would make sense to log the watchdog
timeout and the killing "OOB".
Of course those OOB messages couldn't actually be logged before journald
restarted, so maybe you'd need a second queue (that is emptied first).

Regards,
Ulrich Windl

> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Removing bold fonts from boot messages

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 12.10.2021 um 12:56
in
Nachricht :
> On Di, 12.10.21 12:09, Frank Steiner (fsteiner‑ma...@bio.ifi.lmu.de) wrote:
> 
>> Hi,
>>
>> after upgrading from SLES 15 SP2 (systemd 2.34) to SP3 (systemd 2.46)
>> the boot messages are not only colored (which I like for seeing failures
>> in red) but partially printed in bold face. This makes messages indeed
>> harder to read on serial console (with amt or ipmi), so I wonder if
>> there is a place where the ascii sequences for colors and font faces
>> are defined and can be adjusted?

Stupid question: If you see bold face at the end of the serial line, wouldn't
changing the terminal type ($TERM) do?
Maybe construct your own terminal capabilities.

> 
> Sounds like in an graphics issue in your terminal emulator, no?
> 
>> Or is there some option to remove the bold face only, but not the colors?
>> systemd.log_color=0 removes all formatting, but I'd like to keep the
>> colors...
> 
> No, this is not configurable. We are not a themeable desktop, sorry.
> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Removing bold fonts from boot messages

[Ulrich Windl]

>>> Frank Steiner  schrieb am 13.10.2021 um 
>>> 18:29 in
Nachricht :
> Ulrich Windl wrote:
> 
>> Stupid question: If you see bold face at the end of the serial line, 
> wouldn't
>> changing the terminal type ($TERM) do?
>> Maybe construct your own terminal capabilities.
> 
> I'd need a TERM that has colors but disallows bold fonts. For some
> reason I wasn't even able to construct a terminfo that would disallow
> colors when using that $TERM inside xterm (and starting a new bash).
> It seems that xterm always has certain capabilities, i.e. "ls --color"
> is always showing colors in xterm, also with TERM=xterm-mono and
> everything else I tried.

You are mixing sender and receiver:
You cannot disable colors in the xterm binary (receiver) via termcap; instead 
you must restrict what the sender outputs via termcap:
Xterm won't add colors when not requested. Also, not all programs use termcap. 
Classic VI and Emacs are probably good example who do use termcap/terminfo.


> 
> Anway, settings a translation to bind "allow-bold-fonts(toggle)"
> to a key in xterm resources allows to block bold fonts whenever
> watching systemd boot messages via ipmi or AMT in a xterm...

That's a translation in the receiver. You could also represent bold by colored 
text, but that's something different.
Also if the ANSI escape sequences are hard-coded (monoculture?), you're out of 
luck.


> 
> 
> -- 
> Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/ 
> Lehrstuhl f. BioinformatikMail: http://www.bio.ifi.lmu.de/~steiner/m/ 
> LMU, Amalienstr. 17   Phone: +49 89 2180-4049
> 80333 Muenchen, Germany   Fax:   +49 89 2180-99-4049
> * Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

[systemd-devel] Antw: [EXT] [systemd‑devel] PIDFile creation logic

[Ulrich Windl]

>>> Kenneth Porter  schrieb am 18.10.2021 um 21:43 in
Nachricht <95B112234248FF54AD014623@[192.168.1.16]>:
> I just installed the new‑to‑EPEL ndppd service and am seeing this in my
log:
> 
> Oct 17 21:10:08 saruman systemd: Can't open PID file 
> /var/run/ndppd/ndppd.pid (yet?) after start: No such file or directory
> 
> Examining the source, I see that the pidfile is created by the child 
> process, not the parent. I'm guessing that systemd is expecting the pidfile

> to exist when the parent exits? (I want to file the issue on the upstream 
> program and want to make sure I understand how this should work.)
> 
> Source in question. Note how daemonize() forks and exits and main() invokes

> daemonize and then creates the pidfile. I'm thinking that daemonize() 
> should create the pidfile before it invokes exit().
> 
> 

I had the same incompatibility problem in my own daemon, and the fix was like
this (still somewhat inefficient):
...
else if ( pid == 0 )
{
/* wait for parent to exit */
while ( getppid() != 1 )
milli_sleep(5);
/* child continues */
...

Regards,
Ulrich

[systemd-devel] systemctl start seems to hang with no status

[Ulrich Windl]

Hi!

After a change in configuration ,updates and a reboot, I saw thgis  probel in
SLES12 SP5 (systemd-228-157.30.1.x86_64):
ntpd.service failed during boot. For some reason /etc/ntp.conf was missing (my
fault).
After restoring /etc/ntp.conf and trying a start, start never seems to finish
and I see no logs that something is even attempted.

Some logs:
Oct 22 12:55:46 v07 start-ntpd[1431]: Time server configuration file,
/etc/ntp.conf does not exist.
Oct 22 12:55:46 v07 systemd[1]: ntpd.service: Control process exited,
code=exited status=6
Oct 22 12:55:46 v07 systemd[1]: Failed to start NTP Server Daemon.
Oct 22 12:55:46 v07 systemd[1]: Dependency failed for Wait for ntpd to
synchronize system clock.
Oct 22 12:55:46 v07 systemd[1]: ntp-wait.service: Job ntp-wait.service/start
failed with result 'dependency'.
Oct 22 12:55:46 v07 systemd[1]: ntpd.service: Unit entered failed state.
Oct 22 12:55:46 v07 systemd[1]: ntpd.service: Failed with result 'exit-code'.
Oct 22 12:55:46 v07 systemd[1]: Reached target System Time Synchronized.

Interestingly I wouldn't expect "Reached target System Time Synchronized.".

Status when "systemctl start ntpd" was running for some time already:
v07:~ # systemctl status ntpd.service
● ntpd.service - NTP Server Daemon
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor
preset: disabled)
  Drop-In: /run/systemd/generator/ntpd.service.d
   └─50-insserv.conf-$time.conf
   Active: activating (auto-restart) (Result: exit-code) since Fri 2021-10-22
12:55:46 CEST; 7min ago
 Docs: man:ntpd(1)
Tasks: 0 (limit: 512)

Oct 22 12:55:46 v07 systemd[1]: ntpd.service: Control process exited, ...=6
Oct 22 12:55:46 v07 systemd[1]: Failed to start NTP Server Daemon.
Oct 22 12:55:46 v07 systemd[1]: ntpd.service: Unit entered failed state.
Oct 22 12:55:46 v07 systemd[1]: ntpd.service: Failed with result 'exit...'.
Hint: Some lines were ellipsized, use -l to show in full.

After significant time:
Oct 22 13:06:47 v07 systemd[1]: ntpd.service: Service RestartSec=11min
expired, scheduling restart.
Oct 22 13:06:47 v07 systemd[1]: Stopped NTP Server Daemon.
Oct 22 13:06:47 v07 systemd[1]: Starting NTP Server Daemon...
Oct 22 13:06:47 v07 ntpd[3998]: ntpd 4.2.8p15@1.3728-o Mon Jun 21 18:17:38 UTC
2021 (1): Starting
Oct 22 13:06:47 v07 ntpd[3998]: Command line: /usr/sbin/ntpd -p
/var/run/ntp/ntpd.pid -g -u ntp:ntp -c /etc/ntp.conf
Oct 22 13:06:47 v07 ntpd[3998]:

Oct 22 13:06:47 v07 ntpd[3998]: ntp-4 is maintained by Network Time
Foundation,
Oct 22 13:06:47 v07 ntpd[3998]: Inc. (NTF), a non-profit 501(c)(3)
public-benefit
...
Oct 22 13:06:47 v07 systemd[1]: Started NTP Server Daemon.

I don't understand: Was the missing ntp.conf file blocking the service for 11
minutes?

v07:~ # systemctl status ntp-wait.service
● ntp-wait.service - Wait for ntpd to synchronize system clock
   Loaded: loaded (/usr/lib/systemd/system/ntp-wait.service; enabled; vendor
preset: disabled)
   Active: inactive (dead)

Oct 22 12:55:46 v07 systemd[1]: Dependency failed for Wait for ntpd to
synchronize system clock.
Oct 22 12:55:46 v07 systemd[1]: ntp-wait.service: Job ntp-wait.service/start
failed with result 'dependency'.

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Question about timestamps in the USER_RECORD spec

[Ulrich Windl]

>>> Arian van Putten  schrieb am 26.10.2021 um 10:41 
>>> in
Nachricht
:
> Hey list,
> 
> I'm reading the https://systemd.io/USER_RECORD/ spec and I have a question
> 
> There are some fields in the USER_RECORD spec which are described as
> "unsigned 64 bit integer values".   Specifically the fields describing
> time.
> 
> However JSON lacks integers and only has doubles [0]; which would mean 53

That's nonsense: JSON _has_ integers, but they are restricted (See: 6.  Numbers 
in RFC 7159)

int = zero / ( digit1-9 *DIGIT )

That was some stupid design decision IMHO.

> bit integer precision is about the maximum we can reach.  It's unclear to

The RFC mentions "IEEE 754-2008 binary64 (double precision) numbers"

> me from the spec whether I should use doubles to encode these fields or use
> strings.  Would it be possible to further clarify it?   If it is indeed a
> number literal; this means the maximum date we can encode is
> 9007199254740991 which corresponds to Tuesday, June 5, 2255   . This
> honestly is too soon in the future for my comfort.  I suggest encoding 64
> bit integers as string literals instead to avoid the truncation problem.
> 
> [0] https://datatracker.ietf.org/doc/html/rfc7159#section-6 

[systemd-devel] Antw: [EXT] Re: Output from `tee' is not showing up in system journal consistently on some systems

[Ulrich Windl]

ANother random idea: Did you experiment with tee option "--output-error=..."?

>>> Mitchel Humpherys  schrieb am 27.10.2021 um 18:50
in
Nachricht
:
> On Wed, Oct 27, 2021 at 1:52 AM Colin Guthrie  wrote:
>>
>> Mitchel Humpherys wrote on 26/10/2021 21:16:
>> > On my Manjaro and Ubuntu systems I'm not seeing all of the "pizza" lines
>> > (logs pasted here [3]). On Debian I see the "pizza" and "pie" lines
>> > interleaved as expected. It also works as expected when I run the script
>> > without systemd on Manjaro.
>> >
>> > Any ideas what's going on here? Why am I not seeing all of the output
>> > from `tee' on some systems?
>> >
>> > [2] https://github.com/mgalgs/systemd-tee-output-issue 
>>
>> Looking at the output there, the only thing I can see that's slightly
>> odd is that the "pie" output comes from the service's main pid (as
>> expected) but as tee is forked off the output there that *is* captured
>> is from a different pid each time.
>>
>> So I wonder if this is some kind of cgroup race thing? i.e. the forked
>> off process is not (always) detected as being part of the unit in
>> question and thus isn't logged against the unit? If this is the case is
>> the a bash trick you can use to buffer the output back to the parent
>> process - e.g. just surrounding the command in parenthesis or something?
>> (I'm not a bash expert so no idea if this would work as a workaround!).
>> Appreciate any code change to work around this isn't ideal anyway, but
>> just trying to help debug the underlying problem.
>>
>> This is just a random guess and might not at all be how the logging or
>> cgroup handing works!
> 
> Interesting thought! I think it's totally expected for `tee' to have a
> different PID. Your comment got me curious about the cgroup situation
> in the working vs. non-working scenarios, but it does appear that the
> `tee' processes are appropriately cgroup'd. I'm pasting some mid-tee
> `status' output below (captured with `while :; do systemctl --user
> status tee_test.service; done > /tmp/statuses.txt' and some grepping).
> 
> First, for reference, here's a mid-tee `status' from my Debian 11
> environment where logging is working as expected:
> 
> ● tee_test.service - Test teeing under systemd
>  Loaded: loaded
> (/home/mgalgs/.config/systemd/user/tee_test.service; disabled; vendor
> preset: enabled)
>  Active: active (running) since Tue 2021-10-26 11:44:07 PDT; 21h ago
>Main PID: 487 (tee_test.sh)
>   Tasks: 2 (limit: 4679)
>  Memory: 3.2M
> CPU: 5min 49.049s
>  CGroup: 
> /user.slice/user-1000.slice/user@1000.service/app.slice/tee_test.service
>  ├─   487 /bin/bash /usr/local/bin/tee_test.sh
>  └─229327 tee -a /tmp/testtee.log
> 
> Oct 27 08:57:46 pi-hole-test tee_test.sh[229282]: i shall eat 75937
> slices of pizza
> Oct 27 08:57:46 pi-hole-test tee_test.sh[487]: i shall eat 75937 slices of 
> pie
> Oct 27 08:57:47 pi-hole-test tee_test.sh[229293]: i shall eat 75938
> slices of pizza
> Oct 27 08:57:47 pi-hole-test tee_test.sh[487]: i shall eat 75938 slices of 
> pie
> Oct 27 08:57:48 pi-hole-test tee_test.sh[229304]: i shall eat 75939
> slices of pizza
> Oct 27 08:57:48 pi-hole-test tee_test.sh[487]: i shall eat 75939 slices of 
> pie
> Oct 27 08:57:49 pi-hole-test tee_test.sh[229315]: i shall eat 75940
> slices of pizza
> Oct 27 08:57:49 pi-hole-test tee_test.sh[487]: i shall eat 75940 slices of 
> pie
> Oct 27 08:57:50 pi-hole-test tee_test.sh[229327]: i shall eat 75941
> slices of pizza
> Oct 27 08:57:50 pi-hole-test tee_test.sh[487]: i shall eat 75941 slices of 
> pie
> 
> And here are two `status'es from my Manjaro environment, where logging
> isn't working consistently. In the first case the logs were missing in
> the system journal. The second case was one of the odd cases where the
> logs actually came through.
> 
> Case 1 (logs missing):
> Status:
> ● tee_test.service - Test teeing under systemd
>  Loaded: loaded
> (/home/mgalgs/.config/systemd/user/tee_test.service; disabled; vendor
> preset: enabled)
>  Active: active (running) since Wed 2021-10-27 08:55:04 PDT; 15min ago
>Main PID: 3791090 (tee_test.sh)
>   Tasks: 2 (limit: 38124)
>  Memory: 1.0M
> CPU: 3.874s
>  CGroup: 
> /user.slice/user-1000.slice/user@1000.service/app.slice/tee_test.service
>  ├─3791090 /bin/bash
> /home/mgalgs/src/systemd-tee-output-issue/tee_test.sh
>  └─3841065 tee -a /tmp/testtee.log
> 
> Oct 27 09:10:11 grinchel tee_test.sh[3791090]: i shall eat 906 slices of
pie
> Oct 27 09:10:12 grinchel tee_test.sh[3791090]: i shall eat 907 slices of
pie
> Oct 27 09:10:13 grinchel tee_test.sh[3791090]: i shall eat 908 slices of
pie
> Oct 27 09:10:14 grinchel tee_test.sh[3791090]: i shall eat 909 slices of
pie
> Oct 27 09:10:15 grinchel tee_test.sh[3791090]: i shall eat 910 slices of
pie
> Oct 27 09:10:16 grinchel tee_test.sh[3791090]: i shall eat 911 slices of
pie
> Oct 27 09:10:17 grinchel tee_test.sh[3791090]: i sh

[systemd-devel] Antw: [EXT] the need for a discoverable sub-volumes specification

[Ulrich Windl]

>>> Chris Murphy  schrieb am 03.11.2021 um 18:52 in
Nachricht
:
> There is a Discoverable Partitions Specification
> http://systemd.io/DISCOVERABLE_PARTITIONS/ 
> 
> The problem with this for Btrfs, ZFS, and LVM is a single volume can
> represent multiple use cases via multiple volumes: subvolumes (btrfs),
> datasets (ZFS), and logical volumes (LVM). I'll just use the term
> sub-volume for all of these, but I'm open to some other generic term.
> 
> None of the above volume managers expose the equivalent of GPT's
> partition type GUID per sub-volume.

At least for LVM I think using a GUID makes little sense as the name of an LV 
is persistent (until renamed) and unique in a system (sometimes even too unique 
when using loop mounts).

> 
> One possibility that's available right now is the sub-volume's name.
> All we need is a spec for that naming convention.
> 
> An early prototype of this idea was posted by Lennart:
> https://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html 
> 
> Lennart previously mentioned elsewhere that this is probably outdated.
> So let's update it and bring it more in line with the purpose and goal
> set out in the discoverable partition spec, which is to obviate the
> need for /etc/fstab.
> 
> 
> -- 
> Chris Murphy

[systemd-devel] Antw: [EXT] Re: systemctl start seems to hang with no status

[Ulrich Windl]

>>> Andy Pieters  schrieb am 04.11.2021 um 20:02 in
Nachricht
:
> On Thu, 4 Nov 2021 at 18:29, Michal Koutný  wrote:
> 
>>
>> ntp-wait.service
>> Wants=time-sync.target
>> Before=time-sync.target
>>
>>
> Ok this is not an answer, but why use ntp and not systemd-timesyncd?

Most likely because it doesn't allow performance monitoring (via mode 6
packets) and is unable to do manycasting (auto-discovering servers).

[systemd-devel] Antw: [EXT] Re: Systemd hang when restarting a service during shutdown

[Ulrich Windl]

>>> Uoti Urpala  schrieb am 08.11.2021 um 14:47 in
Nachricht <27d2e5ff8a9b53e71bb599986060b7fbae469f1d.ca...@pp1.inet.fi>:
> On Mon, 2021-11-08 at 12:05 +0100, Sean Nyekjaer wrote:
>> Regarding,
>> https://github.com/systemd/systemd/issues/21203 
>> 
>> I think the point of the issue missed when the issue got closed.
>> 
>> We have a service that is changing configs for systemd-networkd and
>> issuing a `systemctl restart systemd-networkd`.
>> An other service is checking uptime and issues a `systemctl reboot`,
>> when our max uptime have been exeeced.
>> If restart of systemd-networkd happens while a reboot is in progress,
>> the system will hang "forever" (and continue to pet the watchdog).
> 
> The issue shows you using "systemctl start systemd-reboot". That is not
> the right way to reboot. Use "systemctl reboot" instead. I suspect this
> is related to why the reboot may stop partway: your command does not
> start the reboot tasks in "irreversible" mode, which means that any
> following contrary command, such as explicitly (re)starting a unit that
> was going to be shut down, is going to implicitly cancel the
> conflicting reboot action.
> 
> You should also be using "try-restart" instead of "restart". If your
> intent is to change configs, you want to say "make sure old configs are
> not in use" rather than "enforce that the service is running now". (I
> think making the "restart" command have "start" semantics was a design
> mistake, and the "try-restart"/"restart" pair would have been better
> named "restart"/"start-or-restart".)

Discussable: If the desired effect of "restart" is that the service is running 
with current executables and current configuration, it's OK.
If the desired effect is that the service should be restarted if it was 
running, but not otherwise, it's probably just the wrong command.
Why would one use "restart" when the service shouldn't start?

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] the need for a discoverable sub‑volumes specification

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 19.11.2021 um 10:17
in
Nachricht :
> On Do, 18.11.21 14:51, Chris Murphy (li...@colorremedies.com) wrote:
> 
>> How to do swapfiles?
> 
> Is this really a concept that deserves too much attention? I mean, I
> have the suspicion that half the benefit of swap space is that it can
> act as backing store for hibernation. But swap files are icky for that
> since that means the resume code has to mount the fs first, but given
> the fs is dirty during the hibernation state this is highly problematic.
> 
> Hence, I have the suspicion that if you do swap you should probably do
> swap partitions, not swap files, because it can cover all usecase:
> paging *and* hibernation.

Out of curiosity: What about swap LVs, possibly thin-provisioned ones?

> 
>> Currently I'm creating a "swap" subvolume in the top‑level of the file
>> system and /etc/fstab looks like this
>>
>> UUID=$FSUUID/var/swap   btrfs   noatime,subvol=swap 0 0
>> /var/swap/swapfile1 none swap defaults 0 0
>>
>> This seems to work reliably after hundreds of boots.
>>
>> a. Is this naming convention for the subvolume adequate? Seems like it
>> can just be "swap" because the GPT method is just a single partition
>> type GUID that's shared by multiboot Linux setups, i.e. not arch or
>> distro specific
> 
> I'd still put it one level down, and marke it with some non‑typical
> character so that it is less likely to clash with anything else.
> 
>> b. Is the mount point, /var/swap, OK?
> 
> I see no reason why not.
> 
>> c. What should the additional naming convention be for the swapfile
>> itself so swapon happens automatically?
> 
> To me it appears these things should be distinct: if automatic
> activation of swap files is desirable, then there should probably be a
> systemd generator that finds all suitable files in /var/swap/ and
> generates .swap units for them. This would then work with any kind of
> setup, i.e. independently of the btrfs auto‑discovery stuff. The other
> thing would be the btrfs auto‑disocvery to then actually mount
> something there automatically.
> 
>> Also, instead of /@auto/ I'm wondering if we could have
>> /x‑systemd.auto/ ? This makes it more clearly systemd's namespace, and
>> while I'm a big fan of the @ symbol for typographic history reasons,
>> it's being used in the subvolume/snapshot regimes rather haphazardly
>> for different purposes which might be confusing? e.g. Timeshift
>> expects subvolumes it manages to be prefixed with @. Meanwhile SUSE
>> uses @ for its (visible) root subvolume in which everything else goes.
>> And still ZFS uses @ for their (read‑only) snapshots.
> 
> I try to keep the "systemd" name out of entirely generic specs, since
> there are some people who have an issue with that. i.e. this way we
> tricked even Devuan to adopt /etc/os‑release and the /run/ hierarchy,
> since they probably aren't even aware that these are systemd things.
> 
> Other chars could be used too: /+auto/ sounds OK to me too. or
> /_auto/, or /=auto/ or so.
> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] Re: Authenticated Boot: dm-integrity modes

[Ulrich Windl]

>>> Wol  schrieb am 28.11.2021 um 21:56 in Nachricht
:
> On 28/11/2021 19:56, Adrian Vovk wrote:
>> - Journal mode: is slow. It atomically writes data+hash, so the 
>> situation I describe above can never happen. However, to pull this off 
>> it writes the data twice. Effectively every layer of journaled 
>> dm-integrity will cut write speeds in half. This isn't too bad to 
>> protect the rootfs since writes there will be rare, but it is terrible 
>> for /home. Layering systemd-homed's LUKS+dm-integrity image on top of 
>> that will cut performance in half again. So with the whole setup 
>> proposed by the blog post (even with dm-verity) writes to home will be 
>> limited to 1/4 of the drive's performance and the data will be written 
>> four times over. On top of performance issues, won't writing the data 4x 
>> wear out SSDs faster? Am I missing something?
> 
> Why can't you just enable journalling in systemd-homed, so we have 
> LUKS+dm-integrity-journalling?
> 
> If the user needs to separate / and /home, isn't that just sensible design?
> 
> As for SSDs, the latest ones, as far as I can tell, have a lifespan 
> measured in years even if they're being absolutely hammered by a stress 
> test. If you're really worried about wearing out an SSD, put the journal 
> on rotating rust, but I think those in the know are likely to tell you 
> that the rust will die before the SSD.

Today most storage technology is being replaced for larger capacity before it's 
old enough to fail. Some rusts survive many years.
We had some making it for almost 10 years "24x7". SAo SSD we have is old enough.

> 
> Cheers,
> Wol

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Run reboot as normal user

[Ulrich Windl]

>>> Martin Wilck  schrieb am 01.12.2021 um 10:06 in Nachricht
:
> On Tue, 2021‑11‑30 at 14:11 +0100, Mohamed Ali Fodha wrote:
>> Thanks, but I think using setuid has a security risk for attackers,
>> so I understand there is no so much granularity to manage
>> unprivileged access to systemd in case the polkit is not used.
> 
> You could use setcap to set CAP_SYS_ADMIN capabilities on the
> executable you start for rebooting. I don't see a big difference wrt
> using AmbientCapabilities in a systemd service, as long as you restrict
> the program to be executable only by a certain user or group. Polkit
> can't do much more, either. Its main purpose is to serve logged‑in
> users that want to do certain priviliged actions like mount a volume or
> install software, and trigger pop‑ups that ask for either user or admin
> passwords. IIUC it's overengineered for what you're trying to do,
> unless you want to ask for a password or some other extra
> authorization.

And I wonder what's wrong with allowing the shutdown command for the user in
sudoers.
(sudo $(which shutdown) -r now)

> 
> OTOH, if you use CAP_SYS_ADMIN, you might as well use setuid. Same
> argument ‑ if you restrict the program properly, it comes down to
> exactly the same thing that polkit would do, just far simpler.
> 
> Martin

Re: [systemd-devel] Antw: [EXT] Re: [systemd‑devel] Run reboot as normal user

[Ulrich Windl]

>>> Martin Wilck  schrieb am 01.12.2021 um 10:41 in Nachricht
:
> On Wed, 2021‑12‑01 at 10:24 +0100, Ulrich Windl wrote:
>> > > 
>> 
>> And I wonder what's wrong with allowing the shutdown command for the
>> user in
>> sudoers.
>> (sudo $(which shutdown) ‑r now)
> 
> Sure. I thought sudo might not be installed on that embedded system,
> either. If it is, I'd prefer it over other solutions simply because
> it's more transparent. Capability bits tend to go unnoticed.
> 

Quote from OP: "Previously the guest user was in sudoers (so to run reboot the
systemd
service uses "sudo") but actually our customer wants to remove the guest
user from sudoers."

It was some odd security debate: If someone can operate as guest he/she should
not be able to reboot, while the guest user should be.
(Maybe I misunderstood. Any case some details for the problem are missing)


> Martin

[systemd-devel] Antw: [EXT] Re: Additional Information for Ethernet

[Ulrich Windl]

>>> Greg KH  schrieb am 09.12.2021 um 18:24 in
Nachricht :
> On Thu, Dec 09, 2021 at 03:13:42AM +, Spencer Ku (古世瑜) wrote:
>> Hi All,
>> We are developing the redfish of openbmc, and want to expose some Ethernet

> information through the redfish interface, like packet count, drop count. My

> goal is to calculate the bandwidth / drop package rate of the Ethernet. We 
> can get those counts from Ethernet driver 
> (/sys/class/net/{Ethernet_Name}/statistics), but as far as I know, there is

> no suitable d-bus to read those metrics.

I wonder: What's the problem with just reading that file?

>> 
>> 
>> 
>> My question is that: is it possible to create the new D-bus properties ( 
> like under the /org/freedesktop/{network}/network/{Ethernet_Name}), then
sync 
> the counts to that D-Bus?
> 
> Why not just use netlink for this?  I think it provides this information
> already as part of networking stats.  Have you looked into that?
> 
> thanks,
> 
> greg k-h

[systemd-devel] Antw: Re: Antw: [EXT] Re: [systemd‑devel] the need for a discoverable sub‑volumes specification

[Ulrich Windl]

>>> Chris Murphy  schrieb am 10.12.2021 um 16:59 in
Nachricht
:
> On Mon, Nov 22, 2021 at 3:02 AM Ulrich Windl
>  wrote:
>>
>> >>> Lennart Poettering  schrieb am 19.11.2021 um 
>> >>> 10:17
>> in
>> Nachricht :
>> > On Do, 18.11.21 14:51, Chris Murphy (li...@colorremedies.com) wrote:
>> >
>> >> How to do swapfiles?
>> >
>> > Is this really a concept that deserves too much attention? I mean, I
>> > have the suspicion that half the benefit of swap space is that it can
>> > act as backing store for hibernation. But swap files are icky for that
>> > since that means the resume code has to mount the fs first, but given
>> > the fs is dirty during the hibernation state this is highly problematic.
>> >
>> > Hence, I have the suspicion that if you do swap you should probably do
>> > swap partitions, not swap files, because it can cover all usecase:
>> > paging *and* hibernation.
>>
>> Out of curiosity: What about swap LVs, possibly thin-provisioned ones?
> 
> I don't think that's supported.
> https://listman.redhat.com/archives/linux-lvm/2020-November/msg00039.html 
> 

AFAIK Redhat-based Qubes OS uses it; that's where I first saw it.

> 
> -- 
> Chris Murphy

[systemd-devel] Q: When will WorkingDirectory be checked?

[Ulrich Windl]

Hi!

I have a simple question: When will WorkingDirectory be checked? Specifically: 
Will it be checked before ExecStartPre? I could not get it form the manual page.

Regards,
Ulrich Windl

[systemd-devel] Q: journal logging and "{xyz}"

[Ulrich Windl]

Hi!

I just noticed that systemctl kill logs a (mis-named) service "{__SERVICE__}" 
as "\x7b__SERVICE__\x7d.service".
Is that intended? Are braces considered bad?

(systemd-234-24.90.1.x86_64 of SLES15 SP2)

Regards,
Ulrich Windl

[systemd-devel] Q: journal filtering and unit failures

[Ulrich Windl]

Hi!

debugging a problem I found out some surprising fact:
When filtering the journal with "journalctl -b 
_SYSTEMD_UNIT=logrotate.service", I see an error only once, like this:
-- Logs begin at Mon 2020-11-30 11:35:08 CET, end at Mon 2021-12-27 10:19:31 
CET. --
Dec 18 00:00:20 h16 logrotate[41799]: Failed to kill unit 
\x7b__SERVICE__\x7d.service: Unit \x7b__SERVICE__\x7d.s...
Dec 18 00:00:20 h16 logrotate[41799]: error: error running shared postrotate 
script for '/var/log/iotwatch/MD10/...

However when inspecting the full journal I also see:
Dec 18 00:00:20 h16 systemd[1]: logrotate.service: Main process exited, 
code=exited, status=1/FAILURE
Dec 18 00:00:20 h16 systemd[1]: Failed to start Rotate log files.
Dec 18 00:00:20 h16 systemd[1]: logrotate.service: Unit entered failed state.
Dec 18 00:00:20 h16 systemd[1]: logrotate.service: Failed with result 
'exit-code'.

Actually I am surprised that those unit-related messages are not covered by the 
filter.
Is there a better way to get such related messages when filtering?

Also it seems that logrotate.service that should be triggered daily by a timer 
isn't triggered again after the error above; is that true?
(About the same question can be found in superuser.com, in case someone like to 
write an answer there)

(systemd-234-24.90.1.x86_64 of SLES15 SP2)

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Q: When will WorkingDirectory be checked?

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 20.12.2021 um 13:27
in
Nachricht :
> On Fr, 17.12.21 08:11, Ulrich Windl (ulrich.wi...@rz.uni‑regensburg.de)
wrote:
> 
>> Hi!
>>
>> I have a simple question: When will WorkingDirectory be checked?
>> Specifically: Will it be checked before ExecStartPre? I could not
>> get it form the manual page.
> 
> What do you mean by "checked"?

Checked means "whether it exists".

> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] Re: After= and Wants= doesn't seem to have an effect

[Ulrich Windl]

>>> Andrei Borzenkov  schrieb am 20.12.2021 um 13:27 in
Nachricht <8581dcc7-abdc-4f35-ef41-f2ea02077...@gmail.com>:
> On 20.12.2021 15:06, Christopher Wong wrote:
>> # /etc/systemd/system/iris-detection.service
>> After=temperature-controller.service
>> 
>> # /usr/lib/systemd/system/temperature-controller.service
>> After=iris-detection.service
> 
> This is loop and systemd is free to break it by ignoring some dependency.

A good question would be whether it would be preferable to fail all services 
taking a part in the dependency loop instead.
Maybe then those would be fixed sooner...

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: the need for a discoverable sub-volumes specification

[Ulrich Windl]

>>> Ludwig Nussel  schrieb am 21.12.2021 um 14:57 in
Nachricht <662e1a92-beb4-e1f1-05c9-e0b38e40e...@suse.de>:

...
> The way btrfs is used in openSUSE is based on systems from ten years
> ago. A lot has changed since then. Now with the idea to have /usr on a
> separate read-only subvolume the current model doesn't really work very
> well anymore IMO. So I think there's a window of opportunity to change
> the way openSUSE does things :-)

Oh well, while you are doing so: Also improve support for a separate /boot
volume when snapshotting.

> 
> cu
> Ludwig
> 
> -- 
>  (o_   Ludwig Nussel
>  //\
>  V_/_  http://www.suse.com/ 
> SUSE Software Solutions Germany GmbH, GF: Ivo Totev
> HRB 36809 (AG Nürnberg)

[systemd-devel] Antw: [EXT] Re: Q: journal logging and "{xyz}"

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 27.12.2021 um 09:57 in
Nachricht
:
> On Mon, Dec 27, 2021 at 10:53 AM Ulrich Windl <
> ulrich.wi...@rz.uni-regensburg.de> wrote:
> 
>> Hi!
>>
>> I just noticed that systemctl kill logs a (mis-named) service
>> "{__SERVICE__}" as "\x7b__SERVICE__\x7d.service".
>> Is that intended? Are braces considered bad?
>>
> 
> As per systemd.unit(5),
> 
> *Valid unit names consist of a "name prefix" and a dot and a suffix
> specifying the unit type. The "unit prefix" must consist of one or more
> valid characters (ASCII letters, digits, ":", "-", "_", ".", and "\").*
> 
> Anything else goes through C-style escaping (see systemd-escape) before it
> even leaves systemctl.

Well, but why write "Failed to kill unit \x7b__SERVICE__\x7d.service" when
"unit {__SERVICE__} has an invalid name" (the message I'd suggest)?

I mean: I see no problem _outputting_ the original service name, especially
when it's considered to be an invalid name.

Regards,
Ulrich


> 
> -- 
> Mantas Mikulėnas

[systemd-devel] Antw: [EXT] Re: Q: journal filtering and unit failures

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 27.12.2021 um 10:52 in
Nachricht
:
> On Mon, Dec 27, 2021 at 11:26 AM Ulrich Windl <
> ulrich.wi...@rz.uni-regensburg.de> wrote:
> 
>> Hi!
>>
>> debugging a problem I found out some surprising fact:
>> When filtering the journal with "journalctl -b
>> _SYSTEMD_UNIT=logrotate.service", I see an error only once, like this:
>> -- Logs begin at Mon 2020-11-30 11:35:08 CET, end at Mon 2021-12-27
>> 10:19:31 CET. --
>> Dec 18 00:00:20 h16 logrotate[41799]: Failed to kill unit
>> \x7b__SERVICE__\x7d.service: Unit \x7b__SERVICE__\x7d.s...
>> Dec 18 00:00:20 h16 logrotate[41799]: error: error running shared
>> postrotate script for '/var/log/iotwatch/MD10/...
>>
>> However when inspecting the full journal I also see:
>> Dec 18 00:00:20 h16 systemd[1]: logrotate.service: Main process exited,
>> code=exited, status=1/FAILURE
>> Dec 18 00:00:20 h16 systemd[1]: Failed to start Rotate log files.
>> Dec 18 00:00:20 h16 systemd[1]: logrotate.service: Unit entered failed
>> state.
>> Dec 18 00:00:20 h16 systemd[1]: logrotate.service: Failed with result
>> 'exit-code'.
>>
>> Actually I am surprised that those unit-related messages are not covered
>> by the filter.
>>
> 
> They're not generated **by** a process within logrotate.service, and
> journald does not allow a (fake) _SYSTEMD_UNIT to be set – not even if the
> sender is pid1. The same also applies to messages logged by
> systemd-coredump.
> 
> 
>> Is there a better way to get such related messages when filtering?
>>
> 
> Use `journalctl -u logrotate`, which combines several filters to get all
> messages related to the specified unit.
> 
> The actual filtering used by -u (as seen with SYSTEMD_LOG_LEVEL=debug) is a
> union of four filters – the equivalent command would be:
> 
> journalctl _SYSTEMD_UNIT=logrotate.service \
>   + MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1
> _UID=0 COREDUMP_UNIT=logrotate.service \
>   + _PID=1 UNIT=logrotate.service \
>   + _UID=0 OBJECT_SYSTEMD_UNIT=logrotate.service
> 
> The fourth is a generic filter (any process running as root can specify
> OBJECT_SYSTEMD_UNIT=) which I think came a bit later than the special-case
> pid1 and coredump ones.

OK, thanks.  Re-reading the manual page, I'd suggest to add references to
options like "-u" to the early section that deals with "MATCHES". A new user
(like I) will divert to the wrong path using "_SYSTEMD_UNIT=" when probably
"-u" was intended actually. Also the word "filter" occurs for option "-p", but
not for "-u". Maybe list all "filtering" options early in the DESCRIPTION...

Regards,
Ulrich


> 
> -- 
> Mantas Mikulėnas

[systemd-devel] Antw: Re: Antw: [EXT] Re: Q: journal logging and "{xyz}"

[Ulrich Windl]

>>> Wols Lists  schrieb am 27.12.2021 um 13:18 in
Nachricht <8802ee4d-230d-8013-553d-8615515ce...@youngman.org.uk>:
> On 27/12/2021 12:09, Ulrich Windl wrote:
>> Well, but why write "Failed to kill unit \x7b__SERVICE__\x7d.service" when
>> "unit {__SERVICE__} has an invalid name" (the message I'd suggest)?
>> 
>> I mean: I see no problem_outputting_  the original service name, especially
>> when it's considered to be an invalid name.
> 
> I guess it's seen as a possible attack vector.

Well, usually I wouldn't consider braces to be "evil" characters (unless used 
in LOG4J, maybe).

> 
> It's not paranoia if they really are out to get you.
> 
> Cheers,
> Wol

[systemd-devel] Antw: Re: Antw: [EXT] Re: the need for a discoverable sub-volumes specification

[Ulrich Windl]

>>> Chris Murphy  schrieb am 28.12.2021 um 20:02 in
Nachricht
:
> On Mon, Dec 27, 2021 at 3:40 AM Ulrich Windl
>  wrote:
>>
>> >>> Ludwig Nussel  schrieb am 21.12.2021 um 14:57 in
>> Nachricht <662e1a92-beb4-e1f1-05c9-e0b38e40e...@suse.de>:
>>
>> ...
>> > The way btrfs is used in openSUSE is based on systems from ten years
>> > ago. A lot has changed since then. Now with the idea to have /usr on a
>> > separate read-only subvolume the current model doesn't really work very
>> > well anymore IMO. So I think there's a window of opportunity to change
>> > the way openSUSE does things :-)
>>
>> Oh well, while you are doing so: Also improve support for a separate /boot
>> volume when snapshotting.
> 
> Yeah how to handle /boot gives me headaches. We have a kind of
> rollback, the possibility of choosing among kernels. But which kernels
> are bootable depends on the /usr its paired with. We need a mechanism
> to match /boot and /usr together, so that the user doesn't get stuck
> choosing a kernel version for which the modules don't exist in an
> older generation /usr. And then does this imply some additional
> functionality in the bootloader to achieve it, or should this
> information be fully encapsulated in Boot Loader Spec compliant
> snippets?

Well, it's a bit off-topic, but:
I had to use a separate /boot device due to the way my device is encrypted.
Everything had seemed to work fine until I tried to install the AMD graphics 
driver
that resulted in a failed boot (I can't remember the exact details).
Unfortunately AFTER that I found out that there were no snapshots of /boot 
created (only of the root filesystem).
(So recovery was a bit messy)
Meanwhile I had enabled /boot snapshots also, but still no boot snapshots are 
offered in the GRUB menu...

Basically that's what I had meant.

Regards,
Ulrich

> 
> 
> -- 
> Chris Murphy

[systemd-devel] Antw: [EXT] [systemd‑devel] Why is using fstab the preferred approach according to systemd.mount man page?

[Ulrich Windl]

>>> "Manuel Wagesreither"  schrieb am 10.01.2022 um 11:43 in
Nachricht <759bd805-ab37-4dc9-93cd-137668acf...@www.fastmail.com>:
> Hi all,
> 
> currently, the systemd.mount man page [1] claims that "In general, 
> configuring mount points through /etc/fstab is the preferred approach". Why 
> is that? One would assume systemd proposing use of systemd.mount units over 
> /etc/fstab.

Because it's much easier to configure?

> 
> [1] 
> https://www.freedesktop.org/software/systemd/man/systemd.mount.html#fstab 
> 
> Regards, Manuel

[systemd-devel] Q: "systemd[1]: var-tmp-AP_0x6tWaHS.mount: Succeeded."

[Ulrich Windl]

Hi!

When upgrading SLES15 to SP3, a newer version of systemd was installed 
(246.16+suse.191.g3850086c65).
Since then I see new journal messages like these that I cannot associate with a 
unit:

Jan 27 09:29:30 h16 systemd[1]: var-tmp-AP_0xC5KDJP.mount: Succeeded.
Jan 27 09:29:30 h16 systemd[22591]: var-tmp-AP_0xC5KDJP.mount: Succeeded.

Where do these messages originate from, and couldn't they be improved? Or is it 
some debug-leftover?
I do not see corresponding names in /var/tmp.

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: Q: "systemd[1]: var-tmp-AP_0x6tWaHS.mount: Succeeded."

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 28.01.2022 um 10:27 in
Nachricht
:
> On Fri, Jan 28, 2022 at 10:50 AM Ulrich Windl <
> ulrich.wi...@rz.uni-regensburg.de> wrote:
> 
>> Hi!
>>
>> When upgrading SLES15 to SP3, a newer version of systemd was installed
>> (246.16+suse.191.g3850086c65).
>> Since then I see new journal messages like these that I cannot associate
>> with a unit:
>>
>> Jan 27 09:29:30 h16 systemd[1]: var-tmp-AP_0xC5KDJP.mount: Succeeded.
>> Jan 27 09:29:30 h16 systemd[22591]: var-tmp-AP_0xC5KDJP.mount: Succeeded.
>>
>> Where do these messages originate from, and couldn't they be improved? Or
>> is it some debug-leftover?
>> I do not see corresponding names in /var/tmp.
>>
> 
> If I understand correctly, the messages indicate that the filesystem was
> *unmounted*, and the same program which did mounting/unmounting immediately
> cleaned up the mountpoint as well.
> 
> (systemd reacts to external mounts as those also contribute to
> dependencies.)
> 
> If OpenSuSE has the kernel audit subsystem enabled, try using `auditctl` to
> monitor a) what process executes mount-related syscalls, b) what process
> creates directories under /var/tmp.

Thanks,

probably these messages are related to mounting a virtual CD, as nearby are
these messages:
Jan 27 09:29:29 h16 kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
Jan 27 09:29:29 h16 kernel: ISO 9660 Extensions: RRIP_1991A
Jan 27 09:29:30 h16 systemd[1]: var-tmp-AP_0xC5KDJP.mount: Succeeded.
Jan 27 09:29:30 h16 systemd[22591]: var-tmp-AP_0xC5KDJP.mount: Succeeded.
Jan 27 09:29:30 h16 kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
Jan 27 09:29:30 h16 kernel: ISO 9660 Extensions: RRIP_1991A
Jan 27 09:29:31 h16 systemd[22591]: var-tmp-AP_0x6tWaHS.mount: Succeeded.
Jan 27 09:29:31 h16 systemd[1]: var-tmp-AP_0x6tWaHS.mount: Succeeded.

Still I wonder what this is all about (systemd finding a CD, mounting it, just
to find out that no-one needs it?)...

> 
> -- 
> Mantas Mikulėnas

[systemd-devel] Antw: Re: [EXT] Re: Q: "systemd[1]: var-tmp-AP_0x6tWaHS.mount: Succeeded."

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 28.01.2022 um 12:36 in
Nachricht
:
> On Fri, Jan 28, 2022, 11:59 Ulrich Windl 
> wrote:
> 
>> >>> Mantas Mikulenas  schrieb am 28.01.2022 um 10:27 in
>> Nachricht
>> :
>> > On Fri, Jan 28, 2022 at 10:50 AM Ulrich Windl <
>> > ulrich.wi...@rz.uni-regensburg.de> wrote:
>> >
>> >> Hi!
>> >>
>> >> When upgrading SLES15 to SP3, a newer version of systemd was installed
>> >> (246.16+suse.191.g3850086c65).
>> >> Since then I see new journal messages like these that I cannot associate
>> >> with a unit:
>> >>
>> >> Jan 27 09:29:30 h16 systemd[1]: var-tmp-AP_0xC5KDJP.mount: Succeeded.
>> >> Jan 27 09:29:30 h16 systemd[22591]: var-tmp-AP_0xC5KDJP.mount:
>> Succeeded.
>> >>
>> >> Where do these messages originate from, and couldn't they be improved?
>> Or
>> >> is it some debug-leftover?
>> >> I do not see corresponding names in /var/tmp.
>> >>
>> >
>> > If I understand correctly, the messages indicate that the filesystem was
>> > *unmounted*, and the same program which did mounting/unmounting
>> immediately
>> > cleaned up the mountpoint as well.
>> >
>> > (systemd reacts to external mounts as those also contribute to
>> > dependencies.)
>> >
>> > If OpenSuSE has the kernel audit subsystem enabled, try using `auditctl`
>> to
>> > monitor a) what process executes mount-related syscalls, b) what process
>> > creates directories under /var/tmp.
>>
>> Thanks,
>>
>> probably these messages are related to mounting a virtual CD, as nearby are
>> these messages:
>> Jan 27 09:29:29 h16 kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
>> Jan 27 09:29:29 h16 kernel: ISO 9660 Extensions: RRIP_1991A
>> Jan 27 09:29:30 h16 systemd[1]: var-tmp-AP_0xC5KDJP.mount: Succeeded.
>> Jan 27 09:29:30 h16 systemd[22591]: var-tmp-AP_0xC5KDJP.mount: Succeeded.
>> Jan 27 09:29:30 h16 kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
>> Jan 27 09:29:30 h16 kernel: ISO 9660 Extensions: RRIP_1991A
>> Jan 27 09:29:31 h16 systemd[22591]: var-tmp-AP_0x6tWaHS.mount: Succeeded.
>> Jan 27 09:29:31 h16 systemd[1]: var-tmp-AP_0x6tWaHS.mount: Succeeded.
>>
>> Still I wonder what this is all about (systemd finding a CD, mounting it,
>> just
>> to find out that no-one needs it?)...
>>
> 
> Why do you think systemd is finding and/or mounting it?

As I see no other log message around that's the most likely explanation IMHO, 
especially as systemd interferes with everything these days.

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: systemd killing processes on monitor wakeup?

[Ulrich Windl]

>>> Raman Gupta  schrieb am 29.01.2022 um 04:28 in 
>>> Nachricht
:
>> 
>> Try to set the systemd user instance's log level to 'debug'; I'm guessing
>> it's not that systemd kills processes directly but that something triggers
>> a 'systemctl stop' of the session .scope that they were in.
> 
> 
> Here are the logs at debug level with some annotations inline:
> 
> **
> 
> Jan 28 21:57:34 edison plasmashell[3114743]: KCrash: Application
> 'plasmashell' crashing...
> Jan 28 21:57:34 edison plasmashell[3114743]: KCrash: Attempting to start
> /usr/libexec/drkonqi

[...]

This may not answer your question at all, but I think the real culprit is 
modern technology like display port, that allows the OS to detect when a 
monitor connects or disconnects.
I had real trouble with Windows 10 after using a new monitor and a KVM-switch: 
Every time I switched the KVM between two computers, Windows reset the graphics 
resolution to some "standard" (like 800x600), also moving all the windows on 
the desktop to the upper left corner. Besides that KVM switching is slower than 
ever. Back in the times of PS/2 and analog VGA connector I could switch twice 
within a second without a problem.

So for your problem I guess the application cannot really handle the monitor 
hotplug event.

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Udevd and dev file creation

[Ulrich Windl]

>>> Nishant Nayan  schrieb am 30.01.2022 um 04:45 
>>> in
Nachricht
:
> Hi,
>  Does creating a dev file in /dev creates a uevent? Does a dev file
> creation considered as a "device attached" to the machine?

I'd guess "Most likely not" as you can create /dev files without actually 
having the hardware.

> 
> 
> Regards
> Nishant Nayan

[systemd-devel] Antw: Re: Antw: [EXT] [systemd‑devel] Why is using fstab the preferred approach according to systemd.mount man page?

[Ulrich Windl]

>>> Thomas HUMMEL  schrieb am 07.02.2022 um 18:36 in
Nachricht <3aa03772-1981-eea4-ecd2-c50b9c0d0...@pasteur.fr>:

> 
> On 10/01/2022 21:50, Zbigniew Jędrzejewski-Szmek wrote:
> 
>> Pretty much. There isn't any big benefit to using mount units (since the
>> fstab syntax supports all the important use cases), and people are
familiar
>> with fstab. More tools support it too.
> 
> Hello,
> 
> well although I'm not currently using it, I can see one :
> 
> it may be easier to configure .mount units independently (like dropping 
> a config file into a .d/ directory) instead of editing one single file 
> when done with tools like ansible for instance where you have to regexp 
> match lines to edit or use blockinfile like strategies ?

Amazingly that seems to be one case where a human has less problems (adding or
changing a line in /etc/fstab) than tools do.
Maybe that's the true state of artificial intelligence...
For a human mount units are much more complicated (IMHO).

Regards,
Ulrich

> 
> Thanks
> 
> --
> Thomas HUMMEL

[systemd-devel] Antw: [EXT] Need a systemd unit example that checks /etc/fstab for modification and sends a text message

[Ulrich Windl]

 >>> Tony Rodriguez  schrieb am 09.02.2022 um 02:27 in
Nachricht :
> From my understanding, it is frowned by systemd developers to 
> "automatically" reload systemd via "systemctl daemon-reload" when 
> /etc/fstab is modified.  Guessing this is why such functionality hasn't 
> already been incorporated into systemd.  However, I would like to send a 
> simple text message. Instructing users to manually invoke "systemctl  
> deamon-reload" after modifying /etc/fstab via dmesg command or inside 
> /var/log/messages.

The real mess is that a traditional mount works as soon as you have modified 
/etc/fstab. With systemd there are generators that convert fstab to units, and 
the generators have to be triggered, and the new units have to be read.
This is a simple example how things can be made more complicated to use and 
understand.

> 
> Unsure how to do so inside a systemd UNIT.  Will someone please provide 
> an example how to do so?

I had a similar scenario, but the idea of a unit triggering a generator that 
does reload the daemon in turn, just to be able to use the units that were 
created by the generator is just some unreliable mess (compared to LSB scripts 
where you simply started the script and all was fine).

Regards,
Ulrich

> 
> Thanks

[systemd-devel] Help understanding "notify" (SLES15 SP3 restarting smartd)

[Ulrich Windl]

Hi!

I have a support case for SLES15 SP3 (systemd-246.16-7.33.1.x86_64) where 
smartd is restarted for no obvious reason. Support says everything is fine, but 
I disagree.
Specifically smartd.service uses:

[Service]
Type=notify
EnvironmentFile=-/var/lib/smartmontools/smartd_opts
ExecStart=/usr/sbin/smartd -n $smartd_opts
ExecReload=/bin/kill -HUP $MAINPID

And mostly I wonder to what types of changes "notify" does react.
Specifically the smartd_opts was never changed, but stil lsmartd is restarted 
(by SIGTERM, not SIGHUP, BTW).

h16:~ # ll /var/lib/smartmontools/smartd_opts
-rw-r--r-- 1 root root 74 Feb  9 00:03 /var/lib/smartmontools/smartd_opts
h16:~ # stat /var/lib/smartmontools/smartd_opts
  File: /var/lib/smartmontools/smartd_opts
  Size: 74  Blocks: 8  IO Block: 4096   regular file
Device: 3ah/58d Inode: 34314   Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
Access: 2022-02-09 11:17:21.464795678 +0100
Modify: 2022-02-09 00:03:12.0 +0100
Change: 2022-02-09 00:05:01.012019105 +0100
 Birth: 2020-11-27 09:05:09.871171180 +0100
The file just contains:
h16:~ # cat /var/lib/smartmontools/smartd_opts
# Generated by /usr/lib/smartmontools/generate_smartd_opts
smartd_opts=""
--
(Access was the time when I had looked into it)
Logs near 00:03:12:
Feb 09 00:03:12 h16 systemd[1]: Starting Generate issue file for login 
session...
Feb 09 00:03:12 h16 systemd[1]: Starting Update smartd options...
Feb 09 00:03:13 h16 systemd[1]: issue-generator.service: Succeeded.
Feb 09 00:03:13 h16 systemd[1]: Finished Generate issue file for login session.
Feb 09 00:03:13 h16 smartd[40401]: smartd received signal 15: Terminated
Feb 09 00:03:13 h16 systemd[1]: Stopping Self Monitoring and Reporting 
Technology (SMART) Daemon...
Feb 09 00:03:13 h16 smartd[40401]: Device: /dev/bus/0 [megaraid_disk_00] [SAT], 
state written to /var/lib/smartmontools/smartd...
Feb 09 00:03:13 h16 smartd[40401]: Device: /dev/bus/0 [megaraid_disk_01] [SAT], 
state written to /var/lib/smartmontools/smartd...
Feb 09 00:03:13 h16 smartd[40401]: smartd is exiting (exit status 0)
Feb 09 00:03:13 h16 systemd[1]: smartd.service: Succeeded.
Feb 09 00:03:13 h16 systemd[1]: Stopped Self Monitoring and Reporting 
Technology (SMART) Daemon.
Feb 09 00:03:13 h16 systemd[1]: Starting Self Monitoring and Reporting 
Technology (SMART) Daemon...
Feb 09 00:03:13 h16 smartd[11734]: smartd 7.0 2019-05-21 r4917 
[x86_64-linux-5.3.18-150300.59.43-default] (SUSE RPM)

Logs near 00:05:01:
Feb 09 00:01:23 h16 systemd[1]: Started DATA-PROTECTOR-INET (172.20.16.3:44300).
...
Feb 09 00:05:11 h16 systemd[1]: 
omni@168-172.20.16.16:-172.20.16.3:44300.service: Succeeded.

So out backup software was active in that interval, and one feature it has is 
to reset the access times of the files.
(In the past that feature also has ruined SUSE's cron.daily jonbs as they were 
querying the wrong timestamp, but refused to fix it)

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: Help understanding "notify" (SLES15 SP3 restarting smartd)

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 10.02.2022 um 10:34 in
Nachricht
:
> On Thu, Feb 10, 2022 at 11:21 AM Ulrich Windl <
> ulrich.wi...@rz.uni-regensburg.de> wrote:
> 
>> Hi!
>>
>> I have a support case for SLES15 SP3 (systemd-246.16-7.33.1.x86_64) where
>> smartd is restarted for no obvious reason. Support says everything is
fine,
>> but I disagree.
>> Specifically smartd.service uses:
>>
>> [Service]
>> Type=notify
>> EnvironmentFile=-/var/lib/smartmontools/smartd_opts
>> ExecStart=/usr/sbin/smartd -n $smartd_opts
>> ExecReload=/bin/kill -HUP $MAINPID
>>
>> And mostly I wonder to what types of changes "notify" does react.
>>
> 
> None. It's not an activation mechanism and not a configuration monitoring
> mechanism – it's a readiness indication mechanism.
> 
> Type=notify expects the service to send a message to a Unix socket (at
> $NOTIFY_SOCKET) indicating that it is "ready". (And optionally some custom
> text to show in 'systemctl status'.)

Mantas,

thanks for explaining; obviously I was wrong. But still: How can I find out
why the smartd.service restarts?

Regards,
Ulrich

> 
> -- 
> Mantas Mikulėnas

[systemd-devel] Antw: [EXT] Re: Help understanding "notify" (SLES15 SP3 restarting smartd)

[Ulrich Windl]

>>> Ulrich Windl schrieb am 10.02.2022 um 11:16 in Nachricht <6204E61B.426 :
161 :
60728>:
>>>> Mantas Mikulenas  schrieb am 10.02.2022 um 10:34 in
> Nachricht
> :
> > On Thu, Feb 10, 2022 at 11:21 AM Ulrich Windl <
> > ulrich.wi...@rz.uni-regensburg.de> wrote:
> > 
> >> Hi!
> >>
> >> I have a support case for SLES15 SP3 (systemd-246.16-7.33.1.x86_64)
where
> >> smartd is restarted for no obvious reason. Support says everything is
fine,
> >> but I disagree.
> >> Specifically smartd.service uses:
> >>
> >> [Service]
> >> Type=notify
> >> EnvironmentFile=-/var/lib/smartmontools/smartd_opts
> >> ExecStart=/usr/sbin/smartd -n $smartd_opts
> >> ExecReload=/bin/kill -HUP $MAINPID
> >>
> >> And mostly I wonder to what types of changes "notify" does react.
> >>
> > 
> > None. It's not an activation mechanism and not a configuration monitoring
> > mechanism – it's a readiness indication mechanism.
> > 
> > Type=notify expects the service to send a message to a Unix socket (at
> > $NOTIFY_SOCKET) indicating that it is "ready". (And optionally some
custom
> > text to show in 'systemctl status'.)
> 
> Mantas,
> 
> thanks for explaining; obviously I was wrong. But still: How can I find out

> why the smartd.service restarts?

Hi!

I digged further into it:
smartd_generate_opts.path has:
[Path]
Unit=smartd_generate_opts.service
PathChanged=/etc/sysconfig/smartmontools

smartd_generate_opts.service has:
[Service]
Type=oneshot
ExecStart=/usr/lib/smartmontools/generate_smartd_opts

Finally /usr/lib/smartmontools/generate_smartd_opts is not documented.

But running that program causes a restart of smartd!

h16:~ # stat /etc/sysconfig/smartmontools
  File: /etc/sysconfig/smartmontools
  Size: 1377Blocks: 8  IO Block: 4096   regular file
Device: 2dh/45d Inode: 92175   Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
Access: 2022-02-10 00:13:44.843774074 +0100
Modify: 2022-01-27 09:19:36.0 +0100
Change: 2022-02-09 00:03:12.774336278 +0100
 Birth: 2020-11-27 09:05:09.867171140 +0100

The "Change" date was most likely caused by backup software resetting the
Access time.

So back to systemd: What time stamp does it use for PathChanged=?
(I failed to find documentation in the obvious systemd.* manual pages; whare
is it documented?)

Regards,
Ulrich

> 
> Regards,
> Ulrich
> 
> > 
> > -- 
> > Mantas Mikulėnas
> 
> 
> 
> 

[systemd-devel] Antw: [EXT] Re: Run "ipmitool power cycle" after lib/systemd/system-shutdown scripts

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 10.02.2022 um 11:31
in
Nachricht :
> On Mi, 09.02.22 22:05, Etienne Champetier (champetier.etie...@gmail.com) 
> wrote:
> 
>> Hello systemd hackers,
>>
>> After flashing the firmware of some pcie card I need to power cycle
>> the server to finish the flashing process.
>> For now I have a simple script in lib/systemd/system-shutdown/ running
>> "ipmitool power cycle" but I would like to make sure it runs after
>> other scripts like fwupd.shutdown or mdadm.shutdown
>>
>> Is there any way to have systemd cleanly power cycle my server instead
>> of rebooting it ?
> 
> What does "power cycle" entail that "reboot" doesnt? i.e. why doesn't
> "systemctl reboot" suffice?

My guess is that some smart cards with their own foirmware and CPu do not
reboot unless they are power cycled, so maybe if the firmware upgrade on the
card does not force it to reboot, it my need a power cycle.

> 
> /usr/lib/systemd/system-shutdown/ drop-ins are executed before the OS
> transitions back into the initrd — the initrd will then detach the
> root fs (i.e. undo what it attached at boot) and actually reboot. This
> means if your command turns off the power source you should stick it
> in the initrd's shutdown logic, and not into
> /usr/lib/systemd/system-shutdown/. If you are using RHEL this means
> into dracut. But adding it there is something to better discuss with
> the dracut community than here.

My guess is that it would be handled best by some special GRUB boot menu entry
(like "boot 'power cycle' once).

Regards,
Ulrich

> 
> Lennart
> 
> --
> Lennart Poettering, Berlin

[systemd-devel] Antw: Re: [EXT] Re: Run "ipmitool power cycle" after lib/systemd/system-shutdown scripts

[Ulrich Windl]

>>> Etienne Champetier  schrieb am 10.02.2022 um
12:47 in Nachricht
:
> Le jeu. 10 févr. 2022 à 11:49, Ulrich Windl
>  a écrit :
>>
>> >>> Lennart Poettering  schrieb am 10.02.2022 um
11:31
>> in
>> Nachricht :
>> > On Mi, 09.02.22 22:05, Etienne Champetier (champetier.etie...@gmail.com)
>> > wrote:
>> >
>> >> Hello systemd hackers,
>> >>
>> >> After flashing the firmware of some pcie card I need to power cycle
>> >> the server to finish the flashing process.
>> >> For now I have a simple script in lib/systemd/system-shutdown/ running
>> >> "ipmitool power cycle" but I would like to make sure it runs after
>> >> other scripts like fwupd.shutdown or mdadm.shutdown
>> >>
>> >> Is there any way to have systemd cleanly power cycle my server instead
>> >> of rebooting it ?
>> >
>> > What does "power cycle" entail that "reboot" doesnt? i.e. why doesn't
>> > "systemctl reboot" suffice?
>>
>> My guess is that some smart cards with their own firmware and CPu do not
>> reboot unless they are power cycled, so maybe if the firmware upgrade on
the
>> card does not force it to reboot, it my need a power cycle.
>>
>> >
>> > /usr/lib/systemd/system-shutdown/ drop-ins are executed before the OS
>> > transitions back into the initrd — the initrd will then detach the
>> > root fs (i.e. undo what it attached at boot) and actually reboot. This
>> > means if your command turns off the power source you should stick it
>> > in the initrd's shutdown logic, and not into
>> > /usr/lib/systemd/system-shutdown/. If you are using RHEL this means
>> > into dracut. But adding it there is something to better discuss with
>> > the dracut community than here.
>>
>> My guess is that it would be handled best by some special GRUB boot menu 
> entry
>> (like "boot 'power cycle' once).
> 
> This is pretty clean but it means going through "BIOS" init twice
> which can be pretty long on physical servers

Hi!

Of course I have a better solution: Use an external server and just before
restarting send a command to that server that will, after a carefully tuned
delay, will trigger the IPMI power-cycle remotely ;-)
I believe that it's almost impossible to trigger a power cycle through IPMI
while also doing a clean shutdown/reboot, unless the the IPMI features a
delayed execution itself.

Regards,
Ulrich

> 
> Etienne
> 
>> Regards,
>> Ulrich
>>
>> >
>> > Lennart
>> >
>> > --
>> > Lennart Poettering, Berlin

[systemd-devel] Antw: Re: Antw: Re: [EXT] Re: Run "ipmitool power cycle" after lib/systemd/system-shutdown scripts

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 10.02.2022 um 14:15 in
Nachricht
:
> On Thu, Feb 10, 2022 at 3:11 PM Ulrich Windl <
> ulrich.wi...@rz.uni-regensburg.de> wrote:
> 
>> >>> Etienne Champetier  schrieb am
>> 10.02.2022 um
>> 12:47 in Nachricht
>> :
>> > Le jeu. 10 févr. 2022 à 11:49, Ulrich Windl
>> >  a écrit :
>> >>
>> >> >>> Lennart Poettering  schrieb am 10.02.2022
>> um
>> 11:31
>> >> in
>> >> Nachricht :
>> >> > On Mi, 09.02.22 22:05, Etienne Champetier (
>> champetier.etie...@gmail.com)
>> >> > wrote:
>> >> >
>> >> >> Hello systemd hackers,
>> >> >>
>> >> >> After flashing the firmware of some pcie card I need to power cycle
>> >> >> the server to finish the flashing process.
>> >> >> For now I have a simple script in lib/systemd/system-shutdown/
>> running
>> >> >> "ipmitool power cycle" but I would like to make sure it runs after
>> >> >> other scripts like fwupd.shutdown or mdadm.shutdown
>> >> >>
>> >> >> Is there any way to have systemd cleanly power cycle my server
>> instead
>> >> >> of rebooting it ?
>> >> >
>> >> > What does "power cycle" entail that "reboot" doesnt? i.e. why doesn't
>> >> > "systemctl reboot" suffice?
>> >>
>> >> My guess is that some smart cards with their own firmware and CPu do
not
>> >> reboot unless they are power cycled, so maybe if the firmware upgrade
on
>> the
>> >> card does not force it to reboot, it my need a power cycle.
>> >>
>> >> >
>> >> > /usr/lib/systemd/system-shutdown/ drop-ins are executed before the OS
>> >> > transitions back into the initrd — the initrd will then detach the
>> >> > root fs (i.e. undo what it attached at boot) and actually reboot.
This
>> >> > means if your command turns off the power source you should stick it
>> >> > in the initrd's shutdown logic, and not into
>> >> > /usr/lib/systemd/system-shutdown/. If you are using RHEL this means
>> >> > into dracut. But adding it there is something to better discuss with
>> >> > the dracut community than here.
>> >>
>> >> My guess is that it would be handled best by some special GRUB boot
>> menu
>> > entry
>> >> (like "boot 'power cycle' once).
>> >
>> > This is pretty clean but it means going through "BIOS" init twice
>> > which can be pretty long on physical servers
>>
>> Hi!
>>
>> Of course I have a better solution: Use an external server and just before
>> restarting send a command to that server that will, after a carefully
tuned
>> delay, will trigger the IPMI power-cycle remotely ;-)
>> I believe that it's almost impossible to trigger a power cycle through
IPMI
>> while also doing a clean shutdown/reboot, unless the the IPMI features a
>> delayed execution itself.
>>
> 
> Asking the BMC to power the system off is no different than asking the
> system to power itself off – what makes it a "clean shutdown" is stopping
> services, syncing and unmounting filesystems, etc. So in theory these
> shutdown hooks should do the job.
> 
> Though a better place would be a "shutdown initramfs" which runs from a
> tmpfs after *all* storage has been unmounted. I think Dracut has that, and
> explicitly specifies the order for its shutdown hooks, so it should be
> possible to put ipmitool there.

Another solution comes to my mind: How about programming the RTC for
time-based wakeup (say in 5 minutes from now), then "shutdown -h"...


> 
> -- 
> Mantas Mikulėnas

[systemd-devel] Antw: [EXT] Re: Run "ipmitool power cycle" after lib/systemd/system-shutdown scripts

[Ulrich Windl]

>>> Paul Menzel  schrieb am 11.02.2022 um
11:39 in Nachricht :
> Dear Adam,
> 
> 
> First, just a note (for myself), `ipmitool power` is a shortcut for 
> `ipmitool chassis power`.
> 
> 
> Am 11.02.22 um 10:54 schrieb Adam Nielsen:
 After flashing the firmware of some pcie card I need to power cycle
 the server to finish the flashing process.
>>>
>>> Can you please tell us the PCIe device, where a normal system reboot is
>>> not enough to reset it, so I can avoid buying it?
>> 
>> Most remote access cards work like this by design.  You might be
>> connected to a remote computer via a web browser, and need to get into
>> the BIOS to change some settings.  If you reboot the machine, the last
>> thing you want is the card to reset as well, causing your remote
>> session to drop out.  By the time the card's firmware has started and
>> you reconnect, you've probably missed the prompt to press F2/DEL/etc.
> 
> Then these remote management cards should allow to be restarted 
> separately. The BMC I had to deal with allow that.
> 
>> A lot of server hardware (e.g. SAS controllers) have onboard firmware
>> which may or may not restart during a reboot.  This is also by design,
>> because the cards can take many minutes for the firmware to boot up, so
>> by continuing to run the firmware across a restart they can cut the
>> server boot time down by many minutes, which makes a big difference if
>> you're running an OS that requires frequent reboots.
> 
> Why should the firmware need more than one second? There is no reason 
> for that. So, one more point to avoid such a device.

Well, I think (for example) that Dell's iDRAC are complete Linux systems (or 
similar) with a web server and other services. So it takes minutes instead of 
seconds until everything is loaded.

> 
>> I think most hardware vendors (at least in the server space) view a
>> reboot as a means to apply operating system updates rather than a means
>> to actually reset hardware.  This would no doubt be why programs like
>> ipmitool have an option to automatically power cycle the machine, for
>> the rare cases where you do want the hardware reset as well.
> 
> Hmm, that would be very strange. Luckily until now, a normal reboot was 
> totally fine in my experience with Dell and Supermicro servers.

The thing with LOM (Lights Out management) is that the LOM is available when 
the server is powered down, so it makes no sense the roboot the LOM with the 
server.
But usually LOM can restart itself witthout affecting the OS (Dell's IPMI 
watchdog excepted, maybe).

> 
>> It's only really a problem when you try to use this server hardware in
>> consumer machines, which don't really have a means to automatically
>> power cycle themselves.  Most servers seem to be configurable, to do
>> things like intercept a hardware reset and turn it into a power cycle
>> automatically, or have an "always on" mode configured where you just
>> tell the OS to shut down and power off, and the server will
>> automatically power itself back on a few seconds later, or at the very
>> least the server will have one of those remote access controllers that
>> you can use to remotely power it back on again after you've done a
>> normal shut down + power off.
>> 
>> So if you don't like the idea of parts of the system continuing to run
>> across reboots, just stay clear of hardware designed for enterprise
>> servers and you'll likely never encounter it.
> 
> Yes, that is why I asked, so I will never buy such crappy hardware. With 
> Dell and Supermicro servers, rebooting the system was all I needed. (The 
> BMC is not reset, but can be done separately while the server itself is 
> still running.)
> 
> 
> Kind regards,
> 
> Paul

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Run "ipmitool power cycle" after lib/systemd/system‑shutdown scripts

[Ulrich Windl]

>>> Adam Nielsen  schrieb am 11.02.2022 um 12:27 in
Nachricht <20220211212751.14db2...@vorticon.teln.shikadi.net>:
>>  Then these remote management cards should allow to be restarted 
>> separately. The BMC I had to deal with allow that.
> 
> They do under normal circumstances, but for example I had a Dell R720
> recently where the remote access controller (iDRAC) partially crashed.
> It was still running but would no longer connect to the network.  In
> this case not even power cycling the server would fix it (because it
> stays on while the server is off, so you can use it to remotely power
> the server back on again).  The only solution was to unplug the power
> to both the server's power supplies, effectively power cycling the DRAC.
> 
> It is true I probably could've used the RACADM utility to reset it via
> some other method, but I was unable to run it because of various
> library incompatibilities and missing kernel modules, as I'm not
> running one of the tiny number of supported Linux distributions.
> 
>> Why should the firmware need more than one second? There is no reason 
>> for that. So, one more point to avoid such a device.
> 
> I agree the firmware should be fast.  But a lot of these devices are so
> complex they now boot whole embedded operating systems.  Things like
> SAS cards, even normal consumer SATA drives, run their own embedded
> operating systems, and if you can find the debug UARTs on the PCBs you
> can even read the boot messages.  Even LTO tape drives now have an
> Ethernet connection on the drives themselves and provide a TCP stack
> and DHCP server so you can connect a laptop directly to the drive to
> perform diagnostics.
> 
> All that complexity leads to longer boot times as the firmware has to
> do more and more during startup.  I'm not saying it's a good thing, and
> as I sit there waiting for 10 minutes while a Dell server does whatever
> it has to do before it even attempts to load the OS, I often wonder
> whether all that complexity is really necessary.
> 
>> Hmm, that would be very strange. Luckily until now, a normal reboot
>> was totally fine in my experience with Dell and Supermicro servers.
> 
> I recently reflashed a Dell H710 onboard SAS controller, to disable the
> RAID firmware and convert it into IT mode, where it provides direct
> access to the connected drives.  This allows Linux to access them
> directly, without any proprietary RAID algorithms interfering with what
> actually ends up on the disk.
> 
> In order to do this, not only did I have to power cycle the server, but
> I also had to remove the battery from the SAS card in order to ensure
> the last remnants of the original firmware were wiped from the
> controller's onboard memory, otherwise it can persist even with the
> server power supplies unplugged.  So alas even Dell are not immune from
> this.

I think the real reason is that some RAID controllers "back up" the disk 
configuration in NVRAM, so if you want to reconfigure the disks completely, 
you'd probably have to erase the logical disks first, then upgrade the 
firmware, and then recreate the configuration.

> 
>> Yes, that is why I asked, so I will never buy such crappy hardware.
>> With Dell and Supermicro servers, rebooting the system was all I
>> needed. (The BMC is not reset, but can be done separately while the
>> server itself is still running.)
> 
> You are probably just lucky.  A good firmware update will reset the
> device without even requiring a reboot.  I can't speak for Supermicro
> but plenty of Dell hardware certainly doesn't reset completely when you
> reboot the machine, and sometimes not even when you power cycle it!

Well, if the hardware had NVRAM...

> Most of the Dell BIOS updates I've performed (outside of Linux, by
> booting from a USB stick) have automatically power cycled the server
> when they are done, so even they don't just do a hardware reset.

Why don't you upgrade firmware using the built-in IDRAC?

> 
> But, if it's done correctly so a power cycle isn't necessary, does it
> even matter?  I would argue that if the Dell hardware you've used
> hasn't done "clean" hardware resets but has hidden that fact so well
> you haven't noticed, then it doesn't really seem to be a problem.
> 
> Cheers,
> Adam.

[systemd-devel] Antw: [EXT] Service activation

[Ulrich Windl]

>>> Wol  schrieb am 13.02.2022 um 01:03 in Nachricht
<129b4e49-d22d-7efb-7fc6-a44e5faa2...@youngman.org.uk>:
> More fun getting things to work ... :-)
> 
> So I've got a service, scarletdme.service, which fires up my db backend 
> for running interactively. However, I also need a socket service for 
> remote connections.
> 
> I've got the xinetd files, but if I'm running systemd, I want to use 
> systemd :-)
> 
> So I've written scarletdme.socket, and scarletdme@.service, but the more 
> I read, the more I don't understand ...
> 
> Do I enable scarletdme.socket the same as anything else eg "systemctl 
> enable scarletdme.socket"? How does it know the difference between 
> scarletdme.service and scarletdme@.service? I get the impression I need 
> to put something in the .socket file to make it use scarletdme@ rather 
> than scarletdme?
> 
> 
> And once I've got all that sorted, I'm betting I'm going to have grief 
> getting it to work properly, so while it's not much to do with systemd, 
> is there any way I can get systemd to log all traffic back and forth so 
> I can debug it?

Maybe understandable documentation is where xinetd is still better tahn systemd 
;-)

> 
> Cheers,
> Wol

Re: [systemd-devel] Antw: [EXT] Service activation

[Ulrich Windl]

>>> Wols Lists  schrieb am 14.02.2022 um 09:42 in
Nachricht <3320ed4b-2a0c-d7e6-a8a1-b7c83b5f4...@youngman.org.uk>:
> On 14/02/2022 08:27, Ulrich Windl wrote:
>>> And once I've got all that sorted, I'm betting I'm going to have grief
>>> getting it to work properly, so while it's not much to do with systemd,
>>> is there any way I can get systemd to log all traffic back and forth so
>>> I can debug it?
> 
>> Maybe understandable documentation is where xinetd is still better tahn 
> systemd;-)
>> 
> I doubt it. Documentation is excellent at reminding you what you already 
> knew. It's piss poor at getting a newbie started.


I myself also was very much confused about socket activation (wearing the admin 
hat, not the developer's).
For example in SLES there exist:
virtlockd.service
virtlogd.socket
virtlockd.socket
virtlockd-admin.socket
libvirtd.service

libvirtd may need virtlockd (maybe both also need virtlogd, I don't know), so 
how to properly restart virtlockd, and when which servcies exactly do I have to 
enable?
Some combinations don't work as exected (or "as planned").

> 
> I've got an industrial project that was GPL'd, there's nobody really 
> with a proper grasp of how it's written, and I'm trying to help get it 
> into a state where we can make an official release.
> 
> I know bugger all about systemd, or xinetd, so I'm in that twisty maze 
> of passages, are they all alike or different?
> 
> :-)

Of course they are different; be sure to carry your lamp with you ;-)

> 
> Cheers,
> Wol

[systemd-devel] Antw: Re: Antw: [EXT] Service activation

[Ulrich Windl]

>>> Michael Biebl  schrieb am 14.02.2022 um 09:58 in Nachricht
:
> Am Mo., 14. Feb. 2022 um 09:42 Uhr schrieb Wols Lists
> :
>> I doubt it. Documentation is excellent at reminding you what you already
>> knew. It's piss poor at getting a newbie started.
> 
> The existing documentation of systemd is extensive and well written
> but indeed probably not very well suited to get you started learning
> the various concepts of systemd.

Having a set of documents, "where everything is described somewhere" actually 
isn't very helpful to a beginner.
Some conceptional overview with "drill-down links" is needed.

> Being scattered across dozens of man pages and blog posts doesnt't
> help with that either.

Correct!

> 
> What I really like about upstart back in the days was the
> documentation they provided in the form of a cookbook:
> https://upstart.ubuntu.com/cookbook/ 
> 
> I really miss something like that for systemd.

I agree, too.

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: Passive vs Active targets

[Ulrich Windl]

>>> Kenneth Porter  schrieb am 15.02.2022 um 18:14 in
Nachricht :
> Given that interfaces can come and go, does network.target imply that 
> all possible interfaces are up? Or does it just mean that the 
> infrastructure is there for DHCP-capable interfaces to begin searching 
> for an address?

I guess you would not want network.target to be stopped when a single interface 
goes down ;-)

[systemd-devel] Antw: [EXT] Proposal to extend os-release/machine-info with field PREFER_HARDENED_CONFIG

[Ulrich Windl]

>>> Stefan Schröder  schrieb am 15.02.2022 um 19:05 in
Nachricht <531595111.79330.1644948301...@webmail.strato.com>:
> Situation: 
> 
> Many packages in a distribution ship with a default configuration that is 
> not considered 'secure'.

Probably because "secure" isn't considered to be "comfortable" by a majority
of users.

> 
> Hardening guidelines are available for all major distributions. Each is a 
> little different.
> Many configuration suggestions are common-sense among security-conscious 
> administrators, who have to apply more secure configuration using some 
> automation framework after installation.
> 
> PROPOSAL
> 
> os-release or machine-info should be amended with a field
> 
>   PREFER_HARDENED_CONFIG
> 
> If the value is '1' or 'True' or 'yes' a package manager can opt to 
> configure an alternative, more secure default configuration (if avaialble).

I think os-relesase describes the operating system, not policies.

> 
> E.g. 
> 
> According to the 'Securing Debian Manual' [1] the login configuration is 
> configured as
> auth   optional   pam_faildelay.so  delay=300
> whereas 
> delay=1000
> would provide a more secure default. 

Wouldn't /etc/default/* be the place to look such things up?

> 
> The package 'login' contains the file /etc/pam.d/login. If dpkg (or apt or 
> rpm or pacman or whatever) detected that os-release asks for secure
defaults, 
> the alternative /etc/pam.d./login.harden could be made the default. (This 
> file doesn't exist yet, the details are left to the packaging infrastructure

> or package maintainer.)
> 
> The existence of PREFER_HARDENED_CONFIG=1 would allow any distribution to 
> select a more suitable default for use-cases where security is considered 
> more important than convenience.

But IMHO the OS is still the same. (Well, Microsoft sells the same OS under
different names and for different prices, but actually who likes to buy a
license for each network connection?)

> 
> During initial installation of a distribution, the installation tool could 
> ask how to set this value.
> Subsequent package installation could then benefit from the more secure 
> defaults.
> 
> PRO:
> 
> - allows more secure defaults by default
> - hardening solved upstream making plethora of distribution specific 
> hardening guides redundant
> - contribution to defense-in-depth
> - does not impose any particular security policy
> - allows smooth transition towards more secure settings
> 
> CON:
> - yet another entry for os-release
> 
> Alternatives considered:
> - We could just keep on performing just as badly as before.

You are saying: If we don't get THIS, we get NOTHING instead?
I would disagree.

Regards,
Ulrich

> 
> [1] 
>
https://www.debian.org/doc/manuals/securing-debian-manual/ch04s11.en.html#id-

> 1.5.14.11] 
> 
> Best
> Stefan Schroeder

[systemd-devel] Antw: [EXT] [systemd‑devel] How to sychronize service START with STOP of mount units?

[Ulrich Windl]

>>> "Manuel Wagesreither"  schrieb am 16.02.2022 um 13:28
in
Nachricht :
> Hi all,
> 
> We'd like to implement a factory reset for an embedded device we are working

> on.
> 
> I think the cleanest approach would be a factory‑reset.target which 
> WANTS/REQUIRES reboot.target + a factory‑reset.service which wipes the 
> persistent storage.
> 
> The thing is, START of factory‑reset.service would need to be synchronized 
> with the STOPs of some mount units which got started during system boot.
> 
> For easier understanding:
> 
> System start:
> 1. Start path‑to‑first.mount
> 2. Start path‑to‑second.mount
> 
> System halt:
> 1. Stop path‑to‑second.mount
> 2. Stop path‑to‑first.mount
> 
> Factory reset:
> 1. Stop path‑to‑second.mount
> 2. Start factory‑reset.service, wiping /path/to/first/*
> 3. Stop path‑to‑first.mount
> 
> I currently have no idea how to express this in systemd terms. If someone on

> this list could provide some advice, it would be much appreciated.

As for the recent "firmware flashing" thread I think it would be best to
handle that via some boot parameter and something like a "boot once" boot entry
that adds that specific parameter.
I also think the clearing should be best done in initrd before the real root
filesystem is mounted. That should be most flexible.

> 
> Thanks in advance and best regards,
> Manuel

[systemd-devel] Q: Perform action for reboots happen too frequently?

[Ulrich Windl]

Hi!

I wonder: Is it possible with systemd to detect multiple reboots within a 
specific time interval, and react, like executing some systemctl command (that 
is expected to "improve" things)?
With "reboots" I'm mainly thinking of unexpected boots, not so much the 
"shutdown -r" commands, but things like external resets, kernel panics, 
watchdog timeouts, etc.

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: Proposal to extend os-release/machine-info with field PREFER_HARDENED_CONFIG

[Ulrich Windl]

>>> Stefan Schröder  schrieb am 16.02.2022 um 18:11 in
Nachricht <1487454823.169440.1645031460...@webmail.strato.com>:

...
> There are reasons why the (...) decide not to ship 
> with 'hardened' defaults.
...

As said before, in most cases "comfortable" and "secure" contradict.
For example: It's hard to tell a former Windows user that he/she/it cannot
read syslog as normal user, so many "easy" configurations allow that.
OTOH, when using SSH-login and you enter your password where you should have
entered your user name, your password will be logged in syslog.
For a multi-user system you don't want other users see your password, then
(well if aware the user would change the password after that anyhow).

This is just one example:
Other examples:
Require root (or other privileged user) to:
* configure a network (use WLAN)
* shutdown the system
* mount a CD/DVD/USB disk
* use graphics acceleration
* use the clipboard (yes, seriously)
* use the webcam
* use a specific printer
...

Some people really don't want that type of "security". Most people don't even
want to authenticate, but just turn off the computer (or wake it up).

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Q: Perform action for reboots happen too frequently?

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 16.02.2022 um 18:17
in
Nachricht :
> On Mi, 16.02.22 14:09, Ulrich Windl (ulrich.wi...@rz.uni‑regensburg.de)
wrote:
> 
>> Hi!
>>
>> I wonder: Is it possible with systemd to detect multiple reboots
>> within a specific time interval, and react, like executing some
>> systemctl command (that is expected to "improve" things)?  With
>> "reboots" I'm mainly thinking of unexpected boots, not so much the
>> "shutdown ‑r" commands, but things like external resets, kernel
>> panics, watchdog timeouts, etc.
> 
> The information why a system was reset is hard to get. Some watchdog
> hw will tell you if it was recently triggered, and some expensive hw
> might log serious errors to the acpi pstore stuff, but it's all icky
> and lacks integration.

Lennart,

you got me wrong: I don't want to filter on the reasons of the (re)boot, but I
just wanted to rule out any possible solution that stores information during an
orderly shutdown/reboot ;-)

> 
> A safe approach would probably to instead track boots where the root
> fs or so is in dirty state. Pretty much all of today's file systems
> track that.

In the past (when there are a few logins only) I would have been tempted to
use "last" to grep for boots, but if there are many (automated) logins, those
boots may be "rolled out" and thus won't be counted possibly.

> 
> It sounds like an OK feature to add to the systemd root fs mount logic
> to check for the dirty flag before doing that and then using that is
> hint to boot into a target other than default.target that could then
> apply further policy (and maybe then go on to enter default.target).
> 
> TLDR: nope, this does not exist yet, but parts of it sound like
> worthwile feature additions to systemd.

Something very primitive:
a) Increment some counter on every boot, and reset that counter after some
time.
b) Check the counter and if it is >= some threshold, execute a one-shot
command

Could systemd help with a) and b)?

Regards,
Ulrich





> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] [RFC] systemd‑resolved: Send d‑bus signal after DNS resolution

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 16.02.2022 um 18:59
in
Nachricht :
> On Mi, 16.02.22 12:13, Dave Howorth (syst...@howorth.org.uk) wrote:
> 
>> > This could be used by applications for auditing/logging services
>> > downstream of the resolver, or to update the firewall on the system.
>>
>> Perhaps an example use case would help but I'm not clear how a DNS
>> resolution would usefully cause a state change in the firewall without
>> some further external guidance?
> 
> Yeah, I am not sure I grok the relationship to firewalls here,
> either. Updatign firewalls asynchronously based on DNS lookups sounds
> wrong to me...

Unless you want to implement nice backdoors ;-)

> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: Re: [systemd‑devel] [EXT] Proposal to extend os‑release/machine‑info with field PREFER_HARDENED_CONFIG

[Ulrich Windl]

>>> Peter Hoeg  schrieb am 17.02.2022 um 07:07 in Nachricht
<87k0duvvtv@hoeg.com>:

>>> I think os‑relesase describes the operating system, not policies.
>>
>> You are right. Perhaps machine‑info would be a better fit than os‑release.
> 
> To what extent a machine is locked down is a policy choice. There are 
> already loads of tools available to manage policy so this really doesn't 
> belong here and if you want to ensure that your fleet of machines are locked

> down through something like PREFER_HARDENED_CONFIG=1, you're going to need 
> tools to manage *that* anyway. Then why not use the same tool(s) to simply 
> manage the machines?

And what exactly should it do? Also: Do you really believe in "one size fits
all" security-wise?
If (at all), then the parameter should be "SECURITY_POLICY=name" (where name
is one of the predefined policies).
And most of all, selecting a different policy does not make it a different
OS.

Regards,
Ulrich Windl


> 
> It's 2022 ‑ nobody should be doing this by hand.

[systemd-devel] Antw: Re: Antw: [EXT] Re: [systemd‑devel] Q: Perform action for reboots happen too frequently?

[Ulrich Windl]

>>> Silvio Knizek  schrieb am 18.02.2022 um 09:37 in
Nachricht <7d547aa30624f2dc8d22ea7f84507fa1d8e9c678.ca...@gmx.net>:
> Am Donnerstag, dem 17.02.2022 um 10:50 +0100 schrieb Ulrich Windl:
>> > > > Lennart Poettering  schrieb am
>> > > > 16.02.2022 um 18:17
>> in
>> Nachricht :
>> > On Mi, 16.02.22 14:09, Ulrich Windl
>> > (ulrich.wi...@rz.uni‑regensburg.de)
>> wrote:
>> >
>> > > Hi!
>> > >
>> > > I wonder: Is it possible with systemd to detect multiple reboots
>> > > within a specific time interval, and react, like executing some
>> > > systemctl command (that is expected to "improve" things)?  With
>> > > "reboots" I'm mainly thinking of unexpected boots, not so much
>> > > the
>> > > "shutdown ‑r" commands, but things like external resets, kernel
>> > > panics, watchdog timeouts, etc.
>> >
>> > The information why a system was reset is hard to get. Some
>> > watchdog
>> > hw will tell you if it was recently triggered, and some expensive
>> > hw
>> > might log serious errors to the acpi pstore stuff, but it's all
>> > icky
>> > and lacks integration.
>>
>> Lennart,
>>
>> you got me wrong: I don't want to filter on the reasons of the
>> (re)boot, but I
>> just wanted to rule out any possible solution that stores information
>> during an
>> orderly shutdown/reboot ;-)
>>
>> >
>> > A safe approach would probably to instead track boots where the
>> > root
>> > fs or so is in dirty state. Pretty much all of today's file systems
>> > track that.
>>
>> In the past (when there are a few logins only) I would have been
>> tempted to
>> use "last" to grep for boots, but if there are many (automated)
>> logins, those
>> boots may be "rolled out" and thus won't be counted possibly.
>>
>> >
>> > It sounds like an OK feature to add to the systemd root fs mount
>> > logic
>> > to check for the dirty flag before doing that and then using that
>> > is
>> > hint to boot into a target other than default.target that could
>> > then
>> > apply further policy (and maybe then go on to enter
>> > default.target).
>> >
>> > TLDR: nope, this does not exist yet, but parts of it sound like
>> > worthwile feature additions to systemd.
>>
>> Something very primitive:
>> a) Increment some counter on every boot, and reset that counter after
>> some
>> time.
>> b) Check the counter and if it is >= some threshold, execute a one-
>> shot
>> command
>>
>> Could systemd help with a) and b)?
>>
>> Regards,
>> Ulrich
> 
> Hi Ulrich,
> 
> so, incrementing a counter and executing stuff should be easy.
> Something like
> 
> --- boot-counter.service
> [Unit]
> Description=Incrementing the boot counter
> RequiresMountsFor=/var/lib
> DefaultDependencies=no
> 
> [Service]
> Type=oneshot
> Environment="T=5"
> ExecStart=/usr/bin/bash -c 'declare -i c="$(< /var/lib/boot.counter)"; c+=1;

> echo "$c" > /var/lib/boot.counter'
> ExecStart=/usr/bin/bash -c 'if (($(< /var/lib/boot.counter) > T)); then 
> /clean-script.sh; fi'
> 
> [Install]
> WantedBy=multi-user.target
> ---
> 
> For the reset you can employ a .timer and a .service
> 
> --- reset-boot-counter.timer
> [Unit]
> Description=Reset boot counter after 30 minutes
> 
> [Timer]
> OnBoot=30m
> AccuracySec=1us
> 
> [Install]
> WantedBy=timer.target
> ---
> 
> --- reset-boot-counter.service
> [Unit]
> Description=Set /var/lib/boot.counter to zero
> RequiresMountsFor=/var/lib
> 
> [Service]
> Type=oneshot
> ExecStart=/usr/bin/bash -c 'echo "0" > /var/lib/boot.counter
> ---

Thanks Silvio!

That looks like I was after. But admitted: I had expected that systemd could
do some of the "shell scripting magic" (like counting and comparing).

Regards,
Ulrich


> 
> HTH
> Silvio

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Q: Perform action for reboots happen too frequently?

[Ulrich Windl]

>>> Silvio Knizek  schrieb am 18.02.2022 um 23:24 in
Nachricht :
> Am Freitag, dem 18.02.2022 um 09:52 +0100 schrieb Ulrich Windl:
>> > >
>> Thanks Silvio!
>>
>> That looks like I was after. But admitted: I had expected that
>> systemd could
>> do some of the "shell scripting magic" (like counting and comparing).
>>
>> Regards,
>> Ulrich
> Well, counting and comparing goes into the area of turing completness,
> doesn't it? So not having this inside systemd makes the whole system
> just easier.

Systemd can mount a filesystem, handle complex dependencies, but it cannot 
count and compare ;-)

[systemd-devel] Q: Wanting a target

[Ulrich Windl]

Hi!

I think there was a recent discussion saying that no unit should Want= or 
Require= a target, but only use them for ordering.
I have basically three questions:

1) Is the statement above correct?

2) When is a target displayed as "started" by sysctemctl status then?

3: Which of the following (form current SLES12 SP5, grep-style) are wrong?:

/usr/lib/systemd/system/ntpd.service:Wants=network.target
/usr/lib/systemd/system/ntpd.service:After=network.target

/usr/lib/systemd/system/nfs-server.service:Requires= network.target 
proc-fs-nfsd.mount
/usr/lib/systemd/system/nfs-server.service:After= network.target 
proc-fs-nfsd.mount rpcbind.socket nfs-mountd.service

/usr/lib/systemd/system/nmb.service:After=network.target

Regards,
Ulrich

[systemd-devel] Restarting "onshot" services

[Ulrich Windl]

Hi!

It seems "oneshot" services are not retried when they fail.
I read systemd.service, but the manual does not explain that "oneshot" in no 
candidate for RestartSec or Restart.
Is that intentional?

(systemd 228 of SLES12 SP5)

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: Journald: Re-use /var/log/journal hash entry after system upgrade

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 01.03.2022 um 16:52 in
Nachricht
:
> On Tue, Mar 1, 2022 at 4:39 PM eric.zalu...@vertiv.com <
> eric.zalu...@vertiv.com> wrote:
> 
>>  [ System Environment ] 
>> On an embedded x86-64 Linux system, I’m running systemd v241. I have
>> Systemd-Journald logging set to persistent with SystemMaxUse and
>> RuntimeMaxUse both set to 512MB.
>>
>> The Linux system mount loops /var/log on start-up from a var-log.ext4
>> file. The /var/log mount is given a fixed size of the disk (976MB).
Systemd
>> creates a journal entry directory given a hash name in
>> /var/log/journal/.
>>
>>
>>
>>  [ System Information ] 
>>
>> Linux 4.19.0-6-2-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64
>> GNU/Linux
>>
>>
>>  [ Problem ] 
>>
>> When the system firmware is upgraded, Systemd creates a new journal entry
>> directory with a different hash name and no longer recognizes the previous
>> hash directory.
>>
>> The old logs from the previous journal entry can no longer be managed. The
>> old logs are never rotated and cannot be manually rotated using the
>> journaldctl cli. The disk usage calculator by Journald does not account
for
>> the previous journal entry meaning if there are two previous entries in
>> /var/log that consume 900MB of space; then the new journal entry only has
>> 76MB of space to work with. Eventually, disk space will be full. Journald
>> cannot automatically flush, vacuum, clean, rotate previous logs because it
>> does not recognize the previous journal entries.
>>
>> There must be a systemd journald check that occurs where it determines
>> these other entries are not for it to manage.
>>
> 
> This is not a 'hash' – it's the machine ID
> , which
> is directly read from /etc/machine-id. Normally the machine ID is randomly
> generated during *first *boot (it's just a random UUID), and it is supposed
> to be persistent afterwards.
> 
> It sounds like your /etc doesn't have that file and is on a read-only
> rootfs, so systemd generates a new machine-id in tmpfs every boot. The

Dumb question: Is that fact logged in the journal?

> device probably has *some* persistent storage though, so try to find a way
> to make /etc/machine-id persist as well (see the machine-id(5)
>  manual
> page for a few possibilities).
> 
> The intent of using machine-id subdirectories is to allow containers'
> journal directories to be symlinked on the host, or remote system journals
> to be collected on a single system – there's actually the journalctl -m
> option to make it look at "foreign" journals, so e.g. journalctl -m -f
> could be used to watch logs of all containers.

Still I think some funcion like "remove-orphans" may seem useful (just to
prevent the user to make things worse by removing the wrong files)

> 
> (The machine ID is also used as the base for systemd-networkd's DHCPv4
> Client ID, DHCPv6 DUID, and IPv6 address generation. But other than that,
> though, I can't think of any its uses that would be visible externally –
> aside from desktop-specific things like pulseaudio – so in some cases it
> might be "good enough" to pre-define a fixed ID in the template image as a
> last resort.)


Regards,
Ulrich

> 
> -- 
> Mantas Mikulėnas

[systemd-devel] Q: journalctl -g

[Ulrich Windl]

Hi!

In SLES15 SP3 (systemd-246.16-7.33.1.x86_64) I have this effect, wondering 
whether it is a bug or a feature:
When using "journalctl -b -g raid" I see that _ome_ matches are highlighted in 
red, but others aren't. For example:
Mar 01 01:37:09 h16 kernel: mega*raid*_sas :c1:00.0: BAR:0x1  BAR's 
base_addr(phys):0xa550  mapped virt_addr:0xae628322
Mar 01 01:37:09 h16 kernel: megaraid_sas :c1:00.0: FW now in Ready state
...

That means in the line following the raid is not highlighted, even though it 
matched obviously.
Likewise any further "megaraid_sas" aren't highlighted.

But also those are not highlighted:
Mar 01 01:37:20 h16 kernel: raid6: avx2x4   gen() 16182 MB/s

Mar 01 01:37:47 h16 kernel: md/raid1:md127: active with 2 out of 2 mirrors

Mar 01 01:37:48 h16 smartd[5871]: Device: /dev/bus/0 [megaraid_disk_00], type 
changed from 'megaraid,0' to 'sat+megaraid,0'

But here it is highlighted again:
Mar 01 01:38:55 h16 pacemaker-controld[7236]:  notice: Requesting local 
execution of probe operation for prm_lockspace_*raid*_md10 on h16
Mar 01 01:38:55 h16 pacemaker-controld[7236]:  notice: Result of probe 
operation for prm_lockspace_*raid*_md10 on h16: not running

And this, too:
Mar 01 07:58:44 h16 kernel: mega*raid*_sas :c1:00.0: Firmware crash dump is 
not available
Mar 01 08:00:47 h16 supportconfig[671]: Software *Raid*...

And this not:
Mar 02 03:07:48 h16 smartd[5871]: Device: /dev/bus/0 [megaraid_disk_00] [SAT], 
starting scheduled Short Self-Test.

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] How to find out the processes systemd‑shutdown is waiting for?

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 02.03.2022 um 17:22
in
Nachricht :
> On Mi, 02.03.22 13:02, Arian van Putten (arian.vanput...@gmail.com) wrote:
> 
>> I've seen this a lot with docker/containerd. It seems as if for some
reason
>> systemd doesn't wait for their  cgroups to cleaned up on shutdown. It's
>> very easy to reproduce. Start a docker container and then power off the
>> machine. Since the move to cgroups V2 containerd should be using systemd
to
>> manage the cgroup tree so a bit puzzling why it's always happening.
>>
>> Something seems to be off with containerd's integration into systemd but
>> I'm not sure what.
> 
> Docker traditionally has not followed any of our documented ways to

You are implying that "our documented ways" is a definitive standard?

> interact with cgroups, even though they were made aware of them, not
> sure why, I think some systemd hate plays a role there. I am not sure
> if this has changed, but please contact Docker if you have issues with
> Docker, they have to fix their stuff themselves, we cannot work around
> it.

The problem with systemnd (people) is that they try to establish new standards
outside of systemd.

"If A does not work with systemd", it's always A that is broken, never systemd
;-)

Regards,
Ulrich

> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] [systemd‑devel] Q: journalctl ‑g

[Ulrich Windl]

>>> "Ulrich Windl"  schrieb am 04.03.2022
um
08:07 in Nachricht <6221bac102a100048...@gwsmtp.uni-regensburg.de>:
> Hi!
> 
> In SLES15 SP3 (systemd‑246.16‑7.33.1.x86_64) I have this effect, wondering 
> whether it is a bug or a feature:
> When using "journalctl ‑b ‑g raid" I see that _ome_ matches are highlighted
in 

s/_ome_/_some_/; # Sorry

> red, but others aren't. For example:
> Mar 01 01:37:09 h16 kernel: mega*raid*_sas :c1:00.0: BAR:0x1  BAR's 
> base_addr(phys):0xa550  mapped virt_addr:0xae628322
> Mar 01 01:37:09 h16 kernel: megaraid_sas :c1:00.0: FW now in Ready
state
> ...
> 
> That means in the line following the raid is not highlighted, even though it

> matched obviously.
> Likewise any further "megaraid_sas" aren't highlighted.
> 
> But also those are not highlighted:
> Mar 01 01:37:20 h16 kernel: raid6: avx2x4   gen() 16182 MB/s
> 
> Mar 01 01:37:47 h16 kernel: md/raid1:md127: active with 2 out of 2 mirrors
> 
> Mar 01 01:37:48 h16 smartd[5871]: Device: /dev/bus/0 [megaraid_disk_00], 
> type changed from 'megaraid,0' to 'sat+megaraid,0'
> 
> But here it is highlighted again:
> Mar 01 01:38:55 h16 pacemaker‑controld[7236]:  notice: Requesting local 
> execution of probe operation for prm_lockspace_*raid*_md10 on h16
> Mar 01 01:38:55 h16 pacemaker‑controld[7236]:  notice: Result of probe 
> operation for prm_lockspace_*raid*_md10 on h16: not running
> 
> And this, too:
> Mar 01 07:58:44 h16 kernel: mega*raid*_sas :c1:00.0: Firmware crash dump

> is not available
> Mar 01 08:00:47 h16 supportconfig[671]: Software *Raid*...
> 
> And this not:
> Mar 02 03:07:48 h16 smartd[5871]: Device: /dev/bus/0 [megaraid_disk_00] 
> [SAT], starting scheduled Short Self‑Test.
> 
> Regards,
> Ulrich

[systemd-devel] Incomplete call stack logged by systemd-coredump

[Ulrich Windl]

Hi!

In SLES15 SP3 I have found some interesting fact about systemd-coredump:
A process dumped core for strange reasons (investigated by support), and the 
journal had logged:
Mar 04 07:20:02 h19 systemd-coredump[27340]: Process 27338 (sh) of user 0 
dumped core.

  Stack trace of thread 27338:
  #0  0x7fdd7418a938 
locked_vfxprintf (libc.so.6 + 0x84938)
  #1  0x7fdd7418ab55 
__vfxprintf (libc.so.6 + 0x84b55)
  #2  0x7fdd7418acb6 
__fxprintf (libc.so.6 + 0x84cb6)
  #3  0x7fdd7419eb40 
__malloc_assert (libc.so.6 + 0x98b40)
  #4  0x7fdd741a1583 
sysmalloc (libc.so.6 + 0x9b583)
...I'm leaving out the boring lines in between
  #61 0x7fdd7418ab55 
__vfxprintf (libc.so.6 + 0x84b55)
  #62 0x7fdd7418acb6 
__fxprintf (libc.so.6 + 0x84cb6)
  #63 0x7fdd7419eb40 
__malloc_assert (libc.so.6 + 0x98b40)
Mar 04 07:20:02 h19 systemd[1]: systemd-coredump@1-27339-0.service: Succeeded.
Mar 04 07:20:03 h19 kernel: BUG: Bad rss-counter state mm:ba7a09bc 
idx:1 val:16

However when examining the stack trace with gdb, the stack is horribly deep:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7fdd7418a938 in locked_vfxprintf () from /lib64/libc.so.6
Missing separate debuginfos, use: zypper install 
bash-debuginfo-4.4-19.6.1.x86_64
(gdb) bt
#0  0x7fdd7418a938 in locked_vfxprintf () from /lib64/libc.so.6
#1  0x7fdd7418ab55 in __vfxprintf () from /lib64/libc.so.6
#2  0x7fdd7418acb6 in __fxprintf () from /lib64/libc.so.6
#3  0x7fdd7419eb40 in __malloc_assert () from /lib64/libc.so.6
#4  0x7fdd741a1583 in sysmalloc () from /lib64/libc.so.6
#5  0x7fdd741a2380 in _int_malloc () from /lib64/libc.so.6
...
#39520 0x7fdd741a1583 in sysmalloc () from /lib64/libc.so.6
#39521 0x7fdd741a2380 in _int_malloc () from /lib64/libc.so.6
#39522 0x7fdd741a34f4 in malloc () from /lib64/libc.so.6
#39523 0x55f1048cf52e in xmalloc ()
#39524 0x55f1048d9521 in strvec_from_word_list ()
#39525 0x55f1048acb31 in ?? ()
#39526 0x55f1048aa56d in execute_command_internal ()
#39527 0x55f1048d0836 in parse_and_execute ()
#39528 0x55f1048c69ef in command_substitute ()
#39529 0x55f1048c212f in ?? ()
#39530 0x55f1048bd620 in ?? ()
#39531 0x55f1048bf2ab in ?? ()
#39532 0x55f1048ab955 in ?? ()
#39533 0x55f1048aa56d in execute_command_internal ()
#39534 0x55f1048d0836 in parse_and_execute ()
#39535 0x55f1048da695 in ?? ()
#39536 0x55f10489f786 in main ()

So my proposal would be to indicate that the stack dump is incomplete, maybe 
also leaving out the middle part as I did in gdb.

Regards,
Ulrich

[systemd-devel] timer "OnBootSec=15m" not triggering

[Ulrich Windl]

Hi!

I wrote some services that should run when booting and some time after
booting.
As it seems the service to run during boot works, but the timer-triggered one
does not.
I have no idea why.

Here are the details:
# systemctl status prevent-fencing-loop.service
● prevent-fencing-loop.service - Prevent Pacemaker Fencing-Loop
 Loaded: loaded (/usr/lib/systemd/system/prevent-fencing-loop.service;
enabled; vendor preset: disabled)
 Active: inactive (dead) since Sat 2022-03-05 10:33:50 CET; 1 day 23h ago
   Docs: man:boot-loop-handler(8)
   Main PID: 5226 (code=exited, status=0/SUCCESS)

Mar 05 10:33:50 h19 systemd[1]: Starting Prevent Pacemaker Fencing-Loop...
Mar 05 10:33:50 h19 boot-loop-handler[5234]: 1 of 4 allowable boots
Mar 05 10:33:50 h19 systemd[1]: prevent-fencing-loop.service: Succeeded.
Mar 05 10:33:50 h19 systemd[1]: Finished Prevent Pacemaker Fencing-Loop.

So this service ran during boot.

# systemctl status reset-boot-counter.timer
● reset-boot-counter.timer - Reset Boot-Counter after 15 minutes
 Loaded: loaded (/usr/lib/systemd/system/reset-boot-counter.timer;
enabled; vendor preset: disabled)
 Active: inactive (dead)
Trigger: n/a
   Triggers: ● reset-boot-counter.service
   Docs: man:boot-loop-handler(8)

So my guess is the timer never triggered, and the service was never started:
# systemctl status reset-boot-counter.service
● reset-boot-counter.service - Reset Boot-Counter for detecting Fencing-Loop
 Loaded: loaded (/usr/lib/systemd/system/reset-boot-counter.service;
static)
 Active: inactive (dead)
   Docs: man:boot-loop-handler(8)

The timer is:
# systemctl cat reset-boot-counter.timer
# /usr/lib/systemd/system/reset-boot-counter.timer
[Unit]
Description=Reset Boot-Counter after 15 minutes
Documentation=man:boot-loop-handler(8)


[Timer]
OnBootSec=15m
AccuracySec=1s

[Install]
WantedBy=timer.target

"systemd-analyze verify" finds no problems.

systemd being used is that of SLES15 SP3 (probably also openSUSE LEap 15.3)
(systemd-246.16-150300.7.39.1.x86_64)

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: timer "OnBootSec=15m" not triggering

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 07.03.2022 um 10:39 in
Nachricht
:
> On Mon, Mar 7, 2022 at 11:22 AM Ulrich Windl <
> ulrich.wi...@rz.uni-regensburg.de> wrote:
> 
>> Hi!
>>
>> I wrote some services that should run when booting and some time after
>> booting.
>> As it seems the service to run during boot works, but the timer-triggered
>> one
>> does not.
>> I have no idea why.
>>
>> Here are the details:
>> # systemctl status prevent-fencing-loop.service
>> ● prevent-fencing-loop.service - Prevent Pacemaker Fencing-Loop
>>  Loaded: loaded (/usr/lib/systemd/system/prevent-fencing-loop.service;
>> enabled; vendor preset: disabled)
>>  Active: inactive (dead) since Sat 2022-03-05 10:33:50 CET; 1 day 23h
>> ago
>>Docs: man:boot-loop-handler(8)
>>Main PID: 5226 (code=exited, status=0/SUCCESS)
>>
>> Mar 05 10:33:50 h19 systemd[1]: Starting Prevent Pacemaker Fencing-Loop...
>> Mar 05 10:33:50 h19 boot-loop-handler[5234]: 1 of 4 allowable boots
>> Mar 05 10:33:50 h19 systemd[1]: prevent-fencing-loop.service: Succeeded.
>> Mar 05 10:33:50 h19 systemd[1]: Finished Prevent Pacemaker Fencing-Loop.
>>
>> So this service ran during boot.
>>
>> # systemctl status reset-boot-counter.timer
>> ● reset-boot-counter.timer - Reset Boot-Counter after 15 minutes
>>  Loaded: loaded (/usr/lib/systemd/system/reset-boot-counter.timer;
>> enabled; vendor preset: disabled)
>>  Active: inactive (dead)
>> Trigger: n/a
>>
> 
> It's not a problem with any of the [Timer] On* configuration – the problem
> is that the whole .timer unit isn't active, so its triggers won't get
> scheduled in the first place.
> 
> 
>> [Install]
>> WantedBy=timer.target
>>
> 
> The timer is not being scheduled because it's WantedBy a nonexistent
> target. I think you meant 'timer*s*.target' here.

Hi!

Thanks for that. The amazing things are that "systemd.analyze verify" finds no
problem and "enable" virtually succeeds, too:
# systemctl enable reset-boot-counter.timer
Created symlink
/etc/systemd/system/timer.target.wants/reset-boot-counter.timer →
/usr/lib/systemd/system/reset-boot-counter.timer.

But you are correct that the target is "timers":
# ll /etc/systemd/system/timer*
/etc/systemd/system/timer.target.wants:
total 4
lrwxrwxrwx 1 root root 48 Mar  7 12:05 reset-boot-counter.timer ->
/usr/lib/systemd/system/reset-boot-counter.timer

/etc/systemd/system/timers.target.wants:
total 40
lrwxrwxrwx 1 root root 42 Nov  6  2020 backup-rpmdb.timer ->
/usr/lib/systemd/system/backup-rpmdb.timer
lrwxrwxrwx 1 root root 46 Nov  6  2020 backup-sysconfig.timer ->
/usr/lib/systemd/system/backup-sysconfig.timer
lrwxrwxrwx 1 root root 43 Nov  6  2020 btrfs-balance.timer ->
/usr/lib/systemd/system/btrfs-balance.timer
lrwxrwxrwx 1 root root 41 Nov  6  2020 btrfs-scrub.timer ->
/usr/lib/systemd/system/btrfs-scrub.timer
lrwxrwxrwx 1 root root 43 Nov  6  2020 check-battery.timer ->
/usr/lib/systemd/system/check-battery.timer
lrwxrwxrwx 1 root root 36 Nov  6  2020 fstrim.timer ->
/usr/lib/systemd/system/fstrim.timer
lrwxrwxrwx 1 root root 39 Nov  6  2020 logrotate.timer ->
/usr/lib/systemd/system/logrotate.timer
lrwxrwxrwx 1 root root 35 Nov  9  2020 mandb.timer ->
/usr/lib/systemd/system/mandb.timer
lrwxrwxrwx 1 root root 45 Nov  6  2020 snapper-cleanup.timer ->
/usr/lib/systemd/system/snapper-cleanup.timer
lrwxrwxrwx 1 root root 46 Nov  6  2020 snapper-timeline.timer ->
/usr/lib/systemd/system/snapper-timeline.timer

Basically I shouldn't have trusted Message-ID:
<7d547aa30624f2dc8d22ea7f84507fa1d8e9c678.ca...@gmx.net> ;-)

Regards,
Ulrich


> 
> -- 
> Mantas Mikulėnas

[systemd-devel] Antw: Re: [systemd‑devel] Antw: [EXT] Re: timer "OnBootSec=15m" not triggering

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 07.03.2022 um 14:08
in
Nachricht :
> On Mo, 07.03.22 12:24, Ulrich Windl (ulrich.wi...@rz.uni‑regensburg.de)
wrote:
> 
>> Thanks for that. The amazing things are that "systemd.analyze verify" finds

> no
>> problem and "enable" virtually succeeds, too:
> 
> Because there is no problem really: systemd allows you to define your
> targets as you like, and we generally focus on a model where you can
> extend stuff without requiring it to be installed. i.e. we want to
> allow lose coupling, where stuff can be ordered against other stuff,
> or be pulled in by other stuff without requiring that the other stuff
> to be hard installed. Thus you can declare that you want to be pulled
> in by a target that doesn't exist, and that's *not* considered an
> issue, because it might just mean that you haven't installed the
> package that defines it.
> 
> Example: if you install mysql and apache, then there's a good reason
> you want that mysql runs before apache, so that the web apps you run
> on apache can access mysql. Still it should be totally OK to install
> one without the other, and it's not a bug thus if one refers to the
> other in its unit files, even if the other thing is not installed.

Hi!

Well, I think it is more likely that the user wants to refer to an existing
target rather than refering to a future target.
Likewise verify complains about a missing manual page (Documentation) and
about missing executables.

Command ... is not executable: No such file or directory
...: man ... command failed with code 16

To be consistent, verify should also complain about missing targets referred
to.
(The manual page and executable could be added at some later time, too)

Regards,
Ulrich

> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] PrivateNetwork=yes is memory costly

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 09.03.2022 um 16:18
in
Nachricht :
> On Mo, 07.03.22 15:10, Christopher Wong (christopher.w...@axis.com) wrote:
> 
>> Hi,
>>
>>
>> It seems that PrivateNetwork=yes is a memory consuming
>> directive. The kernel seems to allocate quite an amount of memory
>> for each service (~50 kB) that has this directive enabled. I wonder
>> if this is expected and if anyone has had similar experience?

Despite of that I'm tempted to ask: "How many 50kB are there in a GB?" ;-)

> 
> PrivateNetwork=yes means that a private network namespace is allocated
> for the service. If you think network namespaces are too expensive in
> their current implementation, please bring this up with the kernel
> people, because they are a kernel concept after all, we just allocate
> them if told so.
> 
> network namespaces are an effective way to disconnect a service from
> the network, if the service doesn't need it. It's probably one of the
> most relevant sandboxing options we offer, since disabling the attack
> surface called "network" for a service is of such major
> importance. That said, if you disable the network namespace
> functionality in the kernel systemd will handle this gracefully, and
> not use it. If the feature is available in the kernel we will however
> use it.
> 
>> Is there any ways to reduce the usage?
> 
> Besides turning it off? Nothing I was aware of.
> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Q: Difference between AssertPathExists and ConditionPathExists?

[Ulrich Windl]

Hi!

When reading the manual page systemd.unit(5), I'm not quite sure what the 
difference between AssertPathExists and ConditionPathExists is:

(Condition)
If the specified absolute path name does not exist, the condition will fail.

Vs.

(Assert)
However, unlike the conditions settings, any assertion setting that is not met 
results
in failure of the start job it was triggered by.

What does that mean effectively?

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: Q: Difference between AssertPathExists and ConditionPathExists?

[Ulrich Windl]

>>> Andrei Borzenkov  schrieb am 17.03.2022 um 12:08 in
Nachricht
:
> On Thu, Mar 17, 2022 at 12:32 PM Ulrich Windl
>  wrote:
>>
>> Hi!
>>
>> When reading the manual page systemd.unit(5), I'm not quite sure what the 
> difference between AssertPathExists and ConditionPathExists is:
>>
>> (Condition)
>> If the specified absolute path name does not exist, the condition will fail.
>>
>> Vs.
>>
>> (Assert)
>> However, unlike the conditions settings, any assertion setting that is not 
> met results
>> in failure of the start job it was triggered by.
>> 
>> What does that mean effectively?
>>
> 
> Assert makes the start job fail, which triggers any followup failure
> processing (like failure of other units Requiring this one). Condition
> silently skips starting of this unit but proceeds normally with
> starting dependent units.

OK, ist that "skip" actually a "skip once (and silently) for now", or is it 
more like a "delay until no longer true"?

Regards,
Ulrich

[systemd-devel] Q: Relying on exit code of journalctl -g ....

[Ulrich Windl]

Hi!

It's not documented, but "journalctl -g ..." seems to exit with success when 
the pattern was found, and with 1 when it was not.
Can I rely on that?

The other thing is when grepping across reboots, the "-- Reboot --" is always 
output, even if it does not match the pattern. Is that a bug or a feature?

(systemd-246.16 of SLES15 SP3)

Actually I had some "quiet grep" in mind that only sets the exit code depending 
whether some message was found or not.

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: version bump of minimal kernel version supported by systemd?

[Ulrich Windl]

I wonder:

Why not providing some test suite instead: If the test suite succeeds, systemd
might work; if it doesn't, manual steps are needed.
My guess is that most people will quit trying once manual steps are needed.
In addition the configure procedure could include a "support window" (oldest
kernel supported, latest kernel supported).
And f the kernel is outside, print a warning at least that the configuration
is not supported.
Still I guess the kernel version is not the only dependency...

Regards,
Ulrich

>>> Luca Boccassi  schrieb am 23.03.2022 um 16:58 in
Nachricht
<4b96e540891ee7a43917ba377806e520a0d7be02.ca...@debian.org>:
> On Wed, 2022-03-23 at 12:38 +0100, Greg KH wrote:
>> On Wed, Mar 23, 2022 at 11:28:29AM +, Luca Boccassi wrote:
>> > On Wed, 2022-03-23 at 11:59 +0100, Zbigniew Jędrzejewski-Szmek wrote:
>> > > On Wed, Mar 23, 2022 at 09:26:05AM +0100, Greg KH wrote:
>> > > > On Wed, Mar 23, 2022 at 09:17:36AM +0100, Zbigniew Jędrzejewski-Szmek

> wrote:
>> > > > > On Wed, Mar 23, 2022 at 08:07:48AM +0100, Greg KH wrote:
>> > > > > > On Tue, Mar 22, 2022 at 05:27:07PM +0100, Zbigniew
Jędrzejewski-Szmek 
> wrote:
>> > > > > > > Hi all,
>> > > > > > > 
>> > > > > > > we are considering dropping upstream support for kernel
versions < 4.4.
>> > > > > > > Would this be a problem for anyone? (*).
>> > > > > > 
>> > > > > > Given that upstream (i.e. kernel.org) has dropped support for
kernel
>> > > > > > 4.4, why not just move to not supporting kernels older than 4.9?
>> > > > > 
>> > > > > It seems Civil Infrastructure Platform (a project under the Linux
>> > > > > Foundation) still uses 4.4 [1].
>> > > > 
>> > > > Yes, but they are not going to be updating to a newer version of
>> > > > systemd, right?
>> > > > 
>> > > > And they are going to be "supporting" that for 20+ years.  If they
want
>> > > > to do something crazy like this, make them handle supporting code
that
>> > > > is older than 6+ years to start with.  That's not the community's
issue,
>> > > > that's the companies that demand such crazy requirement's issue.
>> > > 
>> > > That's why I (we) asked the question on the list. If people are
compling
>> > > systemd on such old systems, or even older, we want to know about it.
>> > > 
>> > > > > In the Debian world, Stretch which has EOL scheduled for June 2022
has 
> 4.9,
>> > > > > and after that Buster has 4.19.
>> > > > 
>> > > > 4.9 is fine, and is supported by kernel.org until next year as seen
>> > > > here:
>> > > >https://kernel.org/category/releases.html 
>> > > > 
>> > > > I wrote "4.9" above, not "4.19" :)
>> > > 
>> > > Yep. I'd vote for bumping to 4.9, unless some other voices pop up.
>> > > 
>> > > Zbyszek
>> > 
>> > Let's do 4.4 at most please - what's on kernel.org is not really that
>> > important, as real usage is downstream from there anyway.
>> 
>> And I will publically state that anyone still using 4.4.y today has an
>> insecure and unsupported system.  Please let's not encourage _ANYONE_ to
>> do this.
>> 
>> CIP is "special" in that they know what they are doing, and are using
>> 4.4.y in a very limited set of use cases and configurations.  And even
>> they are going to have big problems keeping that kernel alive and
>> secure.  I would never expect anyone else to be able to do it, and I
>> have doubts that they will either.
>> 
>> So any "real usage" of 4.4 after today, should not matter.  And if
>> someone complains, send them to me please.
>> 
>> thanks,
>> 
>> greg k-h
> 
> You can publically state that all day long, but you know perfectly well
> that's not how the world works. In the grand scheme of things few
> production scenarios build their kernel from kernel.org, most will be
> getting it from a distro (best case) or a vendor (worst case), and they
> couldn't care less about what kernel.org tells them to do, they use
> what they get. I fully expect at some point to hear complaints from
> some poor soul stuck on 3.x because of $crappy_arm_vendor with no way
> to move on from there.
> 
> Jumping forward from 3.13 to 4.4 as the baseline, allowing to take
> cgroupsv2 for granted, seems like a good starting point to me. There's
> very obvious and public evidence of that being used in the wild. We can
> start to drop a bunch of backward-compat cruft, wait and see who
> complains, and if nobody does we can re-evaluate again in a couple of
> years.
> 
> -- 
> Kind regards,
> Luca Boccassi

[systemd-devel] Antw: [EXT] Re: /etc/os-release but for images

[Ulrich Windl]

>>> Davide Bettio  schrieb am 23.03.2022 um 17:14 in
Nachricht
:
> Hello,
> 
> Il giorno mer 23 mar 2022 alle ore 13:56 Lennart Poettering <
> lenn...@poettering.net> ha scritto:
> 
>> > Also sadly IMAGE_VERSION doesn't allow + which is used from semver for
>> > build metadata (such as 1.0.0+21AF26D3 or 1.0.0+20130313144700).
>>
>> Ah, interesting. This looks like something to fix in our syntax
>> descriptio though. I am pretty sure we should allow all characters
>> that semver requires.
>>
>> Can you file an RFE issue about this on github? Or even better, submit
>> a PR that adds that?
>>
> 
> I opened this PR: https://github.com/systemd/systemd/pull/22841 
> 
> This doesn't enable full semver support since that would require allowing
> A-Z range, but I'm not sure if it is something we really want to enable
> (uppercase in semver looks quite uncommon by the way).
> 
> That said, I'd always include some time-based counter in automatic
>> builds, so that the builds can be ordered. Things like sd-boot's boot
>> menu entry ordering relies on that.
>>
> 
> This is indeed a good advice.
> 
> 
>> > That's pretty useful when storing a relation between the exact update
>> > bundle that has been used to flash a device, and the system flashed on a
>> > device. It turns out to be pretty useful when collecting stats about a
>> > device fleet or when remote managing system versions and their updates.
>> > So what I would do on os-release would be storing an UUID that is
>> generated
>> > every time a system image is generated, that UUID can be collected/sent
>> at
>> > runtime from a device to a remote management service.
>>
>> Why wouldn't the IMAGE_VERSION suffice for that? Why pick a UUID where
>> a version already works?
>>
> I would use the UUID as a further metadata in addition to IMAGE_VERSION,
> also for the reasons I describe later here in this mail.

I wonder: Why not simply an md5sum over the image (or you favourite checksum)?
Does the same UUID really imply the image is the same? Likewise what about 
different UUIDs?

Regards,
Ulrich

> 
> 
>>
>> > Compared to other options an UUID here would be both reliable and easy to
>> > handle and generate.
>>
>> UUID is are effectively randomly generated. That sucks for build
>> processes I am sure, simply because they hence aren't reproducible.
>>
> 
> Using a reliable digest, such as the one that can be generated with `casync
> digest`, was my first option, however if you think about an update system
> such such as RAUC and its bundles, you might still have the same exact
> filesystem digest mapping to a number of different bundles, since they can
> bring different hook scripts and whatever.
> I'm also aware of scenarios where the same filesystem tree has been used to
> generate different flash images in order to satisfy different flash sizes /
> NAND technologies.
> So from a practical point of view having an UUID, and forcing a different
> one in /etc/os-release every time a filesystem image / RAUC bundle is
> created allows us to have a reasonable 1:1 mapping between the update
> artifact and the system image that is on the device.
> Last but not least having it in /etc/os-release makes it pretty convenient
> to read it (and for sure using an UUID is easier than trying to embed the
> digest of the filesystem where  /etc/os-release is kept ;) )
> Also there is an interesting bonus: UUID is globally unique also in
> scenarios where users try to delete and recreate version tags without
> incrementing the version number (or other messy scenarios).
> 
> BTW, there's now also this:
>>
>> https://systemd.io/BUILDING_IMAGES/#image-metadata 
>>
> 
> Thanks, this feature looks pretty interesting.
> 
> Regards,
> Davide Bettio.

[systemd-devel] Antw: [EXT] Re: version bump of minimal kernel version supported by systemd?

[Ulrich Windl]

>>> Dave Howorth  schrieb am 23.03.2022 um 23:34 in
Nachricht <20220323223400.569e0...@acer-suse.lan>:
> On Wed, 23 Mar 2022 22:26:15 +0100
> Michael Biebl  wrote:
>> Am Mi., 23. März 2022 um 22:11 Uhr schrieb Zbigniew Jędrzejewski-Szmek
>> :
>> 
>> > Or in other words: I'd prefer for such people to speak up for
>> > themselves, rather than us trying to figure out what somebody else
>> > *might* be planning to do.  
>> 
>> That's laudable but keep in mind that users typically don't follow
>> systemd-devel actively. I'm pretty sure we don't even have
>> *maintainers* of embedded Linux following this mailing list.
> 
> I'm just a user and I follow this list. That is to say, it doesn't
> matter what typical users or maintainers do, it's enough if just one
> relevant person is following. It would be nice though, if there are any
> such people, if they would speak up and offer their opinion or say that
> they are going to take it to whichever manager/committee they need to.
> 
> FWIW, I think Greg was a bit too outspoken calling long maintenance
> attempts 'crazy'; that may have intimidated some. I'm thinking of
> moving distro to one that provides longer term maintenance than my
> present one. Although CIP is a completely different ball game; I hope
> they succeed.

Actually I think the audience of this list is irrelevant when the supported
configuration(s) are described in Release Notes and/or README/NEWS.

> 
> Cheers, Dave

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] version bump of minimal kernel version supported by systemd?

[Ulrich Windl]

>>> Greg KH  schrieb am 24.03.2022 um 08:12 in
Nachricht :
> On Wed, Mar 23, 2022 at 10:34:00PM +, Dave Howorth wrote:
>> FWIW, I think Greg was a bit too outspoken calling long maintenance
>> attempts 'crazy'; that may have intimidated some. I'm thinking of
>> moving distro to one that provides longer term maintenance than my
>> present one. Although CIP is a completely different ball game; I hope
>> they succeed.
> 
> It is not "crazy" it is "well documented".  As someone who has been
> doing this work for 20+ years now and sees all of the stable kernel
> patches flow by, it's obvious that a distro that does not keep up with
> them is insecure by design.

If "newer is better" I'd agree. Sometimes "newer is actually worse".
Some new features intended to improve things sometimes actually make things 
worse.

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: version bump of minimal kernel version supported by systemd?

[Ulrich Windl]

>>> Greg KH  schrieb am 24.03.2022 um 14:22 in
Nachricht :
> On Thu, Mar 24, 2022 at 02:05:09PM +0100, Zbigniew Jędrzejewski-Szmek
wrote:
>> On Thu, Mar 24, 2022 at 10:28:39AM +, Luca Boccassi wrote:
>> > On Thu, 2022-03-24 at 09:38 +0100, Lennart Poettering wrote:
>> > > On Mi, 23.03.22 11:28, Luca Boccassi (bl...@debian.org) wrote:
>> > >
>> > > > At least according to our documentation it wouldn't save us much
>> > > > anyway, as the biggest leap is taking cgroupv2 for granted, which
>> > > > requires 4.1, so it's included regardless. Unless there's something
>> > > > undocumented that would make a big difference, in practical terms of
>> > > > maintainability?
>> > >
>> > > Note that "cgroupv2 exists" and "cgroupv2 works well" are two distinct
>> > > things. Initially too few controllers supported cgroupv2 for cgroupv2
>> > > to be actually useful.
>> > >
>> > > What I am trying to say is that it would actually help us a lot if
>> > > we'd not just be able to take croupv2 for granted but to take a
>> > > reasonably complete cgroupv2 for granted.
>> >
>> > Yes, that does sound like worth exploring - our README doesn't document
>> > it though, do we have a list of required controllers and when they were
>> > introduced?
>> 
>> In the README:
>>   Linux kernel >= 4.2 for unified cgroup hierarchy support
>>   Linux kernel >= 4.10 for cgroup-bpf egress and ingress hooks
>>   Linux kernel >= 4.15 for cgroup-bpf device hook
>>   Linux kernel >= 4.17 for cgroup-bpf socket address hooks
>> 
>> In this light, 4.19 is better than 4.4 or 4.9 ;)
> 
> Then move to 4.19.  I strongly doubt that any distro that is using older
> kernels would ever be willing to update systemd.

SLES12 SP5 is running at 4.12, but right: Even with significant rest support
time they won't change the systemd major version, I guess.

> 
> thanks,
> 
> greg k-h

[systemd-devel] Q: Not "Reloading"?

[Ulrich Windl]

Hi!

I wonder: Seeing these messages (really nothing in between) I wonder whether 
such massive systemd reloading can be avoided.
Of course no program knows what the other did or does, but still these messages 
look undesirable:

Apr 01 08:09:35 h19 systemd[1]: Reloading.
Apr 01 08:09:36 h19 systemd[1]: Reloading.
Apr 01 08:09:37 h19 systemd[1]: Reloading.
Apr 01 08:09:37 h19 systemd[1]: Reloading.
Apr 01 08:09:38 h19 systemd[1]: Reloading.
Apr 01 08:09:39 h19 systemd[1]: Reloading.
Apr 01 08:09:39 h19 systemd[1]: Reloading.
Apr 01 08:09:40 h19 systemd[1]: Reloading.
Apr 01 08:09:41 h19 systemd[1]: Reloading.
Apr 01 08:09:41 h19 systemd[1]: Reloading.
Apr 01 08:09:42 h19 systemd[1]: Reloading.
Apr 01 08:09:42 h19 systemd[1]: Reloading.
Apr 01 08:09:43 h19 systemd[1]: Reloading.
Apr 01 08:09:44 h19 systemd[1]: Reloading.
Apr 01 08:09:45 h19 systemd[1]: Reloading.
Apr 01 08:09:45 h19 systemd[1]: Reloading.
Apr 01 08:09:46 h19 systemd[1]: Reloading.
Apr 01 08:09:47 h19 systemd[1]: Reloading.
Apr 01 08:09:47 h19 systemd[1]: Reloading.
Apr 01 08:09:48 h19 systemd[1]: Reloading.
Apr 01 08:09:49 h19 systemd[1]: Reloading.
Apr 01 08:09:49 h19 systemd[1]: Reloading.
Apr 01 08:09:50 h19 systemd[1]: Reloading.
Apr 01 08:09:51 h19 systemd[1]: Reloading.
Apr 01 08:09:51 h19 systemd[1]: Reloading.
Apr 01 08:09:52 h19 systemd[1]: Reloading.
Apr 01 08:09:53 h19 systemd[1]: Reloading.
Apr 01 08:09:53 h19 systemd[1]: Reloading.
Apr 01 08:09:54 h19 systemd[1]: Reloading.
Apr 01 08:09:55 h19 systemd[1]: Reloading.
Apr 01 08:09:55 h19 systemd[1]: Reloading.
Apr 01 08:09:56 h19 systemd[1]: Reloading.
Apr 01 08:09:57 h19 systemd[1]: Reloading.

Those messages were triggered by some resource agents in a pacemaker cluster.

Regards,
Ulrich

[systemd-devel] Antw: Re: Antw: [EXT] Re: [systemd‑devel] version bump of minimal kernel version supported by systemd?

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 01.04.2022 um 14:18
in
Nachricht :
...
> time. Often for good reasons, quite often also for no reason but lack
> of testing. Things like that will happen. But I also think that
> Windows for example is probably better at not breaking their
> interfaces than Linux is.

Nonsense: I had several Windows program that did not work correctly any more
when moving to the next version of Windows.
In the second to next version, some of the programs worked again!
(I'm not talking about monthly updates, I'm talking about really new
versions)

> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Q: journalctl -b -g logrotate

[Ulrich Windl]

Hi!

I have two questions for "journalctl -b -g logrotate":

1) I'm unsure what the exact rules for matching a "-g expression" are: Some 
kernel messages are matched, others not.
2) When the -b restricts messages to the current boot, why is output shown like 
this?:
# journalctl -b -g logrotate
-- Logs begin at Wed 2020-11-25 11:27:53 CET, end at Tue 2022-04-05 08:01:02 
CEST. --

I mean the boot was definitely in 2022, so I think the message is not really 
helpful. Why not show the date and time when the search starts (i.e. boot time)?

The next thing is "-k": If I supply it, kernel messages are _not_ found:
# journalctl -S 2022-04-02 -k | grep "OCFS2:" |head
# journalctl -S 2022-04-02 | grep "OCFS2:" |head
Apr 02 02:00:06 h18 kernel: OCFS2: ERROR (device dm-17): 
ocfs2_validate_gd_self: Group descriptor #209970 has bad signature EXBLK01
Apr 02 02:00:06 h18 kernel: OCFS2: File system is now read-only.
Apr 02 02:00:07 h18 kernel: OCFS2: ERROR (device dm-17): 
ocfs2_validate_gd_self: Group descriptor #209817 has bad signature EXBLK01
Apr 02 02:00:07 h18 kernel: OCFS2: ERROR (device dm-17): 
ocfs2_validate_gd_self: Group descriptor #209946 has bad signature EXBLK01

So can I find kernel messages from previous boots?

(systemd-246.16-150300.7.39.1.x86_64 of SLES15 SP3)

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: Q: journalctl -b -g logrotate

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 05.04.2022 um 11:08 in
Nachricht
:
> On Tue, Apr 5, 2022 at 9:36 AM Ulrich Windl <
> ulrich.wi...@rz.uni-regensburg.de> wrote:
> 
>> Hi!
>>
>> I have two questions for "journalctl -b -g logrotate":
>>
>> 1) I'm unsure what the exact rules for matching a "-g expression" are:
>> Some kernel messages are matched, others not.
>>
> 
> All entries with a MESSAGE= are matched (after doing until/since/boot-id
> checks). They might still be hidden for other reasons though, e.g. messages
> containing weird escape characters (or accidental binary data) will be
> hidden unless you use -a.

And how do I find out whether a kernel message has a MESSAGE=?

As soon as I add _MESSAGE= I get no output any more (even with MESSAGE=.*).

> 
> 
>> 2) When the -b restricts messages to the current boot, why is output shown
>> like this?:
>> # journalctl -b -g logrotate
>> -- Logs begin at Wed 2020-11-25 11:27:53 CET, end at Tue 2022-04-05
>> 08:01:02 CEST. --
>>
>> I mean the boot was definitely in 2022, so I think the message is not
>> really helpful. Why not show the date and time when the search starts
(i.e.
>> boot time)?
>>
> 
> There's no such message in the current systemd version. See
> https://github.com/systemd/systemd/pull/21775.
> 
> 
>>
>> The next thing is "-k": If I supply it, kernel messages are _not_ found:
>> # journalctl -S 2022-04-02 -k | grep "OCFS2:" |head
>> # journalctl -S 2022-04-02 | grep "OCFS2:" |head
>> Apr 02 02:00:06 h18 kernel: OCFS2: ERROR (device dm-17):
>> ocfs2_validate_gd_self: Group descriptor #209970 has bad signature EXBLK01
>> Apr 02 02:00:06 h18 kernel: OCFS2: File system is now read-only.
>> Apr 02 02:00:07 h18 kernel: OCFS2: ERROR (device dm-17):
>> ocfs2_validate_gd_self: Group descriptor #209817 has bad signature EXBLK01
>> Apr 02 02:00:07 h18 kernel: OCFS2: ERROR (device dm-17):
>> ocfs2_validate_gd_self: Group descriptor #209946 has bad signature EXBLK01
>>
>> So can I find kernel messages from previous boots?
>>
> 
> `journalctl -k` is meant to imitate dmesg (except with correct timestamps),
> so it shows the current boot only. You can use _TRANSPORT=kernel to filter
> for kernel messages if you don't want that.
> 
> $ journalctl _TRANSPORT=kernel -g BogoMIPS

Yup, that works!

> 
> -- 
> Mantas Mikulėnas

[systemd-devel] Antw: [EXT] Dropping split-usr/unmerged-usr support

[Ulrich Windl]

>>> Luca Boccassi  schrieb am 05.04.2022 um 22:07 in
Nachricht <05cf10d04274dcbff07fed88e98dca2eebb24b7d.ca...@gmail.com>:
> Hi,
> 
> As part of our spring cleaning effort, we are considering when to drop
> support for split/unmerged-usr filesystem layouts.
> 
> A build-time warning was added last year:
> 
> https://github.com/systemd/systemd/commit/9afd5e7b975e8051c011ff9c07c95e80bd 
> 954469

Honestly to me the requirement that /usr be part of the root filesystem never 
had a reasonable argument.
Instead I think systemd quit the concept of a simple scaled-down subset to 
bring up the system.
Also with initrd/dracut the concept is even more odd, because the /usr found 
there is just some arbitrary subset of the real /usr (similar for other 
filesystems).
So why couldn't that work with a really scaled-down /sbin?

> 
> We are now adding a runtime taint as well.
> 
> Which distributions are left running with systemd on a split/unmerged-
> usr system?
> 
> (reminder: we refer to a system that boots without a populated /usr as
> split-usr, and a system where bin, sbin and lib* are not symlinks to
> their counterparts under /usr as unmerged-usr)

Symlinking /sbin or /usr/sbin binaries to /usr is also a bad concept IMHO.

It seems systemd is the new Microsoft ("We know what is good for you; just 
accept it!") ;-)

Regards,
Ulrich

> 
> -- 
> Kind regards,
> Luca Boccassi

[systemd-devel] Antw: [EXT] Re: Dropping split-usr/unmerged-usr support

[Ulrich Windl]

>>> Mike Gilbert  schrieb am 06.04.2022 um 17:24 in 
>>> Nachricht
:
> On Tue, Apr 5, 2022 at 4:07 PM Luca Boccassi  wrote:
>>
>> Hi,
>>
>> As part of our spring cleaning effort, we are considering when to drop
>> support for split/unmerged-usr filesystem layouts.
>>
>> A build-time warning was added last year:
>>
>> 
> https://github.com/systemd/systemd/commit/9afd5e7b975e8051c011ff9c07c95e80bd9 
> 54469
>>
>> We are now adding a runtime taint as well.
>>
>> Which distributions are left running with systemd on a split/unmerged-
>> usr system?
> 
> Gentoo still supports having /{bin,sbin,lib} and /usr/{bin,sbin,lib}
> as separate directories. We do not support officially booting without
> /usr mounted (via initramfs), but some users do it anyway.
> 
> We are not likely to require merging of / and /usr for the foreseeable
> future. We are a "rolling release" distro and basically never require
> our users to re-install, which makes large file system migrations
> difficult. Also, many of our users would resist any attempt to force
> merged-/usr on them.

Well, many years ago HP-UX (I think it was around major version 9 or 10) 
managed to migrate from /bin to /sbin and /usr/sbin (also from /usres/ to 
/home) without reinstallation.
I always felt what systemd is demanding is a step back.

> 
> I think it would be ok if systemd drops support for installing itself
> in /lib/systemd; we would just move everything under /usr/lib/systemd,
> and possibly set up some symlinks in /lib/systemd for the transition.
> 
> We will still need to keep /bin and /sbin in PATH, and we can't assume
> that all binaries reside in /usr/bin.

What speaks for /sbin and /usr/sbin IMHO is the fact that the typical user did 
not have them in the PATH, because those were mostly management commands a 
normal user does not need.
Now we have a "huge pot" with everything in /usr/bin.

Regards,
Ulrich

[systemd-devel] Antw: Re: Antw: [EXT] Dropping split-usr/unmerged-usr support

[Ulrich Windl]

>>> Wols Lists  schrieb am 06.04.2022 um 21:41 in
Nachricht :
> On 06/04/2022 10:34, Luca Boccassi wrote:
>>> Symlinking /sbin or /usr/sbin binaries to /usr is also a bad concept
>>> IMHO.
>>>
>>> It seems systemd is the new Microsoft ("We know what is good for you;
>>> just accept it!");-)
> 
> Well, I saw a link to WHY we have /bin, /usr/bin, /sbin etc. Interesting 
> read ...
> 
> / was disk0. /usr was apparently originally short for /user, on disk1. 
> Then the system disk ran out of space, so they created /usr/bin to have 
> more space. So when they got a 3rd disk, they called it /home and moved 
> all the user directories across ...

However space is not the only reason: Back in the times of non-journaling 
filesystems (and slow disks where a fsck could take 40 minutes or more) it was 
highly desirable to have a small root filesystem that could be checked quickly, 
to root had the chance to become active.
Even today when something bad happens, one would probably prefer to have 
multiple smaller filesystems to repair rather than one "huge pot". MHO.
Agreed Windows users who only know C: never wasted much thoughts on structure; 
see the mess in C:\Windows. But I thought UNIX was highly structured...

Regards,
Ulrich


>>>
>>> Regards,
>>> Ulrich
>> Sorry, but you are about ~10 years late to this debate:-)  The question
>> today is not whether it's good or bad, but who's left to do the switch.
>> 
>> We know Fedora/RHEL/CentOS/SUSE/Arch/Ubuntu have done the switch, and
>> presumably any of their derivatives.
>> 
>> We know Debian is, er, working on it, as per the most recent article on
>> LWN.
>> 
>> What about other distros that are not derivatives of the aboves and
>> that use systemd? Does anybody have any insight?
> 
> gentoo defaults to OpenRC, but I'm running systemd. As far as I can tell 
> I appear to have three distinct directories, /bin, /sbin, and /usr
> 
> I also have a fourth and fifth distinct directory, /usr/bin and 
> /usr/sbin. What if any plans there are to merge them I have no clue.
> 
> Cheers,
> Wol

[systemd-devel] Antw: Re: Antw: Re: Antw: [EXT] Dropping split-usr/unmerged-usr support

[Ulrich Windl]

>>> Neal Gompa  schrieb am 07.04.2022 um 10:53 in
Nachricht
:

...
> On the contrary, Windows has been much more organized than UNIX has
> been. In the C:\ hierarchy, the "Windows" directory contains all the

Ok, A fan of Windows!

> resources to run Windows itself. System-wide applications are all in
> "Program Files", and user data is in "Users". Those three directories

Several EXEs are in c:\Windows (like notepad.exe), more are in
C:\Windows\System32 like write.exe.
Those are really _not_ system programs.

There are more in SysWOW64, WinSxS, Microsoft.NET, etc.

> form the core of the Windows experience. There are obviously more
> directories, but those three are essential for Windows itself. And if
> you don't need any applications (just the base Windows OS), then you
> can get away with just C:\Windows.

Windows is a hug pot of mess: Why are the ICC color profiles in a "spool"
directory (C:\Windows\System32\spool\drivers\color)?
Why are drivers in a spool directory?

> 
> UNIX, meanwhile, didn't have an opportunity to be thoughtful
> about how the hierarchy worked. Things got stuffed in where they could

UNIX had a hierarchy when MS-DOS still had everything in the root directory 10
years later.
Early versions of the FAT filesytem did not even have subdirectories.

> based on the size of diskettes and what could be held in memory. The
> fact that /usr doesn't actually represent where user data is proves
> it. "Unix System Resources" is a backronym to attempt to deal with the

In fact early UNIX systems _had_ user data in /usr.
Subdirectory in MS-DOS was so terribly slow that subdirectories were avoided.
In fact I think they started using subdirectories when the root filesystem was
full with 144 entries (or so).

---citation (UNIX Version 7 Volume 2B)---
New Users
Install new users by editing the password file /etc/passwd (passwd(5)). This
procedure should be
done once multi-user mode is entered (see init(8)). You’ll have to make a
current directory for each
new user and change its owner to the newly installed name. Login as each user
to make sure the password
file is correctly edited. For example:
ed /etc/passwd
$a
joe::10:1::/usr/joe:
w
q
mkdir /usr/joe
chown joe /usr/joe
login joe
ls – la
login root
This will make a new login entry for joe, who should be encouraged to use
passwd(1) to give himself a
password. His default current directory is /usr/joe which has been created.
The delivered password file
has the user bin in it to be used as a prototype.

(ed once was considered as genuine as systemd is now ;-)

> mistake of not renaming the directory when it evolved away from
> holding user data. The Unix hierarchy is *full* of mistakes and
> post-rationalizations ideally would be fixed someday but probably
> won't be.

Oh well: Windows 10 still has ini-files in \Windows, probably going back to
the times of Windows <=3 when there was no users and permissions, so they had
no concept of protection at all.
Now I think Windows is really not an example how to do things right.

Regards,
Ulrich Windl

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Dropping split‑usr/unmerged‑usr support

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 07.04.2022 um 10:59
in
Nachricht :
> On Mi, 06.04.22 11:24, Mike Gilbert (flop...@gentoo.org) wrote:
> 
>> We are not likely to require merging of / and /usr for the foreseeable
>> future. We are a "rolling release" distro and basically never require
>> our users to re‑install, which makes large file system migrations
>> difficult. Also, many of our users would resist any attempt to force
>> merged‑/usr on them.
> 
> So, my guess would be that the people who dislike merged‑/usr are also
> the ones who dislike systemd, no? i.e. do they really matter if we are
> talking about what to support in systemd? They'd not use our stuff
> anyway, so why bother?

Maybe explain why systemd required /usr to be part of the root filesystem
instead.
Maybe also explain the chicken-egg race with temporary directories that
systemd added.

> 
>> I think it would be ok if systemd drops support for installing itself
>> in /lib/systemd; we would just move everything under /usr/lib/systemd,
>> and possibly set up some symlinks in /lib/systemd for the
>> transition.
> 
> You guys are making your life hell, because you are afraid if making
> it difficult...

Maybe just add a manual page systemd-filesystem-concepts ;-)

Regards,
Ulrich Windl


> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] all numeric usernames not allowed in systemd (version 245 onward)

[Ulrich Windl]

>>> Nikhil Kshirsagar  schrieb am 07.04.2022 um 13:07 in
Nachricht
:
> Hello,
> 
> I gather from the discussion on
> https://github.com/systemd/systemd/issues/15141 that all numeric usernames
> would no longer be supported on servers running systemd version 245 onward.
> This was also reiterated by Frantisek and Lennart (thank you for your email
> responses and for redirecting me to this mailing list.)

AFAIK there never were numeric user names in UNIX: If a user specification ws 
numeric, it was the UID of a user, not the name.
(VAX had different concepts however)
I mean: If you wanted to shoot yourself in the foot you would use numeric user 
names like this:

(pwasswd)
123:333:...
333:123:...

chown 333 file...

> 
> While I completely understand the motivation to do this, my concern is that
> this change will break logins for users on a lot of servers that upgrade to
> the new systemd.

Can you give examples of numeric user names being used?

> 
> Is there any chance systemd could support a configuration option in the
> future to get the earlier "all numeric" user logins to work? With an
> understanding that it would be at the users own risk?
> 
> Are there any pam_systemd configuration options under consideration that
> might help anyone looking for such functionality? Or is the only option
> "not to upgrade" for someone interested in preserving their all numeric
> usernames?
> 
> Regards,
> Nikhil.

[systemd-devel] Antw: [EXT] Re: Dropping split-usr/unmerged-usr support

[Ulrich Windl]

>>> Wol  schrieb am 08.04.2022 um 00:02 in Nachricht
:
> On 07/04/2022 17:47, Mike Gilbert wrote:
>>> So, my guess would be that the people who dislike merged-/usr are also
>>> the ones who dislike systemd, no? i.e. do they really matter if we are
>>> talking about what to support in systemd? They'd not use our stuff
>>> anyway, so why bother?
> 
> There's probably also a big minority of users (like me) who may be 
> pro-systemd, but run a systemd-hostile distro for reasons that are 
> nothing to do with systemd ...

I always thought it was a distro-hostile systemd ;-)
(You know what was first)

...

Ulrich

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Intercepting/Delaying the boot process

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 08.04.2022 um 15:14 
>>> in
Nachricht :

...
> This reminds of an RFE we have had for a while, and which I think
> would make sense to add directly to systemd: a generator that detects
> whether the battery is nearly empty at boot, and if so redirects boot
> to some special target showing a nice message for a bit and then
> powering off.

Why does it have to be a generator; can't a normal unit detect and handle that?

...

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: Samba Config Reload

[Ulrich Windl]

Hi!

I thin Lennart had pointed it out: If the sapplication being reloaded does not 
provide any feedback when the reloading is complete, you can never be sure what 
it did complete.
Adding some sleep may catch a grat number of cases whule waiting too long in 
most cases.

So before discussing systemd meachnisms: How do you know when reload is 
complete?

Regards,
Ulrich

>>> Wols Lists  schrieb am 09.04.2022 um 17:10 in
Nachricht :
> On 09/04/2022 09:00, Yolo von BNANA wrote:
>> Can you please explain this in more Detail?
>> 
>> What does this mean: " "systemctl reload" will basically return
>> immediately without the reload being complete"?
>> 
>> And what is an Example for an synchronous command for ExecReload=
>> 
> Do you understand the difference between "synchronous" and 
> "asynchronous"? The words basically mean "aligned in time" and "without 
> timed alignment".
> 
> Think of writing to files. In the old days of MS-DOS et al, when your 
> program called "write", the CPU went off, saved the data to disk, and 
> returned to your program. That's "synchronous", all nicely ordered in 
> time, and your program knew the data was safe.
> 
> Now, when your linux program calls write, linux itself replies "got it", 
> and your program goes off knowing that something else is going to take 
> care of actually saving the data to disk - that's "asynchronous". Except 
> that sometimes the program needs to know that the data HAS been safely 
> squirreled away (hence all these fsync calls).
> 
> So when systemd calls ExecReload *A*synchronously, it goes off and fires 
> off a load more stuff, knowing that the ExecReload IS GOING (future 
> tense) to happen. What the previous poster wanted was a synchronous 
> ExecReload, so that when systemd goes off do the next thing, the 
> ExecReload HAS ALREADY HAPPENED (past tense). (Which in general is a bad 
> thing because it *seriously* knackers performance).
> 
> Cheers,
> Wol

[systemd-devel] Antw: [systemd‑devel] Antw: [EXT] Re: Samba Config Reload

[Ulrich Windl]

>>> "Ulrich Windl"  schrieb am 11.04.2022
um
08:26 in Nachricht <6253ca1802a100049...@gwsmtp.uni-regensburg.de>:
> Hi!
> 

Sorry for the typos:

> I thin Lennart had pointed it out: If the sapplication being reloaded does 
s/thin/think/
> not provide any feedback when the reloading is complete, you can never be 
> sure what it did complete.

s/what/when/

> Adding some sleep may catch a grat number of cases whule waiting too long in


s/grat/great/; s/whule/while/

> most cases.
> 
> So before discussing systemd meachnisms: How do you know when reload is 
> complete?
> 
> Regards,
> Ulrich
> 
>>>> Wols Lists  schrieb am 09.04.2022 um 17:10 in
> Nachricht :
>> On 09/04/2022 09:00, Yolo von BNANA wrote:
>>> Can you please explain this in more Detail?
>>> 
>>> What does this mean: " "systemctl reload" will basically return
>>> immediately without the reload being complete"?
>>> 
>>> And what is an Example for an synchronous command for ExecReload=
>>> 
>> Do you understand the difference between "synchronous" and 
>> "asynchronous"? The words basically mean "aligned in time" and "without 
>> timed alignment".
>> 
>> Think of writing to files. In the old days of MS‑DOS et al, when your 
>> program called "write", the CPU went off, saved the data to disk, and 
>> returned to your program. That's "synchronous", all nicely ordered in 
>> time, and your program knew the data was safe.
>> 
>> Now, when your linux program calls write, linux itself replies "got it", 
>> and your program goes off knowing that something else is going to take 
>> care of actually saving the data to disk ‑ that's "asynchronous". Except 
>> that sometimes the program needs to know that the data HAS been safely 
>> squirreled away (hence all these fsync calls).
>> 
>> So when systemd calls ExecReload *A*synchronously, it goes off and fires 
>> off a load more stuff, knowing that the ExecReload IS GOING (future 
>> tense) to happen. What the previous poster wanted was a synchronous 
>> ExecReload, so that when systemd goes off do the next thing, the 
>> ExecReload HAS ALREADY HAPPENED (past tense). (Which in general is a bad 
>> thing because it *seriously* knackers performance).
>> 
>> Cheers,
>> Wol

[systemd-devel] Q: Forwarding journal to "remote" syslogd?

[Ulrich Windl]

Hi!

When trying to configure forwarding of journal messages to a remote syslogd, I 
realized that it seems not possible (systemd 228 of SLES12 SP5).
Is that true?
What I want is forwarding to "host:port" via UDP.

Forwarding to a local syslogd just to be able to forward to a remote syslogd 
seems to be a terrible concept (MHO).

Regards,
Ulrich

[systemd-devel] Antw: [EXT] Re: device unit files

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 13.04.2022 um 17:38
in
Nachricht :
> On Di, 12.04.22 14:38, Elbek Mamajonov (emm.boxin...@gmail.com) wrote:
> 
>> On graph I have mmcblk.device taking 1.624s. From the point that
>> this partition is where my rootfs lies, and systems does remounting
>> of rootfs, I did look up for systemd-remount-fs.service, it took
>> 231ms, and systemd-udev-trigger.service, as you suggested, it took
>> 517ms to become active. But even after systemd-udev-trigger.service
>> becomes active there is about 800ms for mmcblk.device to become
>> active. Are those services related to the activation of the
>> mmcblk.device? Can I consider those 231ms and 517ms as a part of the
>> activation time of the mmcblk.device? How can I debug udev in this
>> case?
> 
> "systemd-udev-trigger.service" might take a while before it completes,
> since it triggers basically all devices in the system.
> 
> It might be worth triggering block devices first. With upcoming v251

What is the expected benefit? On bigger servers with hundreds of disks this
may take longest.

> we actually will do that by default. But until then you could extend
> the service by first issuing "udevadm trigger -sblock", before the
> rest.
> 
> udev will announce devices to the system (and thus also PID1) once it
> probed the device. it does this based on rules, and the default rules
> will run blkid on the device, to see what's on it (i.e. to extract fs
> label/uuid, …). maybe that's just terribly slow on your device?
> 
> Lennart
> 
> --
> Lennart Poettering, Berlin

Re: [systemd-devel] Antw: [EXT] Re: [systemd‑devel] device unit files

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 14.04.2022 um 09:45
in
Nachricht :
> On Do, 14.04.22 08:00, Ulrich Windl (ulrich.wi...@rz.uni‑regensburg.de)
wrote:
> 
>> >>> Lennart Poettering  schrieb am 13.04.2022 um
17:38
>> in
>> Nachricht :
>> > On Di, 12.04.22 14:38, Elbek Mamajonov (emm.boxin...@gmail.com) wrote:
>> >
>> >> On graph I have mmcblk.device taking 1.624s. From the point that
>> >> this partition is where my rootfs lies, and systems does remounting
>> >> of rootfs, I did look up for systemd‑remount‑fs.service, it took
>> >> 231ms, and systemd‑udev‑trigger.service, as you suggested, it took
>> >> 517ms to become active. But even after systemd‑udev‑trigger.service
>> >> becomes active there is about 800ms for mmcblk.device to become
>> >> active. Are those services related to the activation of the
>> >> mmcblk.device? Can I consider those 231ms and 517ms as a part of the
>> >> activation time of the mmcblk.device? How can I debug udev in this
>> >> case?
>> >
>> > "systemd‑udev‑trigger.service" might take a while before it completes,
>> > since it triggers basically all devices in the system.
>> >
>> > It might be worth triggering block devices first. With upcoming v251
>>
>> What is the expected benefit? On bigger servers with hundreds of disks
this
>> may take longest.
> 
> There are a myriad of devices on current systems. Traditionally, we
> trigger them at boot in alphabetical order by their sysfs path (more
> or less that is). Only once triggered subsystems waiting for them will
> see the devices. Since at boot typically the most waited for devices
> are block devices it's thus benefical to trigger them first, as this
> unblocks a major part of the rest of the boot process.
> 
> Or in other words: nothing really "waits" for your mouse to show up in
> the device table. Everyting waits for your root block device to show
> up. Hence trigger the root block device first, and the mouse later.

Hi!

I agree, but (how) can you trigger only the root block device?
Apr 01 08:46:25 h16 kernel: sd 0:2:0:0: [sda] 467664896 512-byte logical
blocks: (239 GB/223 GiB)
...
Apr 01 08:46:33 h16 kernel: sd 3:0:7:2: [sdda] 524288 512-byte logical blocks:
(268 MB/256 MiB)
...
That's 8 seconds to discover the devices

Apr 01 08:47:04 h16 kernel: sd 2:0:5:2: alua: port group 01 state A preferred
supports tolusnA

And another 30 seconds until multipath has settled.

Apr 01 08:47:04 h16 systemd[1]: Reached target System Initialization.

Regards,
Ulrich

> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] Re: [systemd‑devel] Waiting for (transient) hostname configuration

[Ulrich Windl]

>>> Lennart Poettering  schrieb am 19.04.2022 um 11:41
in
Nachricht :
> On Di, 19.04.22 11:14, Alessio Igor Bogani (alessio.bog...@elettra.eu)
wrote:
> 
>> Dear systemd developers,
>>
>> Sorry for my very bad english.
> 
> Not bad at all.
> 
>> We use the hostname provided by DHCP to start the right subset of our
>> applications on each of our machines (in other words the transient
>> hostname is used as predictable machine identifier).
>>
>> Make it works on the ancient SysV is straightforward but I can't find
>> a (reasonable) way to achieve the same goal with systemd. We already
>> use both "Requires" and "After" with "network‑online.target
>> nss‑lookup.target" into our units but it isn't enough. When our units
>> start they obtain "localhost" from %H specifier. We don't find a way
>> to wait for (transient) hostname to be configured. The only solution
>> that it seems to work is to replace our binary in ExecStart with a
>> script which make a busy‑loop until `hostname` output doesn't differ
>> from "localhost".
>>
>> Is there a better way?
> 
> First of all: % specifiers (including %H) are resolved at the moment
> the unit files are *loaded*, i.e. typically at earliest boot, long

That's quite some surprise: I thought they are evaluated when they are
executed (as in shell scripts).
Is there a way to "reload" a specific unit file?

> before they are actually *started*. Thus, if you change the hostname
> and want %H to resolve to that, you need to issue a reload at the
> right moment, i.e. issue `systemctl daemon‑reload` once the hostname
> is acquired.
> 
> If your hostname is set via DHCP you need to wait for DHCP to be
> acquired. How that's done, depends on the networking solution you
> use. If you use systemd‑networkd, then the
> sytemd‑network‑wait‑online.service is what you want to use. If you
> enable that then network‑online.target should be the point where DHCP
> is acquired and thus also the hostname in effect.
> 
> Lennart
> 
> ‑‑
> Lennart Poettering, Berlin

[systemd-devel] Antw: [EXT] Re: Waiting for (transient) hostname configuration

[Ulrich Windl]

>>> Alessio Igor Bogani  schrieb am 20.04.2022 um 
>>> 22:09
in Nachricht
:

...
> The %H specifier in the commented ExecStart always returns
> "localhost". The following ExecStart is my workaround to have the

I wonder whether a future version could interpret "\%H" as "substitute %H when 
it's used, not when it was parsed the first time" ("%%" is in use already).

> hostname provided by DHCP (in the first version it was a while loop
> but a sleep works anyway and make things less convoluted).
> 
> Ciao,
> Alessio

[systemd-devel] Antw: [EXT] Re: Disallowing fingerprint authentication if pam_systemd_home.so needs a password

[Ulrich Windl]

>>> juice  schrieb am 25.04.2022 um 17:03 in Nachricht
<4cbf03ca-7a0a-4dbe-ad00-c6f3938ff...@swagman.org>:

> 
> 25. huhtikuuta 2022 16.39.56 GMT+03:00 Benjamin Berg 
> kirjoitti:
>>On Mon, 2022-04-25 at 13:28 +0200, Lennart Poettering wrote:
>>> 
>>> Hmm, not sure I follow? I don't know how fingerprint flow of control
>>> is. Is this about authentication-by-fingerprint? Or already about
>>> user-selection-by-fingerprint?
>>
>>I was just thinking of authentication-by-fingerprint. Though I don't
>>think it makes a big difference here.
>>
> 
> Using fingerprint for *authentication* is totally broken concept which 
> should never be allowed.

Why? Is a PIN any better?

> Fingerprints are *userid*, never *password*.
> 
> We leave our fingerprints lying around all the time, and given malicious 
> enough adversaries they might as well take our fingers too. (I sure would 
> like to avoid that possibility!!)

So you are saying users leave themselves lying around everywhere? ;-)

> 
> Fingerprints can be used on place of username, that is OK and does not 
> present similar risks.

Fingerprints are mote than a userid IMHO.

> 
>   - Juice -

[systemd-devel] Antw: [systemd‑devel] Antw: [EXT] Re: Disallowing fingerprint authentication if pam_systemd_home.so needs a password

[Ulrich Windl]

>>> "Ulrich Windl"  schrieb am 26.04.2022 um
08:41 in Nachricht <6267942302a100049...@gwsmtp.uni-regensburg.de>:
>>>> juice  schrieb am 25.04.2022 um 17:03 in Nachricht

...
>> Fingerprints can be used on place of username, that is OK and does not 
>> present similar risks.
> 
> Fingerprints are mote than a userid IMHO.

s/mote/more/ # sorry

Re: [systemd-devel] Antw: [EXT] Re: Disallowing fingerprint authentication if pam_systemd_home.so needs a password

[Ulrich Windl]

>>> juice  schrieb am 26.04.2022 um 09:11 in Nachricht
<2a780de8-efb2-749b-de43-62978958f...@swagman.org>:
> On 4/26/22 09:41, Ulrich Windl wrote:
>>>
>>> Using fingerprint for *authentication* is totally broken concept which
>>> should never be allowed.
>> Why? Is a PIN any better?
> 
> PIN is much better. You will not be leaving your PIN to any drinking 
> glass you handle or to doorhandles that you open. People leave 
> fingerprints all around the place and it has been repeatedly 
> demonstrated that fingerprints can be easily extracted and replicated to 
> silicone fingers which can be used to fool fingerprint readers.
> 
> 
>>> We leave our fingerprints lying around all the time, and given malicious
>>> enough adversaries they might as well take our fingers too. (I sure would
>>> like to avoid that possibility!!)
>> So you are saying users leave themselves lying around everywhere? ;-)
> 
> People leave fingerprints. Fingerprints can be used to open devices 
> locked by fingerprint. There is also a risk that someone may kill you 
> and cut off your finger.
> 
> 
>>> Fingerprints can be used on place of username, that is OK and does not
>>> present similar risks.
>> Fingerprints are mote than a userid IMHO.
> 
> Fingerprint is exactly that, it is user identification. The police have 
> been using fingerprints now 130 years for identifying people. Some 
> misguided fools have been trying to use fingerprints as substitute for 
> phone unlock PIN for maybe 10 years or so.

Actually I think using a fingerprint to unlock the phone is much safer than 
using a short pin or some swipe pattern:
If someone watches me to unlock my phone using my finger in some public 
transport, he'll have trouble to unlock it if the phone is stolen, but you can 
easily watch the short pins or swipe patterns from the distance.

Regards,
Ulrich

> 
>- juice -