Re: [systemd-devel] [SPAM]Re: Mount units with After=autofs.service cause ordering cycles

2019-10-31 Thread John Florian

On 10/31/19 2:59 PM, Lennart Poettering wrote:

On Do, 31.10.19 14:09, John Florian (jflor...@doubledog.org) wrote:


# /etc/systemd/system/var-www-pub.mount
[Unit]
Description=mount /pub served via httpd
Requires=autofs.service
After=autofs.service

[Mount]
What=/mnt/pub
Where=/var/www/pub
Options=bind,context=system_u:object_r:httpd_sys_content_t

[Install]
WantedBy=multi-user.target

~~~

The above worked for a long time, but once again a `dnf upgrade` seems to
have broken things because now I have a ordering cycle that systemd must
break.  Since I haven't changed my mount units, my ability to mesh with
those shipped by the OS proves fragile. I'm deliberately avoiding too much
detail here because it would seem that there should be a relatively simple
solution to this general sort of task -- I just can't seem to discover it.
Any recommendations that don't involve an entirely different approach?

What precisely is the ordering cycle you are seeing? It's usually
dumped along with the log message.

systemd[1]: local-fs.target: Found ordering cycle on var-www-pub.mount/start
systemd[1]: local-fs.target: Found dependency on autofs.service/start
systemd[1]: local-fs.target: Found dependency on rpc-statd.service/start
systemd[1]: local-fs.target: Found dependency on network-online.target/start
systemd[1]: local-fs.target: Found dependency on network.target/start
systemd[1]: local-fs.target: Found dependency on
NetworkManager.service/start
systemd[1]: local-fs.target: Found dependency on sysinit.target/start
systemd[1]: local-fs.target: Found dependency on
systemd-update-done.service/start
systemd[1]: local-fs.target: Found dependency on local-fs.target/start
systemd[1]: local-fs.target: Job var-www-pub.mount/start deleted to break
ordering cycle starting with local-fs.target/start

The ordering dep between local-fs.target and var-ww-pub.mount is what
you have to get rid of to remove the cycle. Set:

…
[Unit]
DefaultDependencies=no
Conflicts=umount.target
Before=umount.target
…
[Install]
WantedBy=remote-fs.target
…

i.e. make this a dep of remote-fs.target, not the implicit
local-fs.target, so that we don#t pull it in early boot, but only
during late boot, before remote-fs.target.


Thanks Lennart!  That did the trick.  I and others I know have knocked 
heads on this one several times and somehow never came to this 
conclusion.  It makes sense now that I see it, however.   Maybe 
local-fs.target should have stood out to me, but I think it was mostly 
accepted since if you follow all deps far enough, you'll eventually 
cover (most?) everything.


I think this just means I need to use `systemctl show` more even though 
`systemctl cat` is so much easier to digest for what I think I need to 
know.  Abstracting the default deps is both good in expression but also 
difficult in comprehension.  I wish there was something "in between", 
but I don't even know how to define what that means.  Maybe just 
grouping all the settings from `show` somehow, e.g.: ordering, deps, 
etc. or maybe by unit type: unit, exec, mount, etc.


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] [SPAM]Re: Mount units with After=autofs.service cause ordering cycles

2019-10-31 Thread John Florian

On 10/31/19 1:08 PM, Lennart Poettering wrote:

On Mo, 14.10.19 16:23, John Florian (j...@doubledog.org) wrote:


So, I much prefer the expressiveness of systemd's mount units to the naive
era of /etc/fstab, but I've found one situation where I seem to always get
stuck and am never able to find a reliable solution that survives OS (Fedora
& CentOS) updates.  I have a NFS filesystem mounted by autofs at /pub that
needs to be bind mounted in various places such as /var/www/pub and
/var/ftp/pub. So I create a unit that looks like:

~~~

# /etc/systemd/system/var-www-pub.mount
[Unit]
Description=mount /pub served via httpd
Requires=autofs.service
After=autofs.service

[Mount]
What=/mnt/pub
Where=/var/www/pub
Options=bind,context=system_u:object_r:httpd_sys_content_t

[Install]
WantedBy=multi-user.target

~~~

The above worked for a long time, but once again a `dnf upgrade` seems to
have broken things because now I have a ordering cycle that systemd must
break.  Since I haven't changed my mount units, my ability to mesh with
those shipped by the OS proves fragile. I'm deliberately avoiding too much
detail here because it would seem that there should be a relatively simple
solution to this general sort of task -- I just can't seem to discover it.
Any recommendations that don't involve an entirely different approach?

What precisely is the ordering cycle you are seeing? It's usually
dumped along with the log message.


systemd[1]: local-fs.target: Found ordering cycle on var-www-pub.mount/start
systemd[1]: local-fs.target: Found dependency on autofs.service/start
systemd[1]: local-fs.target: Found dependency on rpc-statd.service/start
systemd[1]: local-fs.target: Found dependency on network-online.target/start
systemd[1]: local-fs.target: Found dependency on network.target/start
systemd[1]: local-fs.target: Found dependency on 
NetworkManager.service/start

systemd[1]: local-fs.target: Found dependency on sysinit.target/start
systemd[1]: local-fs.target: Found dependency on 
systemd-update-done.service/start

systemd[1]: local-fs.target: Found dependency on local-fs.target/start
systemd[1]: local-fs.target: Job var-www-pub.mount/start deleted to 
break ordering cycle starting with local-fs.target/start


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[systemd-devel] Mount units with After=autofs.service cause ordering cycles

2019-10-14 Thread John Florian
So, I much prefer the expressiveness of systemd's mount units to the 
naive era of /etc/fstab, but I've found one situation where I seem to 
always get stuck and am never able to find a reliable solution that 
survives OS (Fedora & CentOS) updates.  I have a NFS filesystem mounted 
by autofs at /pub that needs to be bind mounted in various places such 
as /var/www/pub and /var/ftp/pub. So I create a unit that looks like:


~~~

# /etc/systemd/system/var-www-pub.mount
[Unit]
Description=mount /pub served via httpd
Requires=autofs.service
After=autofs.service

[Mount]
What=/mnt/pub
Where=/var/www/pub
Options=bind,context=system_u:object_r:httpd_sys_content_t

[Install]
WantedBy=multi-user.target

~~~

The above worked for a long time, but once again a `dnf upgrade` seems 
to have broken things because now I have a ordering cycle that systemd 
must break.  Since I haven't changed my mount units, my ability to mesh 
with those shipped by the OS proves fragile. I'm deliberately avoiding 
too much detail here because it would seem that there should be a 
relatively simple solution to this general sort of task -- I just can't 
seem to discover it.  Any recommendations that don't involve an entirely 
different approach?


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] `Found ordering cycle on _SERVICE_` loops due to `local-fs.target` with `Requires` instead of `Wants` for mounts generated from `/etc/fstab`

2018-04-27 Thread John Florian

On 2018-04-27 03:56, Jérémy Rosen wrote:

Ok, there are a couple of tools that have not be mentionned..

"systemd-analyze verify " will check all sorts of stuff on a 
service, including checking recursively the deps and finding ordering 
cycles



systemctl list-dependencies can show ordering/req dependencies, up or 
down, recursively or not


to be completely honest, I never use dot for cycles I sometime 
read/grep the .dot file itself, but I never generate the image...


systemd-analyze is my favorite tool for that sort of problems

On 27/04/2018 06:01, Andrei Borzenkov wrote:

Well, you could use "systemctl show -p Id -p After -p Before" but the
problem is to find the correct unit set. You really need something that
simulates transaction and evaluates dependencies like systemd does it.

Note that systemd should print offending cycle when it hits it:

апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
ordering cycle on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on bar.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on baz.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Found
dependency on foo.service/start
апр 27 06:54:14 bor-Latitude-E5450 systemd[1582]: foo.service: Breaking
ordering cycle by deleting job baz.service/start


Well these are good things to know.  I can't recall if I ever tried to 
use them when faced with a loop.  I'd guess that I tried 
`systemd-analyze verify` as it sort of jumps off the man page whereas 
`systemctl list-dependencies` is kind of buried in the details.



It would be helpful it if additionally printed kind of dependency (like
foo.service is After bar.service) because this is by far not obvious for
average user.


Maybe one really helpful immediate change would be to have systemd log 
some hints when it breaks a loop to suggest a few commands that might be 
run to help understand the situation more clearly.

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] `Found ordering cycle on _SERVICE_` loops due to `local-fs.target` with `Requires` instead of `Wants` for mounts generated from `/etc/fstab`

2018-04-26 Thread John Florian

On 2018-04-26 16:04, John Florian wrote:

On 2018-04-25 04:53, Lennart Poettering wrote:

There have been requests in improving the cycle breaking algorithm,
but not much has been done in this area, since it's not clear what can
be done. Ultimately it's just polishing a broken situation, and
the better way out is to fix things properly, i.e. just correct the
cycle in the dependencies in the first place.


Having been the author of numerous custom services that are intended 
to wedge into those provided by systemd/Fedora I've faced resolving 
these several times and I've never felt all that competent at the 
task.  Lennart, you were immensely helpful on one occasion by pointing 
me to `systemctl show FOO` but even then hunting the problem down was 
far from simple (for me).  I've done the dot/graphviz thing and found 
it just as useless to me as when I've tried applying it to Puppet's 
ordering/dependency looping. I'm not blaming those tools (or systemd) 
because I'm well aware much of the problem is my inability to use them 
effectively.  The graphs always seem to be overly simple and revealing 
no problem or overly detailed and obscuring the problem.  Compound 
that with "an arrow pointing this way means what exactly?"


Is there anything else that could be done to make hunting these loops 
down easier?  Is there an example of any product that has a similar 
situation where they excel at helping the developer?  Or are we 
already there and I just need more practice?  I'm sure part of my 
struggle is just not encountering these very regularly and integrating 
into much that I'm only partly familiar with, but the result is the 
same, trepidation and loathing.


Since I hate to grumble w/o so much as offering any possible 
suggestions... It seems like it might be helpful to have something like 
`systemctl show` but only dumping the properties used for ordering and 
dependencies.  Ideally, only ordering OR dependencies as the case 
warrants and for all units involved, but only those units.

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] `Found ordering cycle on _SERVICE_` loops due to `local-fs.target` with `Requires` instead of `Wants` for mounts generated from `/etc/fstab`

2018-04-26 Thread John Florian

On 2018-04-25 04:53, Lennart Poettering wrote:

There have been requests in improving the cycle breaking algorithm,
but not much has been done in this area, since it's not clear what can
be done. Ultimately it's just polishing a broken situation, and
the better way out is to fix things properly, i.e. just correct the
cycle in the dependencies in the first place.


Having been the author of numerous custom services that are intended to 
wedge into those provided by systemd/Fedora I've faced resolving these 
several times and I've never felt all that competent at the task.  
Lennart, you were immensely helpful on one occasion by pointing me to 
`systemctl show FOO` but even then hunting the problem down was far from 
simple (for me).  I've done the dot/graphviz thing and found it just as 
useless to me as when I've tried applying it to Puppet's 
ordering/dependency looping.  I'm not blaming those tools (or systemd) 
because I'm well aware much of the problem is my inability to use them 
effectively.  The graphs always seem to be overly simple and revealing 
no problem or overly detailed and obscuring the problem.  Compound that 
with "an arrow pointing this way means what exactly?"


Is there anything else that could be done to make hunting these loops 
down easier?  Is there an example of any product that has a similar 
situation where they excel at helping the developer?  Or are we already 
there and I just need more practice?  I'm sure part of my struggle is 
just not encountering these very regularly and integrating into much 
that I'm only partly familiar with, but the result is the same, 
trepidation and loathing.

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] About supporting relative values (e.g. 50%) in configuration

2018-03-22 Thread John Florian

On 2018-03-09 03:13, Zbigniew Jędrzejewski-Szmek wrote:

On Thu, Mar 08, 2018 at 03:48:54PM +0800, ChenQi wrote:

Hi All,

I'd like to know if systemd community thinks that adding support for
relative values in configuration is a good idea.

I found a related patch and discussion in 
https://lists.freedesktop.org/archives/systemd-devel/2015-May/032191.html.

Quoting from the comments:
"""

I'd always keep our basic structures as generic as possible, and as
close to the way CPUs work. Hence: store things as fixed-point
32bit/32bit internally, but make it configurable in percent as
user-pointing UI.

"""

According to the comments, it seems that adding such support is acceptable?

I' sending out this email because I want to check with the community
before I start to investigate more.

Hi,

yes, a patch to allow relative values would be very much welcome.


Indeed, very much!  Please see my bug[0] for some history on this.

[0] https://bugzilla.redhat.com/show_bug.cgi?id=874631
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


[systemd-devel] Retaining boot messages on (near-)stateless systems

2017-05-11 Thread John Florian
I maintain a derivative of Fedora Live (built using lorax) that gets
deployed on hundreds of systems, far more than my team has the man-
power to keep a watchful eye.  Occasionally we are notified of a
problematic node and often it would be helpful to see the full journal
for say, the first 15 minutes of run-time in which the node transforms
itself from a generic appliance to a specific role.  Unfortunately, in
many cases we are notified too late and journald has tail-trimmed the
logs we desire most.

Reviewing journald.conf(5) I don't see options to achieve what I want
directly so my thought is to create a service and timer with
OnBootSec=15m to duplicate the journal over to persistent storage,
which would then require non-standard techniques for viewing but at
least the availability would be guaranteed.

Is there a better way to achieve this?  If not, what's the best way to
duplicate the journal data?   Simply cp -a /var/log/journal/ ... or use
journalctl to dump or export it somehow?

I'll also add that I also wish to retain higher priority messages for a
longer period, though for the first ~15m I want everything, DEBUG
included.  That leads me to think about a 2nd logger (journald,
ideally) which had messages forwarded to it, but with different
retention characteristics.___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?

2017-03-30 Thread John Florian
On Thu, 2017-03-30 at 16:04 +0200, Lennart Poettering wrote:
> On Thu, 30.03.17 09:53, John Florian (j...@doubledog.org) wrote:
> 
> > On Thu, 2017-03-30 at 15:32 +0200, Lennart Poettering wrote:
> > > On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote:
> > > 
> > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at
> > > > job
> > > > sshd.service/start conflicted_by=no
> > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at
> > > > job
> > > > sshd.service/stop conflicted_by=yes
> > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing
> > > > conflicting
> > > > jobs sshd.service/start,sshd.service/stop by deleting job
> > > > sshd.service/start
> > > 
> > > So this suggests that you have both a job that pulls in
> > > sshd.service
> > > and one that conflicts with it in your initial transaction. Try
> > > figuring out what that is...
> > > 
> > > (hint: use "systemctl show sshd.service" for that, and look for
> > > Conflicts= and ConlictedBy=)
> > > 
> > 
> > That seems like a reasonable theory, but given the following I
> > can't
> > make any sense of it.
> > 
> > # journalctl -b | grep -i conflict
> > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
> > sshd.service/start conflicted_by=no
> > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
> > sshd.service/stop conflicted_by=yes
> > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Fixing
> > conflicting
> > jobs sshd.service/start,sshd.service/stop by deleting job
> > sshd.service/start
> > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking
> > at
> > job sshd-keygen.target/start conflicted_by=no
> > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking
> > at
> > job sshd-keygen.target/stop conflicted_by=no
> > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Fixing
> > conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop
> > by
> > deleting job sshd-keygen.target/stop
> > 
> > # systemctl show sshd.service | grep -i conflict
> > Conflicts=shutdown.target
> > ConflictedBy=sshd.socket
> 
> Note that you can use sshd on Fedora in socket-activated mode or
> non-socket-activated mode. In socket-activated mode, sshd.socket is
> started at boot, and for each connection one sshd@.service instance
> is
> created. sshd.service otoh is only used in the non-socket-activated
> mode, and in that case is the singleton service processing all
> connections...

Yup, I was aware of that of neat feature...

> My educated guess is that you somehow ended up pull in both
> sshd.service and sshd.socket, but you cannot really do that, you have
> to make a choice between them, as there's only one port 22, and it's
> either systemd that listens on it or sshd.

Bingo!  That was just what I needed to put my train of thought on a
different set of tracks.  systemctl status sshd.socket showed that it
was enabled and the vendor preset was also enabled.  Wait...
what   My package that contains the sshd-persist-keys.service and
sshd-restore-keys.service units also provides a file with unit presets
and in there I had "enabled sshd.*".  That was fine back in F21, but no
so with F25 so I replaced that with a "enable sshd.service" and
"disabled sshd.socket" and all is well now.   It's funny, the problem
seems so obvious now yet only an hour ago I was totally blind to the
problem.

Thank you so much for your time and wisdom.___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?

2017-03-30 Thread John Florian
On Thu, 2017-03-30 at 15:32 +0200, Lennart Poettering wrote:
> On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote:
> 
> > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
> > sshd.service/start conflicted_by=no
> > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
> > sshd.service/stop conflicted_by=yes
> > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing
> > conflicting
> > jobs sshd.service/start,sshd.service/stop by deleting job
> > sshd.service/start
> 
> So this suggests that you have both a job that pulls in sshd.service
> and one that conflicts with it in your initial transaction. Try
> figuring out what that is...
> 
> (hint: use "systemctl show sshd.service" for that, and look for
> Conflicts= and ConlictedBy=)
> 

That seems like a reasonable theory, but given the following I can't
make any sense of it.

# journalctl -b | grep -i conflict
Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
sshd.service/start conflicted_by=no
Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
sshd.service/stop conflicted_by=yes
Mar 30 09:35:57 localhost systemd[1]: sshd.service: Fixing conflicting
jobs sshd.service/start,sshd.service/stop by deleting job
sshd.service/start
Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at
job sshd-keygen.target/start conflicted_by=no
Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at
job sshd-keygen.target/stop conflicted_by=no
Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Fixing
conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop by
deleting job sshd-keygen.target/stop

# systemctl show sshd.service | grep -i conflict
Conflicts=shutdown.target
ConflictedBy=sshd.socket


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?

2017-03-23 Thread John Florian
On Thu, 2017-03-23 at 06:38 +0300, Andrei Borzenkov wrote:
> 22.03.2017 23:47, John Florian пишет:
> > I build an mostly-stateless appliance OS derived from Fedora (25
> > ATM)
> > and have several custom units to make it all possible.  My units
> > had
> > worked great with F21, but are now giving me problems with
> > F25.  One
> > pair of the custom units do some trickery to relocate sshd host
> > keys
> > from /etc/ssh to an alternate location that provides persistence:
> > 
> > 
> 
> Please make full log available as well as actual unit definitions
> that
> are not started.
> 

Okay, attached are a complete boot.log (journalctl > boot.log) as well
as the most relevant units. Please note that while I need a solution,
it's just as important I understand HOW to resolve this kind of issue.
I've struggled through similar problems in the past and I never feel
adept at the process, like I'm not using a fundamental tool for the job
and instead relying more on trial and error.

While there's no doubt a lot of change between F21 and F25, the one
thing that forced a change on my part was systemd transitioning from a
single sshd-keygen.service to a templated sshd-keygen@.service.  For me
that meant the following change:

diff --git lib/systemd/sshd-persist-keys.service lib/systemd/sshd-
persist-keys.service
index d3508a8..7500b68 100644
--- lib/systemd/sshd-persist-keys.service
+++ lib/systemd/sshd-persist-keys.service
@@ -1,8 +1,8 @@
 [Unit]
 Description=OpenSSH server - persist volatile keys for the AOS
-After=sshd-keygen.service
+After=sshd-keygen.target
 Before=sshd.service
-Wants=sshd-keygen.service
+Wants=sshd-keygen.target

 [Service]
 ExecStart=/usr/sbin/sshd-persist-keys
diff --git lib/systemd/sshd-restore-keys.service lib/systemd/sshd-
restore-keys.service
index 475cede..85ac3bc 100644
--- lib/systemd/sshd-restore-keys.service
+++ lib/systemd/sshd-restore-keys.service
@@ -1,7 +1,7 @@
 [Unit]
 Description=OpenSSH server - restore persisted keys for the AOS
 After=aos-storage-init.service
-Before=sshd-keygen.service
+Before=sshd-keygen@rsa.service sshd-keygen@ecdsa.service sshd-keygen@e
d25519.service

 [Service]
 ExecStart=/usr/sbin/sshd-restore-keys


I'd love to know of a better way for my sshd-restore-kes.service to be
coded such that it's unnecessary to list individual key types, but my
experience has led me to believe that targets only define a point in
time when a collection of units have completed and that there's no
equivalent to a point in time prior to any of them having begun.  Thus
I found I couldn't use Before=sshd-keygen.target because keys were
being generated and restored concurrently rather than serially
(restored if possible and then the missing generated as needed).

sshd-startup-failure.tar.gz
Description: application/compressed-tar
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel


[systemd-devel] How do I easily resolve conflicting jobs in custom units?

2017-03-22 Thread John Florian
I build an mostly-stateless appliance OS derived from Fedora (25 ATM)
and have several custom units to make it all possible.  My units had
worked great with F21, but are now giving me problems with F25.  One
pair of the custom units do some trickery to relocate sshd host keys
from /etc/ssh to an alternate location that provides persistence:

$ systemctl cat sshd-persist-keys.service
# /usr/lib/systemd/system/sshd-persist-keys.service
[Unit]
Description=OpenSSH server - persist volatile keys for the AOS
After=sshd-keygen.target
Before=sshd.service
Wants=sshd-keygen.target

[Service]
ExecStart=/usr/sbin/sshd-persist-keys
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target


$ systemctl cat sshd-restore-keys.service
# /usr/lib/systemd/system/sshd-restore-keys.service
[Unit]
Description=OpenSSH server - restore persisted keys for the AOS
After=aos-storage-init.service
Before=sshd-keygen@rsa.service sshd-keygen@ecdsa.service sshd-keygen@ed
25519.service

[Service]
ExecStart=/usr/sbin/sshd-restore-keys
Type=oneshot
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target


I found that on some boots, sshd wasn't getting started.  With the help
of booting with systemd.log_level=debug I learned:

$ sudo journalctl | grep conflict
Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
sshd.service/start conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
sshd.service/stop conflicted_by=yes
Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing conflicting
jobs sshd.service/start,sshd.service/stop by deleting job
sshd.service/start
Mar 22 16:11:42 localhost systemd[1]: sshd.socket: Looking at job
sshd.socket/stop conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd.socket: Looking at job
sshd.socket/start conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd.socket: Fixing conflicting
jobs sshd.socket/stop,sshd.socket/start by deleting job
sshd.socket/stop
Mar 22 16:11:42 localhost systemd[1]: sshd-keygen.target: Looking at
job sshd-keygen.target/start conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd-keygen.target: Looking at
job sshd-keygen.target/stop conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd-keygen.target: Fixing
conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop by
deleting job sshd-keygen.target/stop


I'm sure systemd-analyze is my friend here, but I've not found an
effective way to grok the problem.  s-a dot either gives me too much to
comprehend or not enough to make evident the problem.  I've tried
feeding it different PATTERNS like: none at all, sshd.service, ssh-
keygen.target, sshd-{persist,restore}-keys.service and more.  I either
see a whole forest or just a few trees that seem to be getting along
happily together.

What advice can you suggest?___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel