Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?
On Thu, 2017-03-30 at 16:04 +0200, Lennart Poettering wrote: > On Thu, 30.03.17 09:53, John Florian (j...@doubledog.org) wrote: > > > On Thu, 2017-03-30 at 15:32 +0200, Lennart Poettering wrote: > > > On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote: > > > > > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at > > > > job > > > > sshd.service/start conflicted_by=no > > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at > > > > job > > > > sshd.service/stop conflicted_by=yes > > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing > > > > conflicting > > > > jobs sshd.service/start,sshd.service/stop by deleting job > > > > sshd.service/start > > > > > > So this suggests that you have both a job that pulls in > > > sshd.service > > > and one that conflicts with it in your initial transaction. Try > > > figuring out what that is... > > > > > > (hint: use "systemctl show sshd.service" for that, and look for > > > Conflicts= and ConlictedBy=) > > > > > > > That seems like a reasonable theory, but given the following I > > can't > > make any sense of it. > > > > # journalctl -b | grep -i conflict > > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job > > sshd.service/start conflicted_by=no > > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job > > sshd.service/stop conflicted_by=yes > > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Fixing > > conflicting > > jobs sshd.service/start,sshd.service/stop by deleting job > > sshd.service/start > > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking > > at > > job sshd-keygen.target/start conflicted_by=no > > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking > > at > > job sshd-keygen.target/stop conflicted_by=no > > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Fixing > > conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop > > by > > deleting job sshd-keygen.target/stop > > > > # systemctl show sshd.service | grep -i conflict > > Conflicts=shutdown.target > > ConflictedBy=sshd.socket > > Note that you can use sshd on Fedora in socket-activated mode or > non-socket-activated mode. In socket-activated mode, sshd.socket is > started at boot, and for each connection one sshd@.service instance > is > created. sshd.service otoh is only used in the non-socket-activated > mode, and in that case is the singleton service processing all > connections... Yup, I was aware of that of neat feature... > My educated guess is that you somehow ended up pull in both > sshd.service and sshd.socket, but you cannot really do that, you have > to make a choice between them, as there's only one port 22, and it's > either systemd that listens on it or sshd. Bingo! That was just what I needed to put my train of thought on a different set of tracks. systemctl status sshd.socket showed that it was enabled and the vendor preset was also enabled. Wait... what My package that contains the sshd-persist-keys.service and sshd-restore-keys.service units also provides a file with unit presets and in there I had "enabled sshd.*". That was fine back in F21, but no so with F25 so I replaced that with a "enable sshd.service" and "disabled sshd.socket" and all is well now. It's funny, the problem seems so obvious now yet only an hour ago I was totally blind to the problem. Thank you so much for your time and wisdom.___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?
On Thu, 30.03.17 09:53, John Florian (j...@doubledog.org) wrote: > On Thu, 2017-03-30 at 15:32 +0200, Lennart Poettering wrote: > > On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote: > > > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job > > > sshd.service/start conflicted_by=no > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job > > > sshd.service/stop conflicted_by=yes > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing > > > conflicting > > > jobs sshd.service/start,sshd.service/stop by deleting job > > > sshd.service/start > > > > So this suggests that you have both a job that pulls in sshd.service > > and one that conflicts with it in your initial transaction. Try > > figuring out what that is... > > > > (hint: use "systemctl show sshd.service" for that, and look for > > Conflicts= and ConlictedBy=) > > > > That seems like a reasonable theory, but given the following I can't > make any sense of it. > > # journalctl -b | grep -i conflict > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job > sshd.service/start conflicted_by=no > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job > sshd.service/stop conflicted_by=yes > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Fixing conflicting > jobs sshd.service/start,sshd.service/stop by deleting job > sshd.service/start > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at > job sshd-keygen.target/start conflicted_by=no > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at > job sshd-keygen.target/stop conflicted_by=no > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Fixing > conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop by > deleting job sshd-keygen.target/stop > > # systemctl show sshd.service | grep -i conflict > Conflicts=shutdown.target > ConflictedBy=sshd.socket Note that you can use sshd on Fedora in socket-activated mode or non-socket-activated mode. In socket-activated mode, sshd.socket is started at boot, and for each connection one sshd@.service instance is created. sshd.service otoh is only used in the non-socket-activated mode, and in that case is the singleton service processing all connections... My educated guess is that you somehow ended up pull in both sshd.service and sshd.socket, but you cannot really do that, you have to make a choice between them, as there's only one port 22, and it's either systemd that listens on it or sshd. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?
On Thu, 2017-03-30 at 15:32 +0200, Lennart Poettering wrote: > On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote: > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job > > sshd.service/start conflicted_by=no > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job > > sshd.service/stop conflicted_by=yes > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing > > conflicting > > jobs sshd.service/start,sshd.service/stop by deleting job > > sshd.service/start > > So this suggests that you have both a job that pulls in sshd.service > and one that conflicts with it in your initial transaction. Try > figuring out what that is... > > (hint: use "systemctl show sshd.service" for that, and look for > Conflicts= and ConlictedBy=) > That seems like a reasonable theory, but given the following I can't make any sense of it. # journalctl -b | grep -i conflict Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job sshd.service/start conflicted_by=no Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job sshd.service/stop conflicted_by=yes Mar 30 09:35:57 localhost systemd[1]: sshd.service: Fixing conflicting jobs sshd.service/start,sshd.service/stop by deleting job sshd.service/start Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at job sshd-keygen.target/start conflicted_by=no Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at job sshd-keygen.target/stop conflicted_by=no Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Fixing conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop by deleting job sshd-keygen.target/stop # systemctl show sshd.service | grep -i conflict Conflicts=shutdown.target ConflictedBy=sshd.socket ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?
On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote: > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job > sshd.service/start conflicted_by=no > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job > sshd.service/stop conflicted_by=yes > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing conflicting > jobs sshd.service/start,sshd.service/stop by deleting job > sshd.service/start So this suggests that you have both a job that pulls in sshd.service and one that conflicts with it in your initial transaction. Try figuring out what that is... (hint: use "systemctl show sshd.service" for that, and look for Conflicts= and ConlictedBy=) Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?
23.03.2017 18:44, John Florian пишет: > On Thu, 2017-03-23 at 06:38 +0300, Andrei Borzenkov wrote: >> 22.03.2017 23:47, John Florian пишет: >>> I build an mostly-stateless appliance OS derived from Fedora (25 >>> ATM) >>> and have several custom units to make it all possible. My units >>> had >>> worked great with F21, but are now giving me problems with >>> F25. One >>> pair of the custom units do some trickery to relocate sshd host >>> keys >>> from /etc/ssh to an alternate location that provides persistence: >>> >>> >> >> Please make full log available as well as actual unit definitions >> that >> are not started. >> > > Okay, attached are a complete boot.log (journalctl > boot.log) as well > as the most relevant units. Please note that while I need a solution, > it's just as important I understand HOW to resolve this kind of issue. > I've struggled through similar problems in the past and I never feel > adept at the process, like I'm not using a fundamental tool for the job > and instead relying more on trial and error. > > While there's no doubt a lot of change between F21 and F25, the one I do not use Fedora. You still did not show definitions of units that have problem: sshd.service, sshd-keygen.target, sshd.socket. Providing output of systemctl show sshd.service sshd-keygen.target sshd.socket is probably better as it dumps full information. > thing that forced a change on my part was systemd transitioning from a > single sshd-keygen.service to a templated sshd-keygen@.service. For me > that meant the following change: > > diff --git lib/systemd/sshd-persist-keys.service lib/systemd/sshd- > persist-keys.service > index d3508a8..7500b68 100644 > --- lib/systemd/sshd-persist-keys.service > +++ lib/systemd/sshd-persist-keys.service > @@ -1,8 +1,8 @@ > [Unit] > Description=OpenSSH server - persist volatile keys for the AOS > -After=sshd-keygen.service > +After=sshd-keygen.target > Before=sshd.service > -Wants=sshd-keygen.service > +Wants=sshd-keygen.target > > [Service] > ExecStart=/usr/sbin/sshd-persist-keys > diff --git lib/systemd/sshd-restore-keys.service lib/systemd/sshd- > restore-keys.service > index 475cede..85ac3bc 100644 > --- lib/systemd/sshd-restore-keys.service > +++ lib/systemd/sshd-restore-keys.service > @@ -1,7 +1,7 @@ > [Unit] > Description=OpenSSH server - restore persisted keys for the AOS > After=aos-storage-init.service > -Before=sshd-keygen.service > +Before=sshd-keygen@rsa.service sshd-keygen@ecdsa.service sshd-keygen@e > d25519.service > > [Service] > ExecStart=/usr/sbin/sshd-restore-keys > > > I'd love to know of a better way for my sshd-restore-kes.service to be > coded such that it's unnecessary to list individual key types, but my > experience has led me to believe that targets only define a point in > time when a collection of units have completed and that there's no > equivalent to a point in time prior to any of them having begun. Thus > I found I couldn't use Before=sshd-keygen.target because keys were > being generated and restored concurrently rather than serially > (restored if possible and then the missing generated as needed). > ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?
On Thu, 2017-03-23 at 06:38 +0300, Andrei Borzenkov wrote: > 22.03.2017 23:47, John Florian пишет: > > I build an mostly-stateless appliance OS derived from Fedora (25 > > ATM) > > and have several custom units to make it all possible. My units > > had > > worked great with F21, but are now giving me problems with > > F25. One > > pair of the custom units do some trickery to relocate sshd host > > keys > > from /etc/ssh to an alternate location that provides persistence: > > > > > > Please make full log available as well as actual unit definitions > that > are not started. > Okay, attached are a complete boot.log (journalctl > boot.log) as well as the most relevant units. Please note that while I need a solution, it's just as important I understand HOW to resolve this kind of issue. I've struggled through similar problems in the past and I never feel adept at the process, like I'm not using a fundamental tool for the job and instead relying more on trial and error. While there's no doubt a lot of change between F21 and F25, the one thing that forced a change on my part was systemd transitioning from a single sshd-keygen.service to a templated sshd-keygen@.service. For me that meant the following change: diff --git lib/systemd/sshd-persist-keys.service lib/systemd/sshd- persist-keys.service index d3508a8..7500b68 100644 --- lib/systemd/sshd-persist-keys.service +++ lib/systemd/sshd-persist-keys.service @@ -1,8 +1,8 @@ [Unit] Description=OpenSSH server - persist volatile keys for the AOS -After=sshd-keygen.service +After=sshd-keygen.target Before=sshd.service -Wants=sshd-keygen.service +Wants=sshd-keygen.target [Service] ExecStart=/usr/sbin/sshd-persist-keys diff --git lib/systemd/sshd-restore-keys.service lib/systemd/sshd- restore-keys.service index 475cede..85ac3bc 100644 --- lib/systemd/sshd-restore-keys.service +++ lib/systemd/sshd-restore-keys.service @@ -1,7 +1,7 @@ [Unit] Description=OpenSSH server - restore persisted keys for the AOS After=aos-storage-init.service -Before=sshd-keygen.service +Before=sshd-keygen@rsa.service sshd-keygen@ecdsa.service sshd-keygen@e d25519.service [Service] ExecStart=/usr/sbin/sshd-restore-keys I'd love to know of a better way for my sshd-restore-kes.service to be coded such that it's unnecessary to list individual key types, but my experience has led me to believe that targets only define a point in time when a collection of units have completed and that there's no equivalent to a point in time prior to any of them having begun. Thus I found I couldn't use Before=sshd-keygen.target because keys were being generated and restored concurrently rather than serially (restored if possible and then the missing generated as needed). sshd-startup-failure.tar.gz Description: application/compressed-tar ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?
22.03.2017 23:47, John Florian пишет: > I build an mostly-stateless appliance OS derived from Fedora (25 ATM) > and have several custom units to make it all possible. My units had > worked great with F21, but are now giving me problems with F25. One > pair of the custom units do some trickery to relocate sshd host keys > from /etc/ssh to an alternate location that provides persistence: > > $ systemctl cat sshd-persist-keys.service > # /usr/lib/systemd/system/sshd-persist-keys.service > [Unit] > Description=OpenSSH server - persist volatile keys for the AOS > After=sshd-keygen.target > Before=sshd.service > Wants=sshd-keygen.target > > [Service] > ExecStart=/usr/sbin/sshd-persist-keys > Type=oneshot > RemainAfterExit=yes > > [Install] > WantedBy=multi-user.target > > > $ systemctl cat sshd-restore-keys.service > # /usr/lib/systemd/system/sshd-restore-keys.service > [Unit] > Description=OpenSSH server - restore persisted keys for the AOS > After=aos-storage-init.service > Before=sshd-keygen@rsa.service sshd-keygen@ecdsa.service sshd-keygen@ed > 25519.service > > [Service] > ExecStart=/usr/sbin/sshd-restore-keys > Type=oneshot > RemainAfterExit=yes > > [Install] > WantedBy=multi-user.target > > > I found that on some boots, sshd wasn't getting started. With the help > of booting with systemd.log_level=debug I learned: > > $ sudo journalctl | grep conflict Please make full log available as well as actual unit definitions that are not started. > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job > sshd.service/start conflicted_by=no > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job > sshd.service/stop conflicted_by=yes > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing conflicting > jobs sshd.service/start,sshd.service/stop by deleting job > sshd.service/start ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] How do I easily resolve conflicting jobs in custom units?
I build an mostly-stateless appliance OS derived from Fedora (25 ATM)
and have several custom units to make it all possible. My units had
worked great with F21, but are now giving me problems with F25. One
pair of the custom units do some trickery to relocate sshd host keys
from /etc/ssh to an alternate location that provides persistence:
$ systemctl cat sshd-persist-keys.service
# /usr/lib/systemd/system/sshd-persist-keys.service
[Unit]
Description=OpenSSH server - persist volatile keys for the AOS
After=sshd-keygen.target
Before=sshd.service
Wants=sshd-keygen.target
[Service]
ExecStart=/usr/sbin/sshd-persist-keys
Type=oneshot
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
$ systemctl cat sshd-restore-keys.service
# /usr/lib/systemd/system/sshd-restore-keys.service
[Unit]
Description=OpenSSH server - restore persisted keys for the AOS
After=aos-storage-init.service
Before=sshd-keygen@rsa.service sshd-keygen@ecdsa.service sshd-keygen@ed
25519.service
[Service]
ExecStart=/usr/sbin/sshd-restore-keys
Type=oneshot
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
I found that on some boots, sshd wasn't getting started. With the help
of booting with systemd.log_level=debug I learned:
$ sudo journalctl | grep conflict
Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
sshd.service/start conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
sshd.service/stop conflicted_by=yes
Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing conflicting
jobs sshd.service/start,sshd.service/stop by deleting job
sshd.service/start
Mar 22 16:11:42 localhost systemd[1]: sshd.socket: Looking at job
sshd.socket/stop conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd.socket: Looking at job
sshd.socket/start conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd.socket: Fixing conflicting
jobs sshd.socket/stop,sshd.socket/start by deleting job
sshd.socket/stop
Mar 22 16:11:42 localhost systemd[1]: sshd-keygen.target: Looking at
job sshd-keygen.target/start conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd-keygen.target: Looking at
job sshd-keygen.target/stop conflicted_by=no
Mar 22 16:11:42 localhost systemd[1]: sshd-keygen.target: Fixing
conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop by
deleting job sshd-keygen.target/stop
I'm sure systemd-analyze is my friend here, but I've not found an
effective way to grok the problem. s-a dot either gives me too much to
comprehend or not enough to make evident the problem. I've tried
feeding it different PATTERNS like: none at all, sshd.service, ssh-
keygen.target, sshd-{persist,restore}-keys.service and more. I either
see a whole forest or just a few trees that seem to be getting along
happily together.
What advice can you suggest?___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel
