Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Colin Watson]

On Sun, Dec 20, 2015 at 02:30:50AM +0100, Michael Biebl wrote:
> Am 14.11.2015 um 17:33 schrieb Michael Biebl:
> > Hi Colin, I didn't receive any feedback on this patch yet.
> > Would be great if you can have a look so we can fix this issue for good.
> 
> Do you have any thoughts/concerns regarding the proposed patch?
> If you don't find the patch acceptable, do you have ideas how we can
> address this differently?
> 
> I think it would be important to fix this properly now that systemd is
> the default.

Urgh, apologies!  The perils of using a threaded view of a large inbox
by default - I completely missed that you'd sent a patch here.

Your patch looks entirely fine to me.  I've uploaded it to unstable.

Thanks,

-- 
Colin Watson   [cjwat...@debian.org]

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Michael Biebl]

Hi Colin,

it's me again.

Am 14.11.2015 um 17:33 schrieb Michael Biebl:
> Hi Colin, I didn't receive any feedback on this patch yet.
> Would be great if you can have a look so we can fix this issue for good.

Do you have any thoughts/concerns regarding the proposed patch?
If you don't find the patch acceptable, do you have ideas how we can
address this differently?

I think it would be important to fix this properly now that systemd is
the default.

Regards,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Michael Biebl]

On Thu, 10 Sep 2015 18:13:24 +0200 Michael Biebl  wrote:
> Hi Colin!
> 
> On Tue, 12 May 2015 22:54:04 +0200 Michael Biebl  wrote:
> > Am 12.05.2015 um 17:42 schrieb Michael Biebl:
> > > Am 12.05.2015 um 17:07 schrieb Michael Biebl:
> > 
> > >> As you can see, systemd tries to repeatedly start the service until it 
> > >> hits
> > >> start-limit.
> > >> We should use sd_notify in that case to pass a correct error code to 
> > >> systemd.
> > > 
> > > Or we could use what's been proposed by Colin, i.e.
> > >  ExecStartPre=/usr/bin/sshd -t
> > > or my
> > >  RestartPreventExitStatus=255
> > 
> > Updated patch, adding RestartPreventExitStatus=, attached.
> > 
> > From my limited testing, seems to work fine here.
> 
> Now that jessie is out the door, it would be a great time to apply this
> patch and solve this issue for good.
> 
> Did you have time for a review yet?

Hi Colin, I didn't receive any feedback on this patch yet.
Would be great if you can have a look so we can fix this issue for good.

Cheers,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Michael Biebl]

Hi Colin!

On Tue, 12 May 2015 22:54:04 +0200 Michael Biebl  wrote:
> Am 12.05.2015 um 17:42 schrieb Michael Biebl:
> > Am 12.05.2015 um 17:07 schrieb Michael Biebl:
> 
> >> As you can see, systemd tries to repeatedly start the service until it hits
> >> start-limit.
> >> We should use sd_notify in that case to pass a correct error code to 
> >> systemd.
> > 
> > Or we could use what's been proposed by Colin, i.e.
> >  ExecStartPre=/usr/bin/sshd -t
> > or my
> >  RestartPreventExitStatus=255
> 
> Updated patch, adding RestartPreventExitStatus=, attached.
> 
> From my limited testing, seems to work fine here.

Now that jessie is out the door, it would be a great time to apply this
patch and solve this issue for good.

Did you have time for a review yet?

Cheers,
Michael

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Michael Biebl]

Am 12.05.2015 um 17:42 schrieb Michael Biebl:
 Am 12.05.2015 um 17:07 schrieb Michael Biebl:

 As you can see, systemd tries to repeatedly start the service until it hits
 start-limit.
 We should use sd_notify in that case to pass a correct error code to systemd.
 
 Or we could use what's been proposed by Colin, i.e.
  ExecStartPre=/usr/bin/sshd -t
 or my
  RestartPreventExitStatus=255

Updated patch, adding RestartPreventExitStatus=, attached.

From my limited testing, seems to work fine here.


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
diff --git a/configure.ac b/configure.ac
index f5c65c5..ef154ba 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4137,6 +4137,29 @@ AC_ARG_WITH(consolekit,
 	fi ]
 )
 
+# Check whether user wants systemd support
+SYSTEMD_MSG=no
+AC_ARG_WITH(systemd,
+	[  --with-systemd  Enable systemd support],
+	[ if test x$withval != xno ; then
+		AC_PATH_TOOL([PKGCONFIG], [pkg-config], [no])
+		if test $PKGCONFIG != no; then
+			AC_MSG_CHECKING([for libsystemd])
+			if $PKGCONFIG --exists libsystemd; then
+SYSTEMD_CFLAGS=`$PKGCONFIG --cflags libsystemd`
+SYSTEMD_LIBS=`$PKGCONFIG --libs libsystemd`
+CPPFLAGS=$CPPFLAGS $SYSTEMD_CFLAGS
+SSHDLIBS=$SSHDLIBS $SYSTEMD_LIBS
+AC_MSG_RESULT([yes])
+AC_DEFINE(HAVE_SYSTEMD, 1, [Define if you want systemd support.])
+SYSTEMD_MSG=yes
+			else
+AC_MSG_RESULT([no])
+			fi
+		fi
+	fi ]
+)
+
 # Looking for programs, paths and files
 
 PRIVSEP_PATH=/var/empty
@@ -4939,6 +4962,7 @@ echolibedit support: $LIBEDIT_MSG
 echo   Solaris process contract support: $SPC_MSG
 echoSolaris project support: $SP_MSG
 echo ConsoleKit support: $CONSOLEKIT_MSG
+echosystemd support: $SYSTEMD_MSG
 echoIP address in \$DISPLAY hack: $DISPLAY_HACK_MSG
 echoTranslate v4 in v6 hack: $IPV4_IN6_HACK_MSG
 echo   BSD Auth support: $BSD_AUTH_MSG
diff --git a/debian/control b/debian/control
index c513f4e..6144cf3 100644
--- a/debian/control
+++ b/debian/control
@@ -2,7 +2,7 @@ Source: openssh
 Section: net
 Priority: standard
 Maintainer: Debian OpenSSH Maintainers debian-...@lists.debian.org
-Build-Depends: libwrap0-dev | libwrap-dev, zlib1g-dev (= 1:1.2.3), libssl-dev (= 0.9.8g), libpam0g-dev | libpam-dev, libgtk2.0-dev, libedit-dev, debhelper (= 9~), dh-exec, libselinux1-dev [linux-any], libkrb5-dev | heimdal-dev, dpkg-dev (= 1.16.1~), libck-connector-dev, dh-autoreconf, autotools-dev, dh-systemd (= 1.4)
+Build-Depends: libwrap0-dev | libwrap-dev, zlib1g-dev (= 1:1.2.3), libssl-dev (= 0.9.8g), libpam0g-dev | libpam-dev, libgtk2.0-dev, libedit-dev, debhelper (= 9~), dh-exec, libselinux1-dev [linux-any], libkrb5-dev | heimdal-dev, dpkg-dev (= 1.16.1~), libck-connector-dev, dh-autoreconf, autotools-dev, dh-systemd (= 1.4), libsystemd-dev [linux-any]
 XS-Testsuite: autopkgtest
 Standards-Version: 3.9.6
 Uploaders: Colin Watson cjwat...@debian.org, Matthew Vernon matt...@debian.org
diff --git a/debian/rules b/debian/rules
index 570e651..8429054 100755
--- a/debian/rules
+++ b/debian/rules
@@ -91,6 +91,7 @@ confflags += --with-kerberos5=/usr
 confflags += --with-ssl-engine
 ifeq ($(DEB_HOST_ARCH_OS),linux)
 confflags += --with-selinux
+confflags += --with-systemd
 endif
 ifeq ($(DISTRIBUTOR),Ubuntu)
 confflags += --with-consolekit
diff --git a/debian/systemd/ssh.service b/debian/systemd/ssh.service
index ff28d39..3df8c64 100644
--- a/debian/systemd/ssh.service
+++ b/debian/systemd/ssh.service
@@ -9,6 +9,8 @@ ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
 ExecReload=/bin/kill -HUP $MAINPID
 KillMode=process
 Restart=on-failure
+RestartPreventExitStatus=255
+Type=notify
 
 [Install]
 WantedBy=multi-user.target
diff --git a/sshd.c b/sshd.c
index 23d5a64..180e9eb 100644
--- a/sshd.c
+++ b/sshd.c
@@ -84,6 +84,10 @@
 #include prot.h
 #endif
 
+#ifdef HAVE_SYSTEMD
+#include systemd/sd-daemon.h
+#endif
+
 #include xmalloc.h
 #include ssh.h
 #include ssh1.h
@@ -1927,6 +1931,12 @@ main(int ac, char **av)
 	/* ignore SIGPIPE */
 	signal(SIGPIPE, SIG_IGN);
 
+
+#ifdef HAVE_SYSTEMD
+	/* Signal systemd that we are ready to accept connections */
+	sd_notify(0, READY=1);
+#endif
+
 	/* Get a connection, either from inetd or a listening TCP socket */
 	if (inetd_flag) {
 		server_accept_inetd(sock_in, sock_out);


signature.asc
Description: OpenPGP digital signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Christoph Anton Mitterer]

On Tue, 2015-05-12 at 13:45 +0200, Michael Biebl wrote: 
 (since this was targetted at me, you should have CCed me. I don't get
 openssh-server bug mail)
Sorry,... must have dropped out somehow.

 A patch for that should be not that complicated and might even be worth
 shipping downstream if upstream doesn't want to apply it.
Well I don't know... I just said it might... could you propose the
patch upstream? I can do it for you, but I think it would be impolite
and you're probably better suited to do so, should further modifications
be required.


Cheers,
Chris.


smime.p7s
Description: S/MIME cryptographic signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Michael Biebl]

On Mon, 30 Mar 2015 04:02:01 +0200 Christoph Anton Mitterer
cales...@scientia.net wrote:
 Hi Michael.

(since this was targetted at me, you should have CCed me. I don't get
openssh-server bug mail)

 Your proposal seems to be a good solution for now. Maybe Colin can merge
 it and it will find it's way into jessie.

See my followup [1], Type=forking turned up to have it's own set of
problems.

 As for sd_notify,... a simply google query didn't turn up any existing
 patches for that and it may be hard to convince upstream to do it ;)

A patch for that should be not that complicated and might even be worth
shipping downstream if upstream doesn't want to apply it.

 Since this problem may affect other services as well... is there some
 concentrated effort in Debian to identify these?

Not that I'm aware of.

Michael


[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=778913#45

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Michael Biebl]

Control: tags -1 + patch

Am 12.05.2015 um 13:45 schrieb Michael Biebl:
 On Mon, 30 Mar 2015 04:02:01 +0200 Christoph Anton Mitterer
 As for sd_notify,... a simply google query didn't turn up any existing
 patches for that and it may be hard to convince upstream to do it ;)
 
 A patch for that should be not that complicated and might even be worth
 shipping downstream if upstream doesn't want to apply it.

Attached is a patch which adds support for sd_notify.
The configure.ac changes are a bit more convoluted then I hoped since 
openssh doesn't use the pkg-config provided macros.


A quick test (with a broken configuration file) at least seems to 
properly error out:


root@pluto:~# systemctl status ssh.service 
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/etc/systemd/system/ssh.service; enabled)
   Active: active (running) since Di 2015-05-12 17:03:28 CEST; 4s ago
 Main PID: 13021 (sshd)
   CGroup: /system.slice/ssh.service
   └─13021 /usr/sbin/sshd -D

Mai 12 17:03:28 pluto sshd[13021]: Server listening on 0.0.0.0 port 22.
Mai 12 17:03:28 pluto sshd[13021]: Server listening on :: port 22.

root@pluto:~# echo foobar  /etc/ssh/sshd_config 

root@pluto:~# systemctl restart ssh.service 
Job for ssh.service failed. See 'systemctl status ssh.service' and 'journalctl 
-xn' for details.

root@pluto:~# systemctl status ssh.service 
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/etc/systemd/system/ssh.service; enabled)
   Active: failed (Result: start-limit) since Di 2015-05-12 17:03:51 CEST; 5s 
ago
  Process: 13053 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, 
status=255)
 Main PID: 13053 (code=exited, status=255)

Mai 12 17:03:51 pluto sshd[13053]: /etc/ssh/sshd_config: terminating, 1 bad 
configuration options
Mai 12 17:03:51 pluto systemd[1]: ssh.service: main process exited, 
code=exited, status=255/n/a
Mai 12 17:03:51 pluto systemd[1]: Failed to start OpenBSD Secure Shell server.
Mai 12 17:03:51 pluto systemd[1]: Unit ssh.service entered failed state.
Mai 12 17:03:51 pluto systemd[1]: ssh.service start request repeated too 
quickly, refusing to start.
Mai 12 17:03:51 pluto systemd[1]: Failed to start OpenBSD Secure Shell server.
Mai 12 17:03:51 pluto systemd[1]: Unit ssh.service entered failed state.


As you can see, systemd tries to repeatedly start the service until it hits
start-limit.
We should use sd_notify in that case to pass a correct error code to systemd.

The patch is not complete yet, more a PoC.

That said, would be glad if Colin could give it some proper review.
Don't want to spend time on it, if it's unlikely to get merged.


Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
diff --git a/configure.ac b/configure.ac
index f5c65c5..ef154ba 100644
--- a/configure.ac
+++ b/configure.ac
@@ -4137,6 +4137,29 @@ AC_ARG_WITH(consolekit,
 	fi ]
 )
 
+# Check whether user wants systemd support
+SYSTEMD_MSG=no
+AC_ARG_WITH(systemd,
+	[  --with-systemd  Enable systemd support],
+	[ if test x$withval != xno ; then
+		AC_PATH_TOOL([PKGCONFIG], [pkg-config], [no])
+		if test $PKGCONFIG != no; then
+			AC_MSG_CHECKING([for libsystemd])
+			if $PKGCONFIG --exists libsystemd; then
+SYSTEMD_CFLAGS=`$PKGCONFIG --cflags libsystemd`
+SYSTEMD_LIBS=`$PKGCONFIG --libs libsystemd`
+CPPFLAGS=$CPPFLAGS $SYSTEMD_CFLAGS
+SSHDLIBS=$SSHDLIBS $SYSTEMD_LIBS
+AC_MSG_RESULT([yes])
+AC_DEFINE(HAVE_SYSTEMD, 1, [Define if you want systemd support.])
+SYSTEMD_MSG=yes
+			else
+AC_MSG_RESULT([no])
+			fi
+		fi
+	fi ]
+)
+
 # Looking for programs, paths and files
 
 PRIVSEP_PATH=/var/empty
@@ -4939,6 +4962,7 @@ echolibedit support: $LIBEDIT_MSG
 echo   Solaris process contract support: $SPC_MSG
 echoSolaris project support: $SP_MSG
 echo ConsoleKit support: $CONSOLEKIT_MSG
+echosystemd support: $SYSTEMD_MSG
 echoIP address in \$DISPLAY hack: $DISPLAY_HACK_MSG
 echoTranslate v4 in v6 hack: $IPV4_IN6_HACK_MSG
 echo   BSD Auth support: $BSD_AUTH_MSG
diff --git a/debian/control b/debian/control
index c513f4e..6144cf3 100644
--- a/debian/control
+++ b/debian/control
@@ -2,7 +2,7 @@ Source: openssh
 Section: net
 Priority: standard
 Maintainer: Debian OpenSSH Maintainers debian-...@lists.debian.org
-Build-Depends: libwrap0-dev | libwrap-dev, zlib1g-dev (= 1:1.2.3), libssl-dev (= 0.9.8g), libpam0g-dev | libpam-dev, libgtk2.0-dev, libedit-dev, debhelper (= 9~), dh-exec, libselinux1-dev [linux-any], libkrb5-dev | heimdal-dev, dpkg-dev (= 1.16.1~), libck-connector-dev, dh-autoreconf, autotools-dev, dh-systemd (= 1.4)
+Build-Depends: libwrap0-dev | libwrap-dev, zlib1g-dev (= 1:1.2.3), libssl-dev (= 0.9.8g), libpam0g-dev | libpam-dev, libgtk2.0-dev, libedit-dev, debhelper (= 9~), dh-exec, libselinux1-dev [linux-any], libkrb5-dev | heimdal-dev, dpkg-dev (= 1.16.1~), 

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Michael Biebl]

Am 12.05.2015 um 17:07 schrieb Michael Biebl:
 root@pluto:~# echo foobar  /etc/ssh/sshd_config 
 
 root@pluto:~# systemctl restart ssh.service 
 Job for ssh.service failed. See 'systemctl status ssh.service' and 
 'journalctl -xn' for details.
 
 root@pluto:~# systemctl status ssh.service 
 ● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/etc/systemd/system/ssh.service; enabled)
Active: failed (Result: start-limit) since Di 2015-05-12 17:03:51 CEST; 5s 
 ago
   Process: 13053 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, 
 status=255)
  Main PID: 13053 (code=exited, status=255)
 
 Mai 12 17:03:51 pluto sshd[13053]: /etc/ssh/sshd_config: terminating, 1 bad 
 configuration options
 Mai 12 17:03:51 pluto systemd[1]: ssh.service: main process exited, 
 code=exited, status=255/n/a
 Mai 12 17:03:51 pluto systemd[1]: Failed to start OpenBSD Secure Shell server.
 Mai 12 17:03:51 pluto systemd[1]: Unit ssh.service entered failed state.
 Mai 12 17:03:51 pluto systemd[1]: ssh.service start request repeated too 
 quickly, refusing to start.
 Mai 12 17:03:51 pluto systemd[1]: Failed to start OpenBSD Secure Shell server.
 Mai 12 17:03:51 pluto systemd[1]: Unit ssh.service entered failed state.
 
 
 As you can see, systemd tries to repeatedly start the service until it hits
 start-limit.
 We should use sd_notify in that case to pass a correct error code to systemd.

Or we could use what's been proposed by Colin, i.e.
 ExecStartPre=/usr/bin/sshd -t
or my
 RestartPreventExitStatus=255

The latter has the benefit, that we don't need to parse the config twice
and there is no race condition between ExecStartPre and ExecStart where
the config file might have been modified.

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Michael Biebl]

Am 30.03.2015 um 01:17 schrieb Michael Biebl:
 So I suggest using the Type=forking option but also setting
 RestartPreventExitStatus=255 [1], since 255 seems to be the return code
 on config errors and I don't think it makes sense to restart in that case.
 
 The resulting ssh.service would look like
 
 [Unit]
 Description=OpenBSD Secure Shell server
 After=network.target auditd.service
 ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
 
 [Service]
 EnvironmentFile=-/etc/default/ssh
 ExecStart=/usr/sbin/sshd $SSHD_OPTS
 ExecReload=/bin/kill -HUP $MAINPID
 KillMode=process
 Restart=on-failure
 Type=forking
 PIDFile=/var/run/sshd.pid
 RestartPreventExitStatus=255
 
 [Install]
 WantedBy=multi-user.target
 Alias=sshd.service
 
 
 With those changes, ssh.service ssems to behave as expected on failures.

I spoke too soon. As it turns out, sshd has a rather strange, or let's
say broken, SIGHUP behaviour (when in daemon mode): It reexecs, i.e.
changes its PID but doesn't write a new /var/run/sshd.pid. Since ssh
runs reload in it's if-up.d hook under systemd, this will break make it
break badly, since systemd will lose track of the sshd main process.

Colin, any idea, why sshd behaves so strange on SIGHUP?




-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Christoph Anton Mitterer]

Hi Michael.

Your proposal seems to be a good solution for now. Maybe Colin can merge
it and it will find it's way into jessie.

As for sd_notify,... a simply google query didn't turn up any existing
patches for that and it may be hard to convince upstream to do it ;)


Since this problem may affect other services as well... is there some
concentrated effort in Debian to identify these?

While systemd is great (in theory) I still feel that we (and others
either) don't use even a fraction of the goodness it could do.
Just look at the other bug that you've just closed before, and the
issues about services depending on facilities like firewall rules
loaded (network-pre.target is really a very unfortunate solution for
this... how regrettable that upstream went that way, even though better
ideas were in place :( ).
As you said, it's up to the respective package's maintainers, which
effectively means that either nothing of the new nice stuff gets used or
at least not homogeneously over all services.
So in the end, having a powerful framework where the admin can easily
control which facilites are mandatory for which services, as I've
proposed, would be up to the user.


Best wishes,
Chris.



smime.p7s
Description: S/MIME cryptographic signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Michael Biebl]

Am 22.02.2015 um 19:45 schrieb Russ Allbery:
 That's the problem with forking services that don't have status
 notification.  The default is Type=simple, which per systemd.service(5):
 
 If set to simple (the default value if neither Type= nor BusName=
 are specified), it is expected that the process configured with
 ExecStart= is the main process of the service. In this mode, if the
 process offers functionality to other processes on the system, its
 communication channels should be installed before the daemon is
 started up (e.g. sockets set up by systemd, via socket activation),
 as systemd will immediately proceed starting follow-up units.
 
 That last clause is exactly the problem that you're running into.  A
 Type=simple service says to run the command and immediately assume
 success.
 
 Type=forking plus PIDFile should be a minor improvement, assuming sshd
 does all of its checking before it forks.  The best systemd behavior,
 though, would come from adding sd_notify support so that sshd can
 affirmatively tell systemd whether it succeeded in startup or not, and
 then using Type=notify.  Then sshd startup won't be considered complete
 until the sshd daemon calls sd_notify, and the correct status will be
 reported if it exits for some reason before doing so.
 

Russ' explanation is excellent and exactly the problem you are running into.

If you change the service file to Type=forking as Russ advised, this
would be a slight improvement, as systemd would now detect the failure
to start the daemon. It has a gotcha though. Since you use
Restart=on-failure, systemd would restart the ssh daemon on config
errors, until it hits the restart limit. This would look like this then:

# systemctl start ssh.service
Job for ssh.service failed. See 'systemctl status ssh.service' and
'journalctl -xn' for details.
# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/etc/systemd/system/ssh.service; enabled)
   Active: failed (Result: start-limit) since Mo 2015-03-30 00:59:07
CEST; 2s ago
  Process: 11646 ExecStart=/usr/sbin/sshd $SSHD_OPTS (code=exited,
status=255)
 Main PID: 3849 (code=exited, status=255)

Mär 30 00:59:06 pluto systemd[1]: Failed to start OpenBSD Secure Shell
server.
Mär 30 00:59:06 pluto systemd[1]: Unit ssh.service entered failed state.
Mär 30 00:59:07 pluto systemd[1]: ssh.service start request repeated too
quickly, refusing to start.
Mär 30 00:59:07 pluto systemd[1]: Failed to start OpenBSD Secure Shell
server.
Mär 30 00:59:07 pluto systemd[1]: Unit ssh.service entered failed state.


Not really what we want either.

As Russ pointed out, using sd_notify would be the best option, but it's
too late for jessie, but maybe something to consider for stretch.

So I suggest using the Type=forking option but also setting
RestartPreventExitStatus=255 [1], since 255 seems to be the return code
on config errors and I don't think it makes sense to restart in that case.

The resulting ssh.service would look like

[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStart=/usr/sbin/sshd $SSHD_OPTS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
Type=forking
PIDFile=/var/run/sshd.pid
RestartPreventExitStatus=255

[Install]
WantedBy=multi-user.target
Alias=sshd.service


With those changes, ssh.service ssems to behave as expected on failures.


Michael

[1]
http://www.freedesktop.org/software/systemd/man/systemd.service.html#RestartPreventExitStatus=
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Colin Watson]

On Sat, Feb 21, 2015 at 07:08:56PM +0100, Christoph Anton Mitterer wrote:
 I've just tried that with systemd now,... starting the ssh service
 there always yields in a success (status 0, green OK), even when
 it actually failed to start.
 
 Simply add some invalid config option to sshd_config, restart and
 you'll se.
 Only when you go in deeper like systemctl status ssh.service, you'll
 see the actual error messages.

Thanks.  Do you think this would be suitable?  It seems to have the
right effect here.

diff --git a/debian/systemd/ssh.service b/debian/systemd/ssh.service
index ff28d39..cccafde 100644
--- a/debian/systemd/ssh.service
+++ b/debian/systemd/ssh.service
@@ -5,6 +5,7 @@ ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
 
 [Service]
 EnvironmentFile=-/etc/default/ssh
+ExecStartPre=/usr/sbin/sshd -t
 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
 ExecReload=/bin/kill -HUP $MAINPID
 KillMode=process

-- 
Colin Watson   [cjwat...@debian.org]


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Colin Watson]

On Sun, Feb 22, 2015 at 06:09:28PM +0100, Christoph Anton Mitterer wrote:
 On Sun, 2015-02-22 at 11:46 +, Colin Watson wrote: 
  Thanks.  Do you think this would be suitable?  It seems to have the
  right effect here.
  +ExecStartPre=/usr/sbin/sshd -t
 
 Well I'm a bit unsure... it practically fixes the problem, but
 a) It still doesn't catch any cases when sshd crashes for any other
 reason (and I think it should since systemd is also designed to have
 watchdog capabilities and that like).
 
 b) There's probably a short time between the -t check and running the
 daemon, in which the config could change again.

Well, um, in either case, isn't it pretty weird that systemctl status
shows the unit as failed while the boot progress system shows it as OK?
Feels like a systemd bug to me.

-- 
Colin Watson   [cjwat...@debian.org]


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Christoph Anton Mitterer]

On Sun, 2015-02-22 at 17:53 +, Colin Watson wrote: 
 Well, um, in either case, isn't it pretty weird that systemctl status
 shows the unit as failed while the boot progress system shows it as OK?
 Feels like a systemd bug to me.
Arguably, I'mm CCing the systemd guys, perhaps they can help out!

The exit status returned by sshd is 255 (anyhow special to systemd?): 
# /usr/sbin/sshd -D -p 5
/etc/ssh/sshd_config line 151: AuthorizedKeysCommand must be an absolute path
# echo $?
255



Cheers,
Chris.


smime.p7s
Description: S/MIME cryptographic signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Christoph Anton Mitterer]

On Sun, 2015-02-22 at 11:46 +, Colin Watson wrote: 
 Thanks.  Do you think this would be suitable?  It seems to have the
 right effect here.
 +ExecStartPre=/usr/sbin/sshd -t

Well I'm a bit unsure... it practically fixes the problem, but
a) It still doesn't catch any cases when sshd crashes for any other
reason (and I think it should since systemd is also designed to have
watchdog capabilities and that like).

b) There's probably a short time between the -t check and running the
daemon, in which the config could change again.

Cheers,
Chris.


smime.p7s
Description: S/MIME cryptographic signature

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Russ Allbery]

Colin Watson cjwat...@debian.org writes:
 On Sun, Feb 22, 2015 at 06:09:28PM +0100, Christoph Anton Mitterer wrote:

 Well I'm a bit unsure... it practically fixes the problem, but a) It
 still doesn't catch any cases when sshd crashes for any other reason
 (and I think it should since systemd is also designed to have watchdog
 capabilities and that like).

 b) There's probably a short time between the -t check and running the
 daemon, in which the config could change again.

 Well, um, in either case, isn't it pretty weird that systemctl status
 shows the unit as failed while the boot progress system shows it as OK?
 Feels like a systemd bug to me.

That's the problem with forking services that don't have status
notification.  The default is Type=simple, which per systemd.service(5):

If set to simple (the default value if neither Type= nor BusName=
are specified), it is expected that the process configured with
ExecStart= is the main process of the service. In this mode, if the
process offers functionality to other processes on the system, its
communication channels should be installed before the daemon is
started up (e.g. sockets set up by systemd, via socket activation),
as systemd will immediately proceed starting follow-up units.

That last clause is exactly the problem that you're running into.  A
Type=simple service says to run the command and immediately assume
success.

Type=forking plus PIDFile should be a minor improvement, assuming sshd
does all of its checking before it forks.  The best systemd behavior,
though, would come from adding sd_notify support so that sshd can
affirmatively tell systemd whether it succeeded in startup or not, and
then using Type=notify.  Then sshd startup won't be considered complete
until the sshd daemon calls sd_notify, and the correct status will be
reported if it exits for some reason before doing so.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Bug#778913: openssh-server: init (at least systemd) doesn't notice when sshd fails to start and reports success

[Christoph Anton Mitterer]

Package: openssh-server
Version: 1:6.7p1-3
Severity: important


Hi.

I've just tried that with systemd now,... starting the ssh service
there always yields in a success (status 0, green OK), even when
it actually failed to start.

Simply add some invalid config option to sshd_config, restart and
you'll se.
Only when you go in deeper like systemctl status ssh.service, you'll
see the actual error messages.


Cheers,
Chris.


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org