Public bug reported:
This is mostly the same issue as https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=934663.
With the default configuration of openssh-server and systemd, sshd will
complain and crash when multiple connections are made and terminated in
a quick succession, e.g. with `ssh-keyscan`. It results in the following
errors in /var/log/auth.log:
```
Nov 22 20:53:34 {host} sshd[14567]: Unable to negotiate with {client} port
41460: no matching host key type found. Their offer:
sk-ecdsa-sha2-nistp...@openssh.com [preauth]
Nov 22 20:53:34 {host} sshd[14570]: fatal: chroot("/run/sshd"): No such file or
directory [preauth]
Nov 22 20:53:34 {host} sshd[14569]: fatal: chroot("/run/sshd"): No such file or
directory [preauth]
Nov 22 20:53:34 {host} sshd[14568]: fatal: chroot("/run/sshd"): No such file or
directory [preauth]
Nov 22 20:53:34 {host} sshd[14566]: fatal: chroot("/run/sshd"): No such file or
directory [preauth]
Nov 22 20:53:47 {host} sshd[14584]: Connection closed by {client} port 59312
[preauth]
Nov 22 20:53:47 {host} sshd[14586]: fatal: chroot("/run/sshd"): No such file or
directory [preauth]
Nov 22 20:53:48 {host} sshd[14585]: fatal: chroot("/run/sshd"): No such file or
directory [preauth]
```
as well as e.g. missing responses in ssh-keyscan:
```
$ ssh-keyscan -vvv {host}
debug2: fd 3 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 2
debug2: fd 4 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 4
debug2: fd 5 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 8
debug2: fd 6 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 32
debug2: fd 7 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 64
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: sk-ecdsa-sha2-nistp...@openssh.com
debug2: ciphers ctos:
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com
debug2: ciphers stoc:
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com
debug2: MACs ctos:
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,z...@openssh.com
debug2: compression stoc: none,z...@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms:
rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos:
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com
debug2: ciphers stoc:
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com
debug2: MACs ctos:
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,z...@openssh.com
debug2: compression stoc: none,z...@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: (no match)
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
```
The error is most likely caused by a race condition on removing /run/sshd,
which is easily reproducible by ssh-keyscan.
I noticed that depeding on client, I'd sometimes miss all keys, sometimes get
one, sometimes more.
Modifying the following files (they should me marked as modified in the bug
report) seems to solve the issue, at least temporarily:
/usr/lib/systemd/system/ssh.service
/usr/lib/systemd/system/ssh@.service
In both cases, I added `RuntimeDirectoryPreserve=yes` to the `[Service]`
section, after `RuntimeDirectory=sshd`.
This is the same solution mentioned in the Debian bug, although their bug
report doesn't mention which service files are affected.
This doesn't seem to be a proper long-term solution though, as it seems apt
doesn't respect configuration files in /usr (or they are unlisted somewhere),
because after upgrading system just before filing this bug report, the files
got overwritten and reverted to their original form.
I only got asked about the /etc/ssh/sshd_config, which I have chosen to keep.
ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: openssh-server 1:8.2p1-4ubuntu0.1 [modified:
lib/systemd/system/ssh.service lib/systemd/system/ssh@.service]
ProcVersionSignature: Ubuntu 5.4.0-54.60-generic 5.4.65
Uname: Linux 5.4.0-54-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.12
Architecture: amd64
CasperMD5CheckResult: skip
Date: Mon Nov 23 15:09:32 2020
SourcePackage: openssh
UpgradeStatus: No upgrade log present (probably fresh install)
** Affects: openssh (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug focal
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/1905285
Title:
socket-activated sshd breaks on concurrent connections
Status in openssh package in Ubuntu:
New
Bug description:
This is mostly the same issue as https://bugs.debian.org/cgi-
bin/bugreport.cgi?bug=934663.
With the default configuration of openssh-server and systemd, sshd
will complain and crash when multiple connections are made and
terminated in a quick succession, e.g. with `ssh-keyscan`. It results
in the following errors in /var/log/auth.log:
```
Nov 22 20:53:34 {host} sshd[14567]: Unable to negotiate with {client} port
41460: no matching host key type found. Their offer:
sk-ecdsa-sha2-nistp...@openssh.com [preauth]
Nov 22 20:53:34 {host} sshd[14570]: fatal: chroot("/run/sshd"): No such file
or directory [preauth]
Nov 22 20:53:34 {host} sshd[14569]: fatal: chroot("/run/sshd"): No such file
or directory [preauth]
Nov 22 20:53:34 {host} sshd[14568]: fatal: chroot("/run/sshd"): No such file
or directory [preauth]
Nov 22 20:53:34 {host} sshd[14566]: fatal: chroot("/run/sshd"): No such file
or directory [preauth]
Nov 22 20:53:47 {host} sshd[14584]: Connection closed by {client} port 59312
[preauth]
Nov 22 20:53:47 {host} sshd[14586]: fatal: chroot("/run/sshd"): No such file
or directory [preauth]
Nov 22 20:53:48 {host} sshd[14585]: fatal: chroot("/run/sshd"): No such file
or directory [preauth]
```
as well as e.g. missing responses in ssh-keyscan:
```
$ ssh-keyscan -vvv {host}
debug2: fd 3 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 2
debug2: fd 4 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 4
debug2: fd 5 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 8
debug2: fd 6 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 32
debug2: fd 7 setting O_NONBLOCK
debug3: conalloc: oname {host} kt 64
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: sk-ecdsa-sha2-nistp...@openssh.com
debug2: ciphers ctos:
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com
debug2: ciphers stoc:
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com
debug2: MACs ctos:
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,z...@openssh.com
debug2: compression stoc: none,z...@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms:
rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos:
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com
debug2: ciphers stoc:
chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com
debug2: MACs ctos:
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,z...@openssh.com
debug2: compression stoc: none,z...@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: (no match)
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.1 pat OpenSSH* compat 0x04000000
# {host}:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
```
The error is most likely caused by a race condition on removing /run/sshd,
which is easily reproducible by ssh-keyscan.
I noticed that depeding on client, I'd sometimes miss all keys, sometimes get
one, sometimes more.
Modifying the following files (they should me marked as modified in the bug
report) seems to solve the issue, at least temporarily:
/usr/lib/systemd/system/ssh.service
/usr/lib/systemd/system/ssh@.service
In both cases, I added `RuntimeDirectoryPreserve=yes` to the `[Service]`
section, after `RuntimeDirectory=sshd`.
This is the same solution mentioned in the Debian bug, although their bug
report doesn't mention which service files are affected.
This doesn't seem to be a proper long-term solution though, as it seems apt
doesn't respect configuration files in /usr (or they are unlisted somewhere),
because after upgrading system just before filing this bug report, the files
got overwritten and reverted to their original form.
I only got asked about the /etc/ssh/sshd_config, which I have chosen to keep.
ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: openssh-server 1:8.2p1-4ubuntu0.1 [modified:
lib/systemd/system/ssh.service lib/systemd/system/ssh@.service]
ProcVersionSignature: Ubuntu 5.4.0-54.60-generic 5.4.65
Uname: Linux 5.4.0-54-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.12
Architecture: amd64
CasperMD5CheckResult: skip
Date: Mon Nov 23 15:09:32 2020
SourcePackage: openssh
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1905285/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
