Bug#1064347: openssh-server: sshd crashes under heavy traffic
Hello,
I am no maintainer, just tried to reproduce this issue which I could
inside a minimal Bullseye amd64 qemu VM with the instructions
from the linked Ubuntu bug.
I could not reproduce it within Bookworm or Trixie/testing.
Without "LogLevel DEBUG" it was also not observable.
Unfortunately did also not happen with a ssh package built with asan enabled.
And I upgraded step by step via snapshot.d.o, around 2021-11-15 it
stopped to be an issue. This step brought openssh 8.7p1-1.
Downgrading just openssh 8.4p1-6 in this exact VM showed it again.
Therefore I assume this issue got fixed between openssh 8.4p1-6 and 8.7p1-1.
Kind regards,
Bernhard
#13
#14 malloc_consolidate (av=av@entry=0x7faa3b64cb80 ) at
malloc.c:4518
#15 0x7faa3b5023d5 in _int_malloc (av=av@entry=0x7faa3b64cb80 ,
bytes=bytes@entry=8193) at malloc.c:3699
#16 0x7faa3b503063 in malloc_check (sz=8192, caller=) at
hooks.c:239
#17 0x7faa3b504cea in __libc_calloc (n=n@entry=1,
elem_size=elem_size@entry=8192) at malloc.c:3387
#18 0x7faa3b4f6ef4 in __GI___open_memstream
(bufloc=bufloc@entry=0x7ffe636eb6e0, sizeloc=sizeloc@entry=0x7ffe636eb6e8) at
memstream.c:83
#19 0x7faa3b5726e1 in __vsyslog_internal (pri=39, fmt=0x55b451dcb150
"%.500s", ap=0x7ffe636eb7d0, mode_flags=2) at ../misc/syslog.c:181
#20 0x7faa3b572d5f in __syslog_chk (pri=pri@entry=7, flag=flag@entry=1,
fmt=fmt@entry=0x55b451dcb150 "%.500s") at ../misc/syslog.c:136
#21 0x55b451d87e16 in syslog (__fmt=0x55b451dcb150 "%.500s", __pri=7) at
/usr/include/x86_64-linux-gnu/bits/syslog.h:31
#22 do_log (level=level@entry=SYSLOG_LEVEL_DEBUG1, fmt=fmt@entry=0x55b451dba421
"Forked child %ld.", args=args@entry=0x7ffe636ec110) at ../../log.c:484
#23 0x55b451d88254 in debug (fmt=fmt@entry=0x55b451dba421 "Forked child
%ld.") at ../../log.c:229
#24 0x55b451d3c86e in server_accept_loop (config_s=0x7ffe636ec270, newsock=, sock_out=, sock_in=) at
../../sshd.c:1377
#25 main (ac=, av=) at ../../sshd.c:2089
# 2024-04-23 Bullseye/stable amd64 qemu VM
apt update
apt dist-upgrade
apt install systemd-coredump moreutils parallel htop fakeroot mc ccache gdb
openssh-server-dbgsym
apt build-dep glibc
apt build-dep openssh-server
mkdir /home/benutzer/source/glibc/orig -p
cd/home/benutzer/source/glibc/orig
apt source glibc
mkdir /home/benutzer/source/openssh-server/orig -p
cd/home/benutzer/source/openssh-server/orig
apt source openssh-server
sed -i.bak 's/#LogLevel INFO/LogLevel DEBUG/g' /etc/ssh/sshd_config
systemctl restart sshd
ssh-keygen -b 4096
ssh-copy-id -i .ssh/id_rsa.pub benutzer@localhost
parallel -j 32 -N0 "ssh benutzer@localhost 'true'" ::: {1..2}
benutzer@debian:~/.ssh$ ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/benutzer/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/benutzer/.ssh/id_rsa
Your public key has been saved in /home/benutzer/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:Hgx6dUtFBhKiI0wBYKtXMkwZeRcP/eEZCUsU69bbO+o benutzer@debian
The key's randomart image is:
+---[RSA 4096]+
|+o== ++B+.++|
|.=+ ...=.++o |
| .*.+.. =oo+ |
|. = o = ++. |
|. . . . S o |
| . . o . o |
|. . .|
|.. |
| .E... |
+[SHA256]-+
benutzer@debian:~$ ssh-copy-id -i .ssh/id_rsa.pub benutzer@localhost
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter
out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are
prompted now it is to install the new keys
benutzer@localhost's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'benutzer@localhost'"
and check to make sure that only the key(s) you wanted were added.
parallel -j 800 -N0 "ssh benutzer@localhost 'mount; sleep 1; cat /proc/cpuinfo;
free -h; dd if=/dev/zero of=/dev/null bs=1 count=8192; mount -av; sleep
$(($RANDOM % 5)); lscpu'" ::: {1..1}
# AMD Ryzen 1700, VM, 16 threads
root@debian:~# coredumpctl list
TIMEPID UID GID SIG COREFILE EXE
Tue 2024-04-23 00:20:53 CEST 124297 0 0 6 present /usr/sbin/sshd
Tue 2024-04-23 00:23:02 CEST 159284 0 0 6 present /usr/sbin/sshd
Tue 2024-04-23 00:23:47 CEST 229261 0 0 11 present /usr/sbin/sshd
Tue 2024-04-23 00:24:32 CEST 277265 0 0 11 present /usr/sbin/sshd
Tue 2024-04-23 00:24:54 CEST 301567 0 0 6 present /usr/sbin/sshd
root@debian:~# coredumpctl gdb 301567
PID: 301567 (sshd)
UID: 0 (root)
GID: 0 (root)
Signal: 6 (ABRT)
Timestamp: Tue 2024-04-23 00:24:53 CEST (47s ago)
Command Line: sshd: /usr/sbin/sshd -D [listener] 4 of 10-100 startups
Executable: /usr/sbin/sshd
Bug#1064347: openssh-server: sshd crashes under heavy traffic
Package: openssh-server Version: 1:8.4p1-5+deb11u3 Severity: normal X-Debbugs-Cc: gkiss...@gmail.com Dear Maintainer, * What led up to the situation? We have a public facing sftp server for our customers After upgrading Debian 10 to Debian 11, sshd is crashing under heavy traffic * What exactly did you do (or not do) that was effective (or ineffective)? I tried reconfigure timeouts, keepalives etc but none worked * What was the outcome of this action? No change * What outcome did you expect instead? Not sure Very similar (or the same) with https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114 -- System Information: Debian Release: 11.9 APT prefers oldstable-updates APT policy: (500, 'oldstable-updates'), (500, 'oldstable-security'), (500, 'oldstable') Architecture: amd64 (x86_64) Kernel: Linux 5.10.0-28-amd64 (SMP w/8 CPU threads) Locale: LANG=en_US, LC_CTYPE=en_US (charmap=UTF-8) (ignored: LC_ALL set to en_US.UTF-8), LANGUAGE=en_US:en Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages openssh-server depends on: ii adduser3.118+deb11u1 ii debconf [debconf-2.0] 1.5.77 ii dpkg 1.20.13 ii libaudit1 1:3.0-2 ii libc6 2.31-13+deb11u8 ii libcom-err21.46.2-2 ii libcrypt1 1:4.4.18-4 ii libgssapi-krb5-2 1.18.3-6+deb11u4 ii libkrb5-3 1.18.3-6+deb11u4 ii libpam-modules 1.4.0-9+deb11u1 ii libpam-runtime 1.4.0-9+deb11u1 ii libpam0g 1.4.0-9+deb11u1 ii libselinux13.1-3 ii libssl1.1 1.1.1w-0+deb11u1 ii libsystemd0247.3-7+deb11u4 ii libwrap0 7.6.q-31 ii lsb-base 11.1.0 ii openssh-client 1:8.4p1-5+deb11u3 ii openssh-sftp-server1:8.4p1-5+deb11u3 ii procps 2:3.3.17-5 ii runit-helper 2.10.3 ii ucf3.0043 ii zlib1g 1:1.2.11.dfsg-2+deb11u2 Versions of packages openssh-server recommends: ii libpam-systemd [logind] 247.3-7+deb11u4 ii ncurses-term 6.2+20201114-2+deb11u2 ii xauth1:1.1-1 Versions of packages openssh-server suggests: pn molly-guard pn monkeysphere pn ssh-askpass pn ufw -- Configuration Files: /etc/pam.d/sshd changed [not included] -- debconf information excluded kern.log:Feb 19 22:29:40 ecmif01 kernel: [ 2288.914649] traps: sshd[72022] general protection fault ip:7f6b8116d3b7 sp:7fff87eb22f0 error:0 in libc-2.31.so[7f6b8110d000+159000] kern.log:Feb 19 22:46:04 ecmif01 kernel: [ 3272.826055] traps: sshd[98328] general protection fault ip:7f5e5c0433b7 sp:7fff4d3109f0 error:0 in libc-2.31.so[7f5e5bfe3000+159000] kern.log:Feb 19 23:14:08 ecmif01 kernel: [ 4956.789461] traps: sshd[152300] general protection fault ip:7f62249523b7 sp:7ffc761c4300 error:0 in libc-2.31.so[7f62248f2000+159000] kern.log:Feb 20 01:37:02 ecmif01 kernel: [13531.048898] sshd[437092]: segfault at 5618ab5283c8 ip 7f5f0a4a937f sp 7ffd30ac6640 error 4 in libc-2.31.so[7f5f0a449000+159000] kern.log:Feb 20 01:37:02 ecmif01 kernel: [13531.049177] Code: 00 00 eb a3 0f 1f 80 00 00 00 00 ff d0 eb c5 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 ec 08 48 8b 4f 08 48 89 c8 48 83 e0 f8 <48> 3b 04 07 0f 85 91 00 00 00 48 8b 47 10 48 8b 57 18 48 3b 78 18 kern.log:Feb 20 02:03:10 ecmif01 kernel: [15098.671459] traps: sshd[488901] general protection fault ip:7ff8c0fb23b7 sp:7ffdc8d60140 error:0 in libc-2.31.so[7ff8c0f52000+159000] kern.log:Feb 20 02:13:08 ecmif01 kernel: [15696.975972] sshd[496790]: segfault at 562609c87458 ip 7f7645a4c37f sp 7ffe0c52ee90 error 4 in libc-2.31.so[7f76459ec000+159000] kern.log:Feb 20 02:13:08 ecmif01 kernel: [15696.976252] Code: 00 00 eb a3 0f 1f 80 00 00 00 00 ff d0 eb c5 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 ec 08 48 8b 4f 08 48 89 c8 48 83 e0 f8 <48> 3b 04 07 0f 85 91 00 00 00 48 8b 47 10 48 8b 57 18 48 3b 78 18 kern.log:Feb 20 04:32:05 ecmif01 kernel: [24033.447133] traps: sshd[759145] general protection fault ip:7f6ebe48e3b7 sp:7ffe40a2cf30 error:0 in libc-2.31.so[7f6ebe42e000+159000] kern.log:Feb 20 04:46:01 ecmif01 kernel: [24869.466121] traps: sshd[804956] general protection fault ip:7f926c9863b7 sp:7fffdc0400a0 error:0 in libc-2.31.so[7f926c926000+159000] kern.log:Feb 20 05:16:14 ecmif01 kernel: [26682.432389] traps: sshd[871829] general protection fault ip:7f4ea598b3b7 sp:7ffd003bccf0 error:0 in libc-2.31.so[7f4ea592b000+159000] kern.log:Feb 20 05:58:09 ecmif01 kernel: [29197.006583] traps: sshd[978131] general protection fault ip:7f11eb09c3b7 sp:7ffefa5baaa0 error:0 in libc-2.31.so[7f11eb03c000+159000] kern.log:Feb 20 07:02:05 ecmif01 kernel: [33033.374881] traps: sshd[989] general protection fault ip:7fc463ef43b7 sp:7ffe79095300 error:0 in libc-2.31.so[7fc463e94000+159000] kern.log:Feb 20 07:28:04 ecmif01 kernel:
