Bug#1064347: openssh-server: sshd crashes under heavy traffic

2024-04-23 Thread Bernhard √úbelacker

Hello,
I am no maintainer, just tried to reproduce this issue which I could
inside a minimal Bullseye amd64 qemu VM with the instructions
from the linked Ubuntu bug.

I could not reproduce it within Bookworm or Trixie/testing.

Without "LogLevel DEBUG" it was also not observable.

Unfortunately did also not happen with a ssh package built with asan enabled.

And I upgraded step by step via snapshot.d.o, around 2021-11-15 it
stopped to be an issue. This step brought openssh 8.7p1-1.
Downgrading just openssh 8.4p1-6 in this exact VM showed it again.

Therefore I assume this issue got fixed between openssh 8.4p1-6 and 8.7p1-1.

Kind regards,
Bernhard


#13 
#14 malloc_consolidate (av=av@entry=0x7faa3b64cb80 ) at 
malloc.c:4518
#15 0x7faa3b5023d5 in _int_malloc (av=av@entry=0x7faa3b64cb80 , 
bytes=bytes@entry=8193) at malloc.c:3699
#16 0x7faa3b503063 in malloc_check (sz=8192, caller=) at 
hooks.c:239
#17 0x7faa3b504cea in __libc_calloc (n=n@entry=1, 
elem_size=elem_size@entry=8192) at malloc.c:3387
#18 0x7faa3b4f6ef4 in __GI___open_memstream 
(bufloc=bufloc@entry=0x7ffe636eb6e0, sizeloc=sizeloc@entry=0x7ffe636eb6e8) at 
memstream.c:83
#19 0x7faa3b5726e1 in __vsyslog_internal (pri=39, fmt=0x55b451dcb150 
"%.500s", ap=0x7ffe636eb7d0, mode_flags=2) at ../misc/syslog.c:181
#20 0x7faa3b572d5f in __syslog_chk (pri=pri@entry=7, flag=flag@entry=1, 
fmt=fmt@entry=0x55b451dcb150 "%.500s") at ../misc/syslog.c:136
#21 0x55b451d87e16 in syslog (__fmt=0x55b451dcb150 "%.500s", __pri=7) at 
/usr/include/x86_64-linux-gnu/bits/syslog.h:31
#22 do_log (level=level@entry=SYSLOG_LEVEL_DEBUG1, fmt=fmt@entry=0x55b451dba421 
"Forked child %ld.", args=args@entry=0x7ffe636ec110) at ../../log.c:484
#23 0x55b451d88254 in debug (fmt=fmt@entry=0x55b451dba421 "Forked child 
%ld.") at ../../log.c:229
#24 0x55b451d3c86e in server_accept_loop (config_s=0x7ffe636ec270, newsock=, sock_out=, sock_in=) at 
../../sshd.c:1377
#25 main (ac=, av=) at ../../sshd.c:2089
# 2024-04-23 Bullseye/stable amd64 qemu VM


apt update
apt dist-upgrade
apt install systemd-coredump moreutils parallel htop fakeroot mc ccache gdb 
openssh-server-dbgsym
apt build-dep glibc
apt build-dep openssh-server


mkdir /home/benutzer/source/glibc/orig -p
cd/home/benutzer/source/glibc/orig
apt source glibc

mkdir /home/benutzer/source/openssh-server/orig -p
cd/home/benutzer/source/openssh-server/orig
apt source openssh-server



sed -i.bak 's/#LogLevel INFO/LogLevel DEBUG/g' /etc/ssh/sshd_config
systemctl restart sshd



ssh-keygen -b 4096
ssh-copy-id -i .ssh/id_rsa.pub benutzer@localhost
parallel -j 32 -N0 "ssh benutzer@localhost 'true'" ::: {1..2}







benutzer@debian:~/.ssh$ ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/benutzer/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/benutzer/.ssh/id_rsa
Your public key has been saved in /home/benutzer/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:Hgx6dUtFBhKiI0wBYKtXMkwZeRcP/eEZCUsU69bbO+o benutzer@debian
The key's randomart image is:
+---[RSA 4096]+
|+o==  ++B+.++|
|.=+ ...=.++o |
| .*.+.. =oo+ |
|.  = o = ++. |
|. . . . S o  |
| .   . o . o |
|. . .|
|..   |
| .E...   |
+[SHA256]-+


benutzer@debian:~$ ssh-copy-id -i .ssh/id_rsa.pub benutzer@localhost
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter 
out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are 
prompted now it is to install the new keys
benutzer@localhost's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'benutzer@localhost'"
and check to make sure that only the key(s) you wanted were added.





parallel -j 800 -N0 "ssh benutzer@localhost 'mount; sleep 1; cat /proc/cpuinfo; 
free -h; dd if=/dev/zero of=/dev/null bs=1 count=8192; mount -av; sleep 
$(($RANDOM % 5)); lscpu'" ::: {1..1}
# AMD Ryzen 1700, VM, 16 threads















root@debian:~# coredumpctl list
TIMEPID   UID   GID SIG COREFILE  EXE
Tue 2024-04-23 00:20:53 CEST 124297 0 0   6 present   /usr/sbin/sshd
Tue 2024-04-23 00:23:02 CEST 159284 0 0   6 present   /usr/sbin/sshd
Tue 2024-04-23 00:23:47 CEST 229261 0 0  11 present   /usr/sbin/sshd
Tue 2024-04-23 00:24:32 CEST 277265 0 0  11 present   /usr/sbin/sshd
Tue 2024-04-23 00:24:54 CEST 301567 0 0   6 present   /usr/sbin/sshd





root@debian:~# coredumpctl gdb 301567
   PID: 301567 (sshd)
   UID: 0 (root)
   GID: 0 (root)
Signal: 6 (ABRT)
 Timestamp: Tue 2024-04-23 00:24:53 CEST (47s ago)
  Command Line: sshd: /usr/sbin/sshd -D [listener] 4 of 10-100 startups
Executable: /usr/sbin/sshd
 

Bug#1064347: openssh-server: sshd crashes under heavy traffic

2024-02-20 Thread George Kissandrakis
Package: openssh-server
Version: 1:8.4p1-5+deb11u3
Severity: normal
X-Debbugs-Cc: gkiss...@gmail.com

Dear Maintainer,

   * What led up to the situation?
We have a public facing sftp server for our customers
After upgrading Debian 10 to Debian 11, sshd is crashing under heavy traffic

   * What exactly did you do (or not do) that was effective (or
 ineffective)?
I tried reconfigure timeouts, keepalives etc but none worked

   * What was the outcome of this action?
No change

   * What outcome did you expect instead?
Not sure

Very similar (or the same) with
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114


-- System Information:
Debian Release: 11.9
  APT prefers oldstable-updates
  APT policy: (500, 'oldstable-updates'), (500, 'oldstable-security'), (500, 
'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-28-amd64 (SMP w/8 CPU threads)
Locale: LANG=en_US, LC_CTYPE=en_US (charmap=UTF-8) (ignored: LC_ALL set to 
en_US.UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages openssh-server depends on:
ii  adduser3.118+deb11u1
ii  debconf [debconf-2.0]  1.5.77
ii  dpkg   1.20.13
ii  libaudit1  1:3.0-2
ii  libc6  2.31-13+deb11u8
ii  libcom-err21.46.2-2
ii  libcrypt1  1:4.4.18-4
ii  libgssapi-krb5-2   1.18.3-6+deb11u4
ii  libkrb5-3  1.18.3-6+deb11u4
ii  libpam-modules 1.4.0-9+deb11u1
ii  libpam-runtime 1.4.0-9+deb11u1
ii  libpam0g   1.4.0-9+deb11u1
ii  libselinux13.1-3
ii  libssl1.1  1.1.1w-0+deb11u1
ii  libsystemd0247.3-7+deb11u4
ii  libwrap0   7.6.q-31
ii  lsb-base   11.1.0
ii  openssh-client 1:8.4p1-5+deb11u3
ii  openssh-sftp-server1:8.4p1-5+deb11u3
ii  procps 2:3.3.17-5
ii  runit-helper   2.10.3
ii  ucf3.0043
ii  zlib1g 1:1.2.11.dfsg-2+deb11u2

Versions of packages openssh-server recommends:
ii  libpam-systemd [logind]  247.3-7+deb11u4
ii  ncurses-term 6.2+20201114-2+deb11u2
ii  xauth1:1.1-1

Versions of packages openssh-server suggests:
pn  molly-guard   
pn  monkeysphere  
pn  ssh-askpass   
pn  ufw   

-- Configuration Files:
/etc/pam.d/sshd changed [not included]

-- debconf information excluded
kern.log:Feb 19 22:29:40 ecmif01 kernel: [ 2288.914649] traps: sshd[72022] 
general protection fault ip:7f6b8116d3b7 sp:7fff87eb22f0 error:0 in 
libc-2.31.so[7f6b8110d000+159000]
kern.log:Feb 19 22:46:04 ecmif01 kernel: [ 3272.826055] traps: sshd[98328] 
general protection fault ip:7f5e5c0433b7 sp:7fff4d3109f0 error:0 in 
libc-2.31.so[7f5e5bfe3000+159000]
kern.log:Feb 19 23:14:08 ecmif01 kernel: [ 4956.789461] traps: sshd[152300] 
general protection fault ip:7f62249523b7 sp:7ffc761c4300 error:0 in 
libc-2.31.so[7f62248f2000+159000]
kern.log:Feb 20 01:37:02 ecmif01 kernel: [13531.048898] sshd[437092]: segfault 
at 5618ab5283c8 ip 7f5f0a4a937f sp 7ffd30ac6640 error 4 in 
libc-2.31.so[7f5f0a449000+159000]
kern.log:Feb 20 01:37:02 ecmif01 kernel: [13531.049177] Code: 00 00 eb a3 0f 1f 
80 00 00 00 00 ff d0 eb c5 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 ec 08 48 
8b 4f 08 48 89 c8 48 83 e0 f8 <48> 3b 04 07 0f 85 91 00 00 00 48 8b 47 10 48 8b 
57 18 48 3b 78 18
kern.log:Feb 20 02:03:10 ecmif01 kernel: [15098.671459] traps: sshd[488901] 
general protection fault ip:7ff8c0fb23b7 sp:7ffdc8d60140 error:0 in 
libc-2.31.so[7ff8c0f52000+159000]
kern.log:Feb 20 02:13:08 ecmif01 kernel: [15696.975972] sshd[496790]: segfault 
at 562609c87458 ip 7f7645a4c37f sp 7ffe0c52ee90 error 4 in 
libc-2.31.so[7f76459ec000+159000]
kern.log:Feb 20 02:13:08 ecmif01 kernel: [15696.976252] Code: 00 00 eb a3 0f 1f 
80 00 00 00 00 ff d0 eb c5 66 66 2e 0f 1f 84 00 00 00 00 00 90 48 83 ec 08 48 
8b 4f 08 48 89 c8 48 83 e0 f8 <48> 3b 04 07 0f 85 91 00 00 00 48 8b 47 10 48 8b 
57 18 48 3b 78 18
kern.log:Feb 20 04:32:05 ecmif01 kernel: [24033.447133] traps: sshd[759145] 
general protection fault ip:7f6ebe48e3b7 sp:7ffe40a2cf30 error:0 in 
libc-2.31.so[7f6ebe42e000+159000]
kern.log:Feb 20 04:46:01 ecmif01 kernel: [24869.466121] traps: sshd[804956] 
general protection fault ip:7f926c9863b7 sp:7fffdc0400a0 error:0 in 
libc-2.31.so[7f926c926000+159000]
kern.log:Feb 20 05:16:14 ecmif01 kernel: [26682.432389] traps: sshd[871829] 
general protection fault ip:7f4ea598b3b7 sp:7ffd003bccf0 error:0 in 
libc-2.31.so[7f4ea592b000+159000]
kern.log:Feb 20 05:58:09 ecmif01 kernel: [29197.006583] traps: sshd[978131] 
general protection fault ip:7f11eb09c3b7 sp:7ffefa5baaa0 error:0 in 
libc-2.31.so[7f11eb03c000+159000]
kern.log:Feb 20 07:02:05 ecmif01 kernel: [33033.374881] traps: sshd[989] 
general protection fault ip:7fc463ef43b7 sp:7ffe79095300 error:0 in 
libc-2.31.so[7fc463e94000+159000]
kern.log:Feb 20 07:28:04 ecmif01 kernel: