Package: libc6 Version: 2.31-1 Severity: critical Justification: breaks unrelated software; breaks remote access
TL;DR: sshd privsep child dies with SIGSYS in clock_nanosleep() (libc6 2.31-1) while it succeeded using nanosleep() under libc6 2.30-8 The machine in question is running buster with some selected packages (mainly compilers and development stuff) from bullseye (and is located at a remote location). The running kernel is 4.19.0-9-amd64 4.19.118-2. openssh-server 1:7.9p1-10+deb10u2 is running. After upgrading libc6 from 2.30-8 to 2.31-1 (which caused sshd to restart), sshd was running, but dropped incoming connections during authentication. Luckily I still had a terminal open and could downgrade again to 2.30-8 which restored accessibility. Thanks to the people trying to guess usernames and passwords, I noticed this difference in /var/log/auth.log: with 2.31-1: Jul 20 21:52:11 hostname sshd[25603]: Invalid user ping from 139.219.0.102 port 39588 Jul 20 21:52:11 hostname sshd[25603]: pam_unix(sshd:auth): check pass; user unknown Jul 20 21:52:11 hostname sshd[25603]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=139.219.0.102 Jul 20 21:52:13 hostname sshd[25603]: Failed password for invalid user ping from 139.219.0.102 port 39588 ssh2 after downgrading to 2.30-8: Jul 20 21:54:33 hostname sshd[26824]: Invalid user mickey from 51.83.131.123 port 32822 Jul 20 21:54:33 hostname sshd[26824]: pam_unix(sshd:auth): check pass; user unknown Jul 20 21:54:33 hostname sshd[26824]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=51.83.131.123 Jul 20 21:54:35 hostname sshd[26824]: Failed password for invalid user mickey from 51.83.131.123 port 32822 ssh2 Jul 20 21:54:35 hostname sshd[26824]: Received disconnect from 51.83.131.123 port 32822:11: Bye Bye [preauth] Jul 20 21:54:35 hostname sshd[26824]: Disconnected from invalid user mickey 51.83.131.123 port 32822 [preauth] I can reproduce this by running sshd in a mininmal buster chroot and upgrading libc6 (+ libgcc-s1 libcrypto1 libc-bin). (There is even no need to restart sshd (which was started under 2.31-1) after downgrading libc6 again to 2.30-8 to get it functional again.) I haven't tried sshd/bullseye. I haven't tried booting with 2.31-1. $ ssh -vvv foo@localhost -p 9922 [...] debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ecdsa-sha2-nistp256 SHA256:/07awyZSdCd9QgaTWi1dn3kEg9rbZtYC+ejHd6ZFi2w debug3: put_host_port: [127.0.0.1]:9922 debug3: put_host_port: [localhost]:9922 debug3: hostkeys_foreach: reading file "/home/beckmann/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /home/beckmann/.ssh/known_hosts:4 debug3: load_hostkeys: loaded 1 keys from [localhost]:9922 debug1: Host '[localhost]:9922' is known and matches the ECDSA host key. debug1: Found key in /home/beckmann/.ssh/known_hosts:4 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey after 134217728 blocks debug1: Will attempt key: /home/beckmann/.ssh/id_dsa debug1: Will attempt key: /home/beckmann/.ssh/id_ecdsa debug1: Will attempt key: /home/beckmann/.ssh/id_ed25519 debug1: Will attempt key: /home/beckmann/.ssh/id_xmss debug2: pubkey_prepare: done debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 Connection closed by 127.0.0.1 port 9922 # /usr/sbin/sshd -ddd debug2: load_server_config: filename /etc/ssh/sshd_config debug2: load_server_config: done config len = 328 debug2: parse_server_config: config /etc/ssh/sshd_config len 328 debug3: /etc/ssh/sshd_config:13 setting Port 9922 debug3: /etc/ssh/sshd_config:26 setting SyslogFacility LOCAL7 debug3: /etc/ssh/sshd_config:27 setting LogLevel DEBUG3 debug3: /etc/ssh/sshd_config:56 setting PasswordAuthentication yes debug3: /etc/ssh/sshd_config:61 setting ChallengeResponseAuthentication no debug3: /etc/ssh/sshd_config:84 setting UsePAM yes debug3: /etc/ssh/sshd_config:89 setting X11Forwarding yes debug3: /etc/ssh/sshd_config:93 setting PrintMotd no debug3: /etc/ssh/sshd_config:111 setting AcceptEnv LANG LC_* debug3: /etc/ssh/sshd_config:114 setting Subsystem sftp /usr/lib/openssh/sftp-server debug1: sshd version OpenSSH_7.9, OpenSSL 1.1.1d 10 Sep 2019 debug1: private host key #0: ssh-rsa SHA256:Ny3efyKdYNkwzXduBRO9Fzl0k2K505paO5QFGGw0o1s debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:/07awyZSdCd9QgaTWi1dn3kEg9rbZtYC+ejHd6ZFi2w debug1: private host key #2: ssh-ed25519 SHA256:COeHE8usWY7gfl1+F5DqGx8pptr/4duLPiaPai3J5uo debug1: rexec_argv[0]='/usr/sbin/sshd' debug1: rexec_argv[1]='-ddd' debug3: oom_adjust_setup debug1: Set /proc/self/oom_score_adj from 0 to -1000 debug2: fd 3 setting O_NONBLOCK debug1: Bind to port 9922 on 0.0.0.0. Server listening on 0.0.0.0 port 9922. debug2: fd 4 setting O_NONBLOCK debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY debug1: Bind to port 9922 on ::. Server listening on :: port 9922. debug3: fd 5 is not O_NONBLOCK debug1: Server will not fork when running in debugging mode. debug3: send_rexec_state: entering fd = 8 config len 328 debug3: ssh_msg_send: type 0 debug3: send_rexec_state: done debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8 debug1: inetd sockets after dupping: 3, 3 Connection from 127.0.0.1 port 42960 on 127.0.0.1 port 9922 debug1: Client protocol version 2.0; client software version OpenSSH_7.9p1 Debian-10+deb10u2 debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000 debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2 debug2: fd 3 setting O_NONBLOCK debug3: ssh_sandbox_init: preparing seccomp filter sandbox debug2: Network child is on pid 3021 debug3: preauth child monitor started debug3: privsep user:group 101:65534 [preauth] debug1: permanently_set_uid: 101/65534 [preauth] debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth] debug3: ssh_sandbox_child: attaching seccomp filter program [preauth] debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth] debug3: send packet: type 20 [preauth] debug1: SSH2_MSG_KEXINIT sent [preauth] debug3: receive packet: type 20 [preauth] debug1: SSH2_MSG_KEXINIT received [preauth] debug2: local server KEXINIT proposal [preauth] debug2: KEX algorithms: curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 [preauth] debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth] debug2: ciphers ctos: chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com [preauth] debug2: ciphers stoc: chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com [preauth] debug2: MACs ctos: umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth] debug2: MACs stoc: umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth] debug2: compression ctos: none,z...@openssh.com [preauth] debug2: compression stoc: none,z...@openssh.com [preauth] debug2: languages ctos: [preauth] debug2: languages stoc: [preauth] debug2: first_kex_follows 0 [preauth] debug2: reserved 0 [preauth] debug2: peer client KEXINIT proposal [preauth] debug2: KEX algorithms: curve25519-sha256,curve25519-sha...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c [preauth] debug2: host key algorithms: ecdsa-sha2-nistp256-cert-...@openssh.com,ecdsa-sha2-nistp384-cert-...@openssh.com,ecdsa-sha2-nistp521-cert-...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-...@openssh.com,rsa-sha2-512-cert-...@openssh.com,rsa-sha2-256-cert-...@openssh.com,ssh-rsa-cert-...@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth] debug2: ciphers ctos: chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com [preauth] debug2: ciphers stoc: chacha20-poly1...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-...@openssh.com,aes256-...@openssh.com [preauth] debug2: MACs ctos: umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth] debug2: MACs stoc: umac-64-...@openssh.com,umac-128-...@openssh.com,hmac-sha2-256-...@openssh.com,hmac-sha2-512-...@openssh.com,hmac-sha1-...@openssh.com,umac...@openssh.com,umac-...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth] debug2: compression ctos: none,z...@openssh.com,zlib [preauth] debug2: compression stoc: none,z...@openssh.com,zlib [preauth] debug2: languages ctos: [preauth] debug2: languages stoc: [preauth] debug2: first_kex_follows 0 [preauth] debug2: reserved 0 [preauth] debug1: kex: algorithm: curve25519-sha256 [preauth] debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth] debug1: kex: client->server cipher: chacha20-poly1...@openssh.com MAC: <implicit> compression: none [preauth] debug1: kex: server->client cipher: chacha20-poly1...@openssh.com MAC: <implicit> compression: none [preauth] debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth] debug3: receive packet: type 30 [preauth] debug3: mm_sshkey_sign entering [preauth] debug3: mm_request_send entering: type 6 [preauth] debug3: mm_sshkey_sign: waiting for MONITOR_ANS_SIGN [preauth] debug3: mm_request_receive_expect entering: type 7 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 6 debug3: mm_answer_sign debug3: mm_answer_sign: hostkey proof signature 0x557e074790c0(100) debug3: mm_request_send entering: type 7 debug2: monitor_read: 6 used once, disabling now debug3: send packet: type 31 [preauth] debug3: send packet: type 21 [preauth] debug2: set_newkeys: mode 1 [preauth] debug1: rekey after 134217728 blocks [preauth] debug1: SSH2_MSG_NEWKEYS sent [preauth] debug1: expecting SSH2_MSG_NEWKEYS [preauth] debug3: send packet: type 7 [preauth] debug3: receive packet: type 21 [preauth] debug1: SSH2_MSG_NEWKEYS received [preauth] debug2: set_newkeys: mode 0 [preauth] debug1: rekey after 134217728 blocks [preauth] debug1: KEX done [preauth] debug3: receive packet: type 5 [preauth] debug3: send packet: type 6 [preauth] debug3: receive packet: type 50 [preauth] debug1: userauth-request for user foo service ssh-connection method none [preauth] debug1: attempt 0 failures 0 [preauth] debug3: mm_getpwnamallow entering [preauth] debug3: mm_request_send entering: type 8 [preauth] debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth] debug3: mm_request_receive_expect entering: type 9 [preauth] debug3: mm_request_receive entering [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 8 debug3: mm_answer_pwnamallow debug2: parse_server_config: config reprocess config len 328 debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1 debug3: mm_request_send entering: type 9 debug2: monitor_read: 8 used once, disabling now debug2: input_userauth_request: setting up authctxt for foo [preauth] debug3: mm_start_pam entering [preauth] debug3: mm_request_send entering: type 100 [preauth] debug3: mm_inform_authserv entering [preauth] debug3: mm_request_send entering: type 4 [preauth] debug2: input_userauth_request: try method none [preauth] debug3: mm_request_receive entering debug3: monitor_read: checking request 100 debug1: PAM: initializing for "foo" debug1: PAM: setting PAM_RHOST to "127.0.0.1" debug1: PAM: setting PAM_TTY to "ssh" debug2: monitor_read: 100 used once, disabling now debug3: user_specific_delay: user specific delay 0.000ms [preauth] debug3: ensure_minimum_time_since: elapsed 1.381ms, delaying 4.141ms (requested 5.522ms) [preauth] debug1: monitor_read_log: child log fd closed debug3: mm_request_receive entering debug3: monitor_read: checking request 4 debug3: mm_answer_authserv: service=ssh-connection, style=, role= debug2: monitor_read: 4 used once, disabling now debug3: mm_request_receive entering debug1: do_cleanup debug1: PAM: cleanup debug3: PAM: sshpam_thread_cleanup entering debug1: Killing privsep child 3021 debug1: audit_event: unhandled event 12 This debug output does not look very helpful ... trying strace on sshd This might be something interesting: the first child forked from sshd dies with SIGSYS when running under 2.31-1: (diff is from 2.30-8 (successful run) to 2.31-1 (failed run)) [...] write(4, "\0\0\0\16ssh-connection\0\0\0\0\0\0\0\0", 26) = 26 write(7, "\0\0\0/\0\0\0\6\0\0\0'input_userauth_reque"..., 51) = 51 write(7, "\0\0\08\0\0\0\7\0\0\0000user_specific_delay:"..., 60) = 60 -clock_gettime(CLOCK_BOOTTIME, {tv_sec=4151042, tv_nsec=403970688}) = 0 +clock_gettime(CLOCK_BOOTTIME, {tv_sec=4150710, tv_nsec=544415415}) = 0 write(7, "\0\0\0X\0\0\0\7\0\0\0Pensure_minimum_time_"..., 92) = 92 -nanosleep({tv_sec=0, tv_nsec=2340526}, NULL) = 0 +clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=2426619}, <unfinished ...>) = ? ++++ killed by SIGSYS +++ If you need me to try something else to get more information, don't hesitate to ask. Andreas