[Touch-packages] [Bug 2043114] Re: sshd segmentation fault on 20.04.6 (focal)
We’re seeing it happen on both bare metal and virtual machines. The VMs are libvirt/QEMU/KVM managed by OpenNebula. The hypervisor is running AlmaLinux 8 with the 4.18.0-477.15.1.el8_8.x86_64 kernel. The VMs have 2 physical/8 vCPUs allocated to them. The VM I used for the reproduction has 2GB of RAM. The VMs where we saw the problem initially have 22GB, but the setup there is a bit more complicated - there are various cgroup v1 limits, such as 2.5GB for system.slice, and 2GB for user slice. However none of that is necessary to reproduce the issue. Since the initial report, we've done a lot of test runs with INFO-level logging on 20.04, 18.04, 22.04, as well as RedHat-family distributions, and we haven’t seen the issue again. Could be some memory corruption bug that is present only on 20.04, I guess. So far I’ve been unable to reproduce it on a 22.04 VM with DEBUG logging on. Thanks. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2043114 Title: sshd segmentation fault on 20.04.6 (focal) Status in openssh package in Ubuntu: Confirmed Bug description: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 CasperMD5CheckResult: skip DistroRelease: Ubuntu 20.04 Package: openssh-server 1:8.2p1-4ubuntu0.9 PackageArchitecture: amd64 ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210 Tags: focal Uname: Linux 5.4.0-128-generic x86_64 UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago) UserGroups: N/A _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2043114] Re: sshd segmentation fault on 20.04.6 (focal)
Yes, seems pretty consistent. I just tried it again before I write out
the reproduction steps, and it took maybe a couple of minutes.
The configuration is basically just:
sed -i.bak 's/#LogLevel INFO/LogLevel DEBUG/g' /etc/ssh/sshd_config
systemctl restart sshd
As for making enough connections to cause the issue, I use something
along the lines of:
parallel -j 99 -N0 "ssh root@example 'mount; sleep 1; cat /proc/cpuinfo;
free -h; dd if=/dev/zero of=/dev/null bs=1 count=8192; mount -av; sleep
$(($RANDOM % 5)); lshw'" ::: {1..5000}
It is essentially running random commands to simulate some sort of
activity on the machine. The amount of parallel connections (`-j 99`)
might need to be adjusted - too little connections won't trigger the
bug, too many can cause an OOM situation.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2043114
Title:
sshd segmentation fault on 20.04.6 (focal)
Status in openssh package in Ubuntu:
Confirmed
Bug description:
We have a physical server running Ubuntu 20.04.6 LTS (amd64) and
openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation
fault on remote login with key authentication:
[193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53
sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000]
We’ve changed only the following values in the stock sshd_config file:
LogLevel DEBUG
PasswordAuthentication no
MaxStartups 100:30:100
The server is used for automated software testing, and sometimes our test
suite might make a large amount of SSH connections in a short period of time,
which seems to be correlated with the crashes. But at the same time, I have to
note that the connection count was not near the MaxStartups limit, and we’ve
had crashes before adding that setting.
Since the backtrace shows the debug logging function in the stack, we’re
currently experimenting with using `LogLevel INFO` to try and isolate the issue.
I am attaching the backtrace. I could provide the full dump file,
although I am hesitant due to the possibility of private keys or other
sensitive information leaking.
# apt-cache policy openssh-server
openssh-server:
Installed: 1:8.2p1-4ubuntu0.9
Candidate: 1:8.2p1-4ubuntu0.9
Version table:
*** 1:8.2p1-4ubuntu0.9 500
500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main
amd64 Packages
500 http://security.ubuntu.com/ubuntu focal-security/main amd64
Packages
100 /var/lib/dpkg/status
1:8.2p1-4 500
500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64
Packages
---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.27
Architecture: amd64
CasperMD5CheckResult: skip
DistroRelease: Ubuntu 20.04
Package: openssh-server 1:8.2p1-4ubuntu0.9
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210
Tags: focal
Uname: Linux 5.4.0-128-generic x86_64
UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago)
UserGroups: N/A
_MarkForUpload: True
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2043114] Re: sshd segmentation fault on 20.04.6 (focal)
After some more "stress testing" with GNU parallel, I was able to reproduce the segmentation fault, too. See attachment. ** Attachment added: "sshd segmentation fault crash report" https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+attachment/5717935/+files/_usr_sbin_sshd.0.crash -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2043114 Title: sshd segmentation fault on 20.04.6 (focal) Status in openssh package in Ubuntu: Confirmed Bug description: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 CasperMD5CheckResult: skip DistroRelease: Ubuntu 20.04 Package: openssh-server 1:8.2p1-4ubuntu0.9 PackageArchitecture: amd64 ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210 Tags: focal Uname: Linux 5.4.0-128-generic x86_64 UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago) UserGroups: N/A _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2043114] Re: sshd segmentation fault on 20.04.6 (focal)
Apologies for the multiple messages, but I was under the impression that apport-collect would upload the actual core dump file here :) ** Attachment added: "sshd abort crash report" https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+attachment/5717896/+files/_usr_sbin_sshd.0.crash -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2043114 Title: sshd segmentation fault on 20.04.6 (focal) Status in openssh package in Ubuntu: Confirmed Bug description: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 CasperMD5CheckResult: skip DistroRelease: Ubuntu 20.04 Package: openssh-server 1:8.2p1-4ubuntu0.9 PackageArchitecture: amd64 ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210 Tags: focal Uname: Linux 5.4.0-128-generic x86_64 UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago) UserGroups: N/A _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2043114] Dependencies.txt
apport information ** Attachment added: "Dependencies.txt" https://bugs.launchpad.net/bugs/2043114/+attachment/5717893/+files/Dependencies.txt -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2043114 Title: sshd segmentation fault on 20.04.6 (focal) Status in openssh package in Ubuntu: Confirmed Bug description: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 CasperMD5CheckResult: skip DistroRelease: Ubuntu 20.04 Package: openssh-server 1:8.2p1-4ubuntu0.9 PackageArchitecture: amd64 ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210 Tags: focal Uname: Linux 5.4.0-128-generic x86_64 UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago) UserGroups: N/A _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2043114] ProcEnviron.txt
apport information ** Attachment added: "ProcEnviron.txt" https://bugs.launchpad.net/bugs/2043114/+attachment/5717895/+files/ProcEnviron.txt -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2043114 Title: sshd segmentation fault on 20.04.6 (focal) Status in openssh package in Ubuntu: Confirmed Bug description: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 CasperMD5CheckResult: skip DistroRelease: Ubuntu 20.04 Package: openssh-server 1:8.2p1-4ubuntu0.9 PackageArchitecture: amd64 ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210 Tags: focal Uname: Linux 5.4.0-128-generic x86_64 UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago) UserGroups: N/A _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2043114] ProcCpuinfoMinimal.txt
apport information ** Attachment added: "ProcCpuinfoMinimal.txt" https://bugs.launchpad.net/bugs/2043114/+attachment/5717894/+files/ProcCpuinfoMinimal.txt -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2043114 Title: sshd segmentation fault on 20.04.6 (focal) Status in openssh package in Ubuntu: Confirmed Bug description: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 CasperMD5CheckResult: skip DistroRelease: Ubuntu 20.04 Package: openssh-server 1:8.2p1-4ubuntu0.9 PackageArchitecture: amd64 ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210 Tags: focal Uname: Linux 5.4.0-128-generic x86_64 UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago) UserGroups: N/A _MarkForUpload: True To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2043114] Re: sshd segmentation fault on 20.04.6 (focal)
While trying to reproduce the segfault, I encountered another crash, an abort in malloc this time. I cannot be absolutely sure, but it seems that they might be related, so I am uploading the report now. Edit: forgot to mention, it seems that the abort has also occurred on the physical machine that I initially reported the crash on. ** Tags added: apport-collected focal ** Description changed: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages + --- + ProblemType: Bug + ApportVersion: 2.20.11-0ubuntu27.27 + Architecture: amd64 + CasperMD5CheckResult: skip + DistroRelease: Ubuntu 20.04 + Package: openssh-server 1:8.2p1-4ubuntu0.9 + PackageArchitecture: amd64 + ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210 + Tags: focal + Uname: Linux 5.4.0-128-generic x86_64 + UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago) + UserGroups: N/A + _MarkForUpload: True -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2043114 Title: sshd segmentation fault on 20.04.6 (focal) Status in openssh package in Ubuntu: Confirmed Bug description: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu27.27 Architecture: amd64 CasperMD5CheckResult: skip DistroRelease: Ubuntu 20.04 Package: openssh-server 1:8.2p1-4ubuntu0.9 PackageArchitecture: amd64 ProcVersionSignature: Ubuntu 5.4.0-128.144-generic 5.4.210 Tags: focal Uname: Linux 5.4.0-128-generic x86_64 UpgradeStatus: Upgraded to focal on 2021-01-13 (1030 days ago) UserGroups: N/A _MarkForUpload: True To manage notifications about this bug go to:
[Touch-packages] [Bug 2043114] Re: sshd segmentation fault on 20.04.6 (focal)
Thanks for getting back to me! 1. The crash seems to happen in a large percentage of the runs of a particular test that does a lot of rapid SSH connections to the machine. Sadly, I don’t have an exact number, but 50% probably wouldn’t be an exaggeration. It was also happening in the setup part of the test harness when it was making a large amount of parallel connections, but we spread those out a bit. 2. The connections all happen from a single test machine. I could try to test if this is repeatable when they’re spread over multiple sources, but that’s not how our test suite works. As for the debugging information, I will first try to reproduce the crash on a throwaway machine (VM shouldn’t make a difference, I think), as to avoid any potential leaks of host keys or other sensitive information. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2043114 Title: sshd segmentation fault on 20.04.6 (focal) Status in openssh package in Ubuntu: Confirmed Bug description: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2043114] [NEW] sshd segmentation fault on 20.04.6 (focal)
Public bug reported: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages ** Affects: openssh (Ubuntu) Importance: Undecided Status: Confirmed ** Tags: segfault segv sshd ** Attachment added: "backtrace" https://bugs.launchpad.net/bugs/2043114/+attachment/5717504/+files/backtrace -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2043114 Title: sshd segmentation fault on 20.04.6 (focal) Status in openssh package in Ubuntu: Confirmed Bug description: We have a physical server running Ubuntu 20.04.6 LTS (amd64) and openssh-server 1:8.2p1-4ubuntu0.9. Sometimes sshd crashes with a segmentation fault on remote login with key authentication: [193107.651745] sshd[1229630]: segfault at 5557eba6a008 ip 7f2326a2ca53 sp 7ffcba40c510 error 4 in libc-2.31.so[7f23269b8000+178000] We’ve changed only the following values in the stock sshd_config file: LogLevel DEBUG PasswordAuthentication no MaxStartups 100:30:100 The server is used for automated software testing, and sometimes our test suite might make a large amount of SSH connections in a short period of time, which seems to be correlated with the crashes. But at the same time, I have to note that the connection count was not near the MaxStartups limit, and we’ve had crashes before adding that setting. Since the backtrace shows the debug logging function in the stack, we’re currently experimenting with using `LogLevel INFO` to try and isolate the issue. I am attaching the backtrace. I could provide the full dump file, although I am hesitant due to the possibility of private keys or other sensitive information leaking. # apt-cache policy openssh-server openssh-server: Installed: 1:8.2p1-4ubuntu0.9 Candidate: 1:8.2p1-4ubuntu0.9 Version table: *** 1:8.2p1-4ubuntu0.9 500 500 http://mirrors.storpool.com/ubuntu/archive focal-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages 100 /var/lib/dpkg/status 1:8.2p1-4 500 500 http://mirrors.storpool.com/ubuntu/archive focal/main amd64 Packages To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2043114/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
