[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
This bug was fixed in the package openssh - 1:8.9p1-3ubuntu0.1 --- openssh (1:8.9p1-3ubuntu0.1) jammy; urgency=medium * d/p/fix-poll-spin.patch: Fix poll(2) spin when a channel's output fd closes without data in the channel buffer. (LP: #1986521) -- Bryce Harrington Tue, 22 Nov 2022 23:38:19 -0800 ** Changed in: openssh (Ubuntu Jammy) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Fix Released Bug description: [Impact] In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. While this is an uncommon error, googling indicates many people have run into it in several different ways. It seems important to get this fixed in stable releases. This is a regression in jammy presumably due to change from select() to poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() spin when a channel's output fd closes..." [2]. 1: https://lwn.net/Articles/885886/ 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) You can shell into the container from a second terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores: $ lxc shell ssh-cpu # htop This should show one CPU pegged at 100% due to the 'ssh localhost' process Next, return to the first terminal, exit out of the sub-ssh session and install the fix: # logout # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 # apt-get full-upgrade -y Now repeat the test in the first terminal window, while viewing htop in the second terminal: # ssh localhost 2> >({exec 1>&2}) [Where Problems Could Occur] While the patch in question is well tested upstream, it has a relatively high line count and as such is difficult to assure correctness by visual code checking. However, it's not clear that the line count could be significantly reduced without risking loss of correctness. Thus this relies more on testing to assure robustness, than on code review. The code involves polling behavior, so issues to watch for would more likely involve process handling, i.e. problems with socket polling. Beyond that, the usual generic issues to watch for - build issues, dependency issues during build or on upgrade, and service restarting. [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
I've verified the test case as written. I reproduced the issue, enabled the -proposed package and did apt-get full-upgrade to pull in the new openssh from -proposed. The CPU usage dropped from 100% to <1% as soon as the operation concluded. ** Tags removed: verification-needed verification-needed-jammy ** Tags added: verification-done verification-done-jammy -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Fix Committed Bug description: [Impact] In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. While this is an uncommon error, googling indicates many people have run into it in several different ways. It seems important to get this fixed in stable releases. This is a regression in jammy presumably due to change from select() to poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() spin when a channel's output fd closes..." [2]. 1: https://lwn.net/Articles/885886/ 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) You can shell into the container from a second terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores: $ lxc shell ssh-cpu # htop This should show one CPU pegged at 100% due to the 'ssh localhost' process Next, return to the first terminal, exit out of the sub-ssh session and install the fix: # logout # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 # apt-get full-upgrade -y Now repeat the test in the first terminal window, while viewing htop in the second terminal: # ssh localhost 2> >({exec 1>&2}) [Where Problems Could Occur] While the patch in question is well tested upstream, it has a relatively high line count and as such is difficult to assure correctness by visual code checking. However, it's not clear that the line count could be significantly reduced without risking loss of correctness. Thus this relies more on testing to assure robustness, than on code review. The code involves polling behavior, so issues to watch for would more likely involve process handling, i.e. problems with socket polling. Beyond that, the usual generic issues to watch for - build issues, dependency issues during build or on upgrade, and service restarting. [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
I can confirm that the updated package works in that the real use case which triggered the original bug (involving a script setting up an ssh tunnel) no longer does. Sorry for the delay in responding -- things have been busy, so I have also not tested the updated package much. I have not updated the verification status as the VM on which I tested this is not entirely up-to-date and standard. I hope that someone is in a better position than I to test more thoroughly. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Fix Committed Bug description: [Impact] In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. While this is an uncommon error, googling indicates many people have run into it in several different ways. It seems important to get this fixed in stable releases. This is a regression in jammy presumably due to change from select() to poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() spin when a channel's output fd closes..." [2]. 1: https://lwn.net/Articles/885886/ 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) You can shell into the container from a second terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores: $ lxc shell ssh-cpu # htop This should show one CPU pegged at 100% due to the 'ssh localhost' process Next, return to the first terminal, exit out of the sub-ssh session and install the fix: # logout # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 # apt-get full-upgrade -y Now repeat the test in the first terminal window, while viewing htop in the second terminal: # ssh localhost 2> >({exec 1>&2}) [Where Problems Could Occur] While the patch in question is well tested upstream, it has a relatively high line count and as such is difficult to assure correctness by visual code checking. However, it's not clear that the line count could be significantly reduced without risking loss of correctness. Thus this relies more on testing to assure robustness, than on code review. The code involves polling behavior, so issues to watch for would more likely involve process handling, i.e. problems with socket polling. Beyond that, the usual generic issues to watch for - build issues, dependency issues during build or on upgrade, and service restarting. [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
Hello Michael, or anyone else affected, Accepted openssh into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssh/1:8.9p1-3ubuntu0.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-jammy. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: openssh (Ubuntu Jammy) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-jammy -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Fix Committed Bug description: [Impact] In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. While this is an uncommon error, googling indicates many people have run into it in several different ways. It seems important to get this fixed in stable releases. This is a regression in jammy presumably due to change from select() to poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() spin when a channel's output fd closes..." [2]. 1: https://lwn.net/Articles/885886/ 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) You can shell into the container from a second terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores: $ lxc shell ssh-cpu # htop This should show one CPU pegged at 100% due to the 'ssh localhost' process Next, return to the first terminal, exit out of the sub-ssh session and install the fix: # logout # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 # apt-get full-upgrade -y Now repeat the test in the first terminal window, while viewing htop in the second terminal: # ssh localhost 2> >({exec 1>&2}) [Where Problems Could Occur] While the patch in question is well tested upstream, it has a relatively high line count and as such is difficult to assure correctness by visual code checking. However, it's not clear that the line count could be significantly reduced without risking loss of correctness. Thus this relies more on testing to assure robustness, than on code review. The code involves polling behavior, so issues to watch for would more likely involve process handling, i.e. problems with socket polling. Beyond that, the usual generic issues to watch for - build issues, dependency issues during build or on upgrade, and service restarting. [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
** Changed in: openssh (Ubuntu Jammy) Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: In Progress Bug description: [Impact] In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. While this is an uncommon error, googling indicates many people have run into it in several different ways. It seems important to get this fixed in stable releases. This is a regression in jammy presumably due to change from select() to poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() spin when a channel's output fd closes..." [2]. 1: https://lwn.net/Articles/885886/ 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) You can shell into the container from a second terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores: $ lxc shell ssh-cpu # htop This should show one CPU pegged at 100% due to the 'ssh localhost' process Next, return to the first terminal, exit out of the sub-ssh session and install the fix: # logout # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 # apt-get full-upgrade -y Now repeat the test in the first terminal window, while viewing htop in the second terminal: # ssh localhost 2> >({exec 1>&2}) [Where Problems Could Occur] While the patch in question is well tested upstream, it has a relatively high line count and as such is difficult to assure correctness by visual code checking. However, it's not clear that the line count could be significantly reduced without risking loss of correctness. Thus this relies more on testing to assure robustness, than on code review. The code involves polling behavior, so issues to watch for would more likely involve process handling, i.e. problems with socket polling. Beyond that, the usual generic issues to watch for - build issues, dependency issues during build or on upgrade, and service restarting. [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
** Description changed: [Impact] - In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. + In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. While this is an uncommon error, googling indicates many people have run into it in several different ways. It seems important to get this fixed in stable releases. This is a regression in jammy presumably due to change from select() to poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() spin when a channel's output fd closes..." [2]. 1: https://lwn.net/Articles/885886/ 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) You can shell into the container from a second terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores: $ lxc shell ssh-cpu # htop This should show one CPU pegged at 100% due to the 'ssh localhost' process Next, return to the first terminal, exit out of the sub-ssh session and install the fix: # logout # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 # apt-get full-upgrade -y Now repeat the test in the first terminal window, while viewing htop in the second terminal: # ssh localhost 2> >({exec 1>&2}) [Where Problems Could Occur] While the patch in question is well tested upstream, it has a relatively high line count and as such is difficult to assure correctness by visual code checking. However, it's not clear that the line count could be significantly reduced without risking loss of correctness. Thus this relies more on testing to assure robustness, than on code review. The code involves polling behavior, so issues to watch for would more likely involve process handling, i.e. problems with socket polling. Beyond that, the usual generic issues to watch for - build issues, dependency issues during build or on upgrade, and service restarting. - [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Triaged Bug description: [Impact] In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. While this is an uncommon error, googling indicates many people have run into it in several different ways. It seems important to get this fixed in stable releases. This is a regression in jammy presumably due to change from select() to poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() spin when a channel's output fd closes..." [2]. 1: https://lwn.net/Articles/885886/ 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # se
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
** Merge proposal linked: https://code.launchpad.net/~bryce/ubuntu/+source/openssh/+git/openssh/+merge/433911 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Triaged Bug description: [Impact] In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. While this is an uncommon error, googling indicates many people have run into it in several different ways. It seems important to get this fixed in stable releases. This is a regression in jammy presumably due to change from select() to poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() spin when a channel's output fd closes..." [2]. 1: https://lwn.net/Articles/885886/ 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) You can shell into the container from a second terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores: $ lxc shell ssh-cpu # htop This should show one CPU pegged at 100% due to the 'ssh localhost' process Next, return to the first terminal, exit out of the sub-ssh session and install the fix: # logout # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 # apt-get full-upgrade -y Now repeat the test in the first terminal window, while viewing htop in the second terminal: # ssh localhost 2> >({exec 1>&2}) [Where Problems Could Occur] While the patch in question is well tested upstream, it has a relatively high line count and as such is difficult to assure correctness by visual code checking. However, it's not clear that the line count could be significantly reduced without risking loss of correctness. Thus this relies more on testing to assure robustness, than on code review. The code involves polling behavior, so issues to watch for would more likely involve process handling, i.e. problems with socket polling. Beyond that, the usual generic issues to watch for - build issues, dependency issues during build or on upgrade, and service restarting. [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
** Description changed: + [Impact] + In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. + + This is a regression in jammy presumably due to change from select() to + poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream + commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() + spin when a channel's output fd closes..." [2]. + + 1: https://lwn.net/Articles/885886/ + 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c + [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) - You can shell into the container from a second terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores: $ lxc shell ssh-cpu # htop This should show one CPU pegged at 100% due to the 'ssh localhost' process Next, return to the first terminal, exit out of the sub-ssh session and install the fix: # logout # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 # apt-get full-upgrade -y Now repeat the test in the first terminal window, while viewing htop in the second terminal: # ssh localhost 2> >({exec 1>&2}) + [Where Problems Could Occur] + + While the patch in question is well tested upstream, it has a relatively + high line count and as such is difficult to assure correctness by visual + code checking. However, it's not clear that the line count could be + significantly reduced without risking loss of correctness. Thus this + relies more on testing to assure robustness, than on code review. + + The code involves polling behavior, so issues to watch for would more + likely involve process handling, i.e. problems with socket polling. + + Beyond that, the usual generic issues to watch for - build issues, + dependency issues during build or on upgrade, and service restarting. + + [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Triaged Bug description: [Impact] In certain edge cases where the terminal goes away while an ssh process is running, ssh can be left consuming 100% CPU. This increases processing costs for cloud users and wastes energy. While this is an uncommon error, googling indicates many people have run into it in several different ways. It seems important to get this fixed in stable releases. This is a regression in jammy presumably due to change from select() to poll() (see OpennSSH 8.9 Release Announcement [1] ), fixed by upstream commit d6556de1db0822c76ba2745cf5c097d9472adf7c "upstream: fix poll() spin when a channel's output fd closes..." [2]. 1: https://lwn.net/Articles/885886/ 2. https://github.com/openssh/openssh-portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) You can shel
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
I've reproduced the issue and confirmed the PPA fixes it, as expected. ** Description changed: + [Test Case] + $ lxc launch ubuntu-daily:jammy ssh-cpu + $ lxc shell ssh-cpu + + # passwd -d root + # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa + + # cat << EOF >>/etc/ssh/ssh_config + StrictHostKeyChecking accept-new + EOF + + # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config + # cat << EOF >>/etc/ssh/sshd_config + PermitRootLogin yes + PubkeyAuthentication yes + PermitEmptyPasswords yes + PasswordAuthentication yes + ChallengeResponseAuthentication no + UsePAM no + EOF + + # systemctl restart sshd + + # ssh localhost 2> >({exec 1>&2}) + + + You can shell into the container from a second terminal and use "htop" + to verify that ssh is using 100% of one of the CPU cores: + + $ lxc shell ssh-cpu + # htop + + This should show one CPU pegged at 100% due to the 'ssh localhost' + process + + Next, return to the first terminal, exit out of the sub-ssh session and + install the fix: + + # logout + # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 + # apt-get full-upgrade -y + + Now repeat the test in the first terminal window, while viewing htop in + the second terminal: + + # ssh localhost 2> >({exec 1>&2}) + + [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Triaged Bug description: [Test Case] $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # passwd -d root # ssh-keygen -t rsa -N '' -f /root/.ssh/id_rsa # cat << EOF >>/etc/ssh/ssh_config StrictHostKeyChecking accept-new EOF # sed -ri 's/^PasswordAuthentication/#PasswordAuthentication/' /etc/ssh/sshd_config # cat << EOF >>/etc/ssh/sshd_config PermitRootLogin yes PubkeyAuthentication yes PermitEmptyPasswords yes PasswordAuthentication yes ChallengeResponseAuthentication no UsePAM no EOF # systemctl restart sshd # ssh localhost 2> >({exec 1>&2}) You can shell into the container from a second terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores: $ lxc shell ssh-cpu # htop This should show one CPU pegged at 100% due to the 'ssh localhost' process Next, return to the first terminal, exit out of the sub-ssh session and install the fix: # logout # add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 # apt-get full-upgrade -y Now repeat the test in the first terminal window, while viewing htop in the second terminal: # ssh localhost 2> >({exec 1>&2}) [Original Report] The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
I've posted a PPA with the patch to fix this issue here: https://launchpad.net/~bryce/+archive/ubuntu/openssh-sru-lp1986521 This can be installed via: $ sudo add-apt-repository -yus ppa:bryce/openssh-sru-lp1986521 $ sudo apt-get install openssh Can you please upgrade to this and verify it fixes the reported issue? -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Triaged Bug description: The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
** Changed in: openssh (Ubuntu Jammy) Assignee: (unassigned) => Bryce Harrington (bryce) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Triaged Bug description: The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
** Tags added: server-todo -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Triaged Bug description: The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
Thanks for taking the time to report the bug and make Ubuntu better. I can reproduce the bug using your testcase, but that requires a VM with a graphical environment installed. Another way to reproduce the bug is (from the upstream bug report): $ lxc launch ubuntu-daily:jammy ssh-cpu $ lxc shell ssh-cpu # ssh HOST 2> >({exec 1>&2}) You can shell into the container from another terminal and use "htop" to verify that ssh is using 100% of one of the CPU cores. This seems to have been fixed upstream by the following commit: https://github.com/openssh/openssh- portable/commit/d6556de1db0822c76ba2745cf5c097d9472adf7c I confirmed that this only happens on Jammy. Focal and Kinetic are not affected. ** Also affects: openssh (Ubuntu Jammy) Importance: Undecided Status: New ** Changed in: openssh (Ubuntu) Status: New => Fix Released ** Changed in: openssh (Ubuntu Jammy) Status: New => Triaged ** Changed in: openssh (Ubuntu Jammy) Importance: Undecided => Medium ** Bug watch added: OpenSSH Portable Bugzilla #3405 https://bugzilla.mindrot.org/show_bug.cgi?id=3405 ** Also affects: openssh via https://bugzilla.mindrot.org/show_bug.cgi?id=3405 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in portable OpenSSH: Unknown Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Jammy: Triaged Bug description: The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1986521] Re: ssh client spins if output fd closed
And I can confirm that the patch at https://bugzilla.mindrot.org/attachment.cgi?id=3581 applies cleanly and fixes this issue. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/1986521 Title: ssh client spins if output fd closed Status in openssh package in Ubuntu: New Bug description: The OpenSSH package 8.9p1 as shipped with U22.04 (8.9p1-3) suffers from the bug described at https://bugzilla.mindrot.org/show_bug.cgi?id=3411 and https://bugzilla.mindrot.org/show_bug.cgi?id=3405 A command such as "xterm -e 'ssh -f remote.host sleep 60'" will pop up an xterm, ask for whatever authentication is needed, close the xterm, and leave the ssh client spinning consuming CPU time for 60 seconds before it exits. It should leave the ssh client idle for 60 seconds. Many uses of ssh to launch graphical applications will be caught by this bug. This is fixed in OpenSSH 9.0p1 as the first bugfix listed in its release notes at https://www.openssh.com/txt/release-9.0 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1986521/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp