Re: Ssh.. can we please have HPN back?
On 23/5/17 12:13 am, Allan Jude wrote: On 2017-05-22 03:50, Julian Elischer wrote: On 22/5/17 2:20 pm, Allan Jude wrote: On 2017-05-18 22:28, Julian Elischer wrote: So after stripping out the HPN version of ssh from our product becasue "it was no longer needed" we dicovered that we were premature in doing so. Apparently ssh still really needs HPN to get any throughput at all when there are latencies involved. For example, with HPN we get 13MB/sec between the Azure US west Data center and the Azure East data center.But the standard ssh in 10.3 (with HPN stripped out) can barely manage 2MB/sec transfers. I did ask at the time whether it was proved that the new ssh didn't require the HPN changes, and was assured, "no" but it would appear that the picture isn't as clear. tht seems silly to have to import the port when we have what would otherwise be a perfectly good ssh as part of hte system, and it's really annoying having to specify /usr/local/bin/scp or /usr/local/bin/ssh in every script. So can we please have the latest version of the HPN changes back in the default system please? It seem rather odd that the upstream openssh has had this problem for SO LONG and not fixed it. Julian ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" I have this stand-alone patch ready now: https://github.com/openssh/openssh-portable/compare/master...allanjude:V_7_5_dynamic_window In my benchmarks with 100ms of latency (from dummynet) is increases SSH send throughput from 1 megabyte/sec to 225 megabytes/sec provided a large enough socket buffer. Still seeing lesser performance on the recv case, working on it. We have two patches of our own that upstream have ignored an option to set NoDelay and a pushwindow setting option (though I'm not sure the second is operational in the current code, it does apply with no errors...) The NoDelay options massively speeds up the case where you have chatty back and forth traffic (client/server) through a ssh tunnel. Are your changes against the sources in current? what about stable/10? That change is against upstream's latest release, but should apply cleanly to any version of OpenSSH from the past 10 years that does not have the HPN patches applied already. However, the function channel_check_window() in stable/10 is currently the same as the latest upstream since the commit that removed HPN. The function is actually unchanged from upstream if you go back far enough to before we added HPN as well. If you revert both HPN commits, the most recent change to that function is Aug 2008, when upstream introduced the 'move the window forward every 3 packets' thing that seams to be the main cause of the window scaling problem in the first place. I think the nodelay option might be what mitigates that, as it causes rapid back and forth and never allows the window to grow to even the 2mb allowed by SSH since version 4.7 great I'll look at applying it at $JOB and let you know what happens .. assuming I can get the patch downloaded and applied. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Ssh.. can we please have HPN back?
On 2017-05-22 03:50, Julian Elischer wrote: > On 22/5/17 2:20 pm, Allan Jude wrote: >> On 2017-05-18 22:28, Julian Elischer wrote: >>> So after stripping out the HPN version of ssh from our product becasue >>> "it was no longer needed" we dicovered that we were premature in >>> doing so. >>> Apparently ssh still really needs HPN to get any throughput at all when >>> there are latencies involved. >>> >>> >>> For example, with HPN we get 13MB/sec between the Azure US west >>> Data center and the Azure East data center.But the standard ssh in 10.3 >>> (with HPN stripped out) can barely manage 2MB/sec transfers. >>> >>> I did ask at the time whether it was proved that the new ssh didn't >>> require the HPN changes, >>> and was assured, "no" but it would appear that the picture isn't as >>> clear. >>> tht seems silly to have to import the port when we have what would >>> otherwise be a >>> perfectly good ssh as part of hte system, and it's really annoying >>> having to specify >>> /usr/local/bin/scp or /usr/local/bin/ssh in every script. >>> >>> So can we please have the latest version of the HPN changes back in the >>> default system please? >>> It seem rather odd that the upstream openssh has had this problem for SO >>> LONG and not fixed it. >>> >>> Julian >>> >>> >>> ___ >>> freebsd-current@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-current >>> To unsubscribe, send any mail to >>> "freebsd-current-unsubscr...@freebsd.org" >> I have this stand-alone patch ready now: >> >> https://github.com/openssh/openssh-portable/compare/master...allanjude:V_7_5_dynamic_window >> >> >> In my benchmarks with 100ms of latency (from dummynet) is increases SSH >> send throughput from 1 megabyte/sec to 225 megabytes/sec provided a >> large enough socket buffer. >> >> Still seeing lesser performance on the recv case, working on it. > > We have two patches of our own that upstream have ignored an option to > set NoDelay and a pushwindow setting option > (though I'm not sure the second is operational in the current code, it > does apply with no errors...) > > The NoDelay options massively speeds up the case where you have chatty > back and forth traffic (client/server) through a ssh tunnel. > > Are your changes against the sources in current? what about stable/10? > >> > That change is against upstream's latest release, but should apply cleanly to any version of OpenSSH from the past 10 years that does not have the HPN patches applied already. However, the function channel_check_window() in stable/10 is currently the same as the latest upstream since the commit that removed HPN. The function is actually unchanged from upstream if you go back far enough to before we added HPN as well. If you revert both HPN commits, the most recent change to that function is Aug 2008, when upstream introduced the 'move the window forward every 3 packets' thing that seams to be the main cause of the window scaling problem in the first place. I think the nodelay option might be what mitigates that, as it causes rapid back and forth and never allows the window to grow to even the 2mb allowed by SSH since version 4.7 -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: Ssh.. can we please have HPN back?
On 22/5/17 2:20 pm, Allan Jude wrote: On 2017-05-18 22:28, Julian Elischer wrote: So after stripping out the HPN version of ssh from our product becasue "it was no longer needed" we dicovered that we were premature in doing so. Apparently ssh still really needs HPN to get any throughput at all when there are latencies involved. For example, with HPN we get 13MB/sec between the Azure US west Data center and the Azure East data center.But the standard ssh in 10.3 (with HPN stripped out) can barely manage 2MB/sec transfers. I did ask at the time whether it was proved that the new ssh didn't require the HPN changes, and was assured, "no" but it would appear that the picture isn't as clear. tht seems silly to have to import the port when we have what would otherwise be a perfectly good ssh as part of hte system, and it's really annoying having to specify /usr/local/bin/scp or /usr/local/bin/ssh in every script. So can we please have the latest version of the HPN changes back in the default system please? It seem rather odd that the upstream openssh has had this problem for SO LONG and not fixed it. Julian ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" I have this stand-alone patch ready now: https://github.com/openssh/openssh-portable/compare/master...allanjude:V_7_5_dynamic_window In my benchmarks with 100ms of latency (from dummynet) is increases SSH send throughput from 1 megabyte/sec to 225 megabytes/sec provided a large enough socket buffer. Still seeing lesser performance on the recv case, working on it. We have two patches of our own that upstream have ignored an option to set NoDelay and a pushwindow setting option (though I'm not sure the second is operational in the current code, it does apply with no errors...) The NoDelay options massively speeds up the case where you have chatty back and forth traffic (client/server) through a ssh tunnel. Are your changes against the sources in current? what about stable/10? ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: Ssh.. can we please have HPN back?
On 2017-05-18 22:28, Julian Elischer wrote: > So after stripping out the HPN version of ssh from our product becasue > "it was no longer needed" we dicovered that we were premature in doing so. > Apparently ssh still really needs HPN to get any throughput at all when > there are latencies involved. > > > For example, with HPN we get 13MB/sec between the Azure US west > Data center and the Azure East data center.But the standard ssh in 10.3 > (with HPN stripped out) can barely manage 2MB/sec transfers. > > I did ask at the time whether it was proved that the new ssh didn't > require the HPN changes, > and was assured, "no" but it would appear that the picture isn't as clear. > tht seems silly to have to import the port when we have what would > otherwise be a > perfectly good ssh as part of hte system, and it's really annoying > having to specify > /usr/local/bin/scp or /usr/local/bin/ssh in every script. > > So can we please have the latest version of the HPN changes back in the > default system please? > It seem rather odd that the upstream openssh has had this problem for SO > LONG and not fixed it. > > Julian > > > ___ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org" I have this stand-alone patch ready now: https://github.com/openssh/openssh-portable/compare/master...allanjude:V_7_5_dynamic_window In my benchmarks with 100ms of latency (from dummynet) is increases SSH send throughput from 1 megabyte/sec to 225 megabytes/sec provided a large enough socket buffer. Still seeing lesser performance on the recv case, working on it. -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: Ssh.. can we please have HPN back?
On 2017-05-19 00:01, Rodney W. Grimes wrote: >> So after stripping out the HPN version of ssh from our product becasue >> "it was no longer needed" we dicovered that we were premature in doing so. >> Apparently ssh still really needs HPN to get any throughput at all when >> there are latencies involved. >> >> >> For example, with HPN we get 13MB/sec between the Azure US west >> Data center and the Azure East data center.But the standard ssh in 10.3 >> (with HPN stripped out) can barely manage 2MB/sec transfers. >> >> I did ask at the time whether it was proved that the new ssh didn't >> require the HPN changes, >> and was assured, "no" but it would appear that the picture isn't as clear. >> tht seems silly to have to import the port when we have what would >> otherwise be a >> perfectly good ssh as part of hte system, and it's really annoying >> having to specify >> /usr/local/bin/scp or /usr/local/bin/ssh in every script. >> >> So can we please have the latest version of the HPN changes back in >> the default system please? >> It seem rather odd that the upstream openssh has had this problem for >> SO LONG and not fixed it. > > Allan Jude has recently done a bunch of work on this though I do > not know its current status of being either upstreamed (I know > some of it well not be accepted from conversations with Allan) > or commited to the tree. > I hope to have the most important part of the patch rebased on the latest upstream version of OpenSSH by the end of this weekend. The versions I built and benchmarked for AsiaBSDCon were based on the HPN patched openssh-portable from ports, but I think the change required to actually make ssh not suck will only be a few lines, and will be acceptable by upstream. The big issue is in the channel_check_window() The condition for growing the SSH window is if we have received 3 times the max packet size. For some reason this constant small growth of the SSH window never lets the TCP socket buffer grow. This behaviour was added in OpenSSH 4.7 (Jun 2007), and makes sense for interactive ssh sessions. More detail in my paper, see the 'Broken Windows' chapter: http://allanjude.com/bsd/AsiaBSDCon2017_-_SSH_Performance.pdf Anyway, my fix was to only allow that condition to result in moving the SSH window forward if packet_is_interactive(). In the bulk transfer case, it falls back to using the other (original) condition of 'half of the local window max has been consumed'. The other condition is a modified version of one of the HPN patches. We do a getsockopt SO_RCVBUF to check the size of the tcp socket buffer, and if the remaining part of the SSH window has fall below the size of the socket buffer, we grow the SSH window by 150%, up to SSHBUF_SIZE_MAX https://github.com/rapier1/openssh-portable/compare/master...allanjude:dynamic_window_fix.diff I just need to rejigger it a bit so it doesn't depend on the HPN support functions and becomes an independent patch. Figure 3 and Figure 4 show what difference HPN makes when you add latency, but also show without my patch, HPN only solves the recv case, not the send case. -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: Ssh.. can we please have HPN back?
> So after stripping out the HPN version of ssh from our product becasue > "it was no longer needed" we dicovered that we were premature in doing so. > Apparently ssh still really needs HPN to get any throughput at all when > there are latencies involved. > > > For example, with HPN we get 13MB/sec between the Azure US west > Data center and the Azure East data center.But the standard ssh in 10.3 > (with HPN stripped out) can barely manage 2MB/sec transfers. > > I did ask at the time whether it was proved that the new ssh didn't > require the HPN changes, > and was assured, "no" but it would appear that the picture isn't as clear. > tht seems silly to have to import the port when we have what would > otherwise be a > perfectly good ssh as part of hte system, and it's really annoying > having to specify > /usr/local/bin/scp or /usr/local/bin/ssh in every script. > > So can we please have the latest version of the HPN changes back in > the default system please? > It seem rather odd that the upstream openssh has had this problem for > SO LONG and not fixed it. Allan Jude has recently done a bunch of work on this though I do not know its current status of being either upstreamed (I know some of it well not be accepted from conversations with Allan) or commited to the tree. -- Rod Grimes rgri...@freebsd.org ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Ssh.. can we please have HPN back?
So after stripping out the HPN version of ssh from our product becasue "it was no longer needed" we dicovered that we were premature in doing so. Apparently ssh still really needs HPN to get any throughput at all when there are latencies involved. For example, with HPN we get 13MB/sec between the Azure US west Data center and the Azure East data center.But the standard ssh in 10.3 (with HPN stripped out) can barely manage 2MB/sec transfers. I did ask at the time whether it was proved that the new ssh didn't require the HPN changes, and was assured, "no" but it would appear that the picture isn't as clear. tht seems silly to have to import the port when we have what would otherwise be a perfectly good ssh as part of hte system, and it's really annoying having to specify /usr/local/bin/scp or /usr/local/bin/ssh in every script. So can we please have the latest version of the HPN changes back in the default system please? It seem rather odd that the upstream openssh has had this problem for SO LONG and not fixed it. Julian ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
