Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
niedz., 2 wrz 2018 o 13:33 Jan Ehrhardt via curl-library napisał(a): > > Do you have a compiled version somewhere? I'm hacking this on Linux, I don't have a proper testing environment on my Windows machine. > I tried to build my own with the 3 patches: > > 1. winsock > 2. oploadbuffer 512 KB > 3. sftp writeback in libssh2 > > Dissappointing results for a 274MB sftp upload against a remote CoreFTP > server: > - bash / ssh / lftp: 68 seconds > - openssh portable sftp: 50 seconds > - curl triple patched: 734 seconds > ping of the remote machine: 3-4 ms. That's like 5MB/s best case, 300KB/s worst case; what's your limiting factor? Network? Daniel --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Jan Ehrhardt via curl-library (Sun, 26 Aug 2018 16:03:16 +0200): >Daniel Jelinski via curl-library (Sun, 26 Aug 2018 09:09:58 +0200): >>added 100ms: original 310kB/sec, patched 9900kB/sec > >Impressive results! Do you have a compiled version somewhere? I tried to build my own with the 3 patches: 1. winsock 2. oploadbuffer 512 KB 3. sftp writeback in libssh2 Dissappointing results for a 274MB sftp upload against a remote CoreFTP server: - bash / ssh / lftp: 68 seconds - openssh portable sftp: 50 seconds - curl triple patched: 734 seconds ping of the remote machine: 3-4 ms. Used cipher in openssh portable sftp: AES128-CBC. In production we have an old version of the CoreFTP server. On our dev machine I am going to install the most recent CoreFTP server version. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Jan Ehrhardt via curl-library (Sun, 26 Aug 2018 16:28:47 +0200): >Daniel Jelinski via curl-library (Sun, 26 Aug 2018 09:09:58 +0200): >>The patched version is CPU-bound on lower latencies; when the latency >>goes higher, SSH window becomes the limiting factor. I read that HPN >>SSH should do better on high latency links, didn't try it out. > >A lot of PR's for HPN-SSH came from @allanjude. Interesting stuff: >http://www.allanjude.com/bsd/AsiaBSDCon2017_-_SSH_Performance.pdf In May 2017 Allan Jude posted a new HPN patch here: https://lists.freebsd.org/pipermail/freebsd-current/2017-May/066015.html Quote: > In my benchmarks with 100ms of latency (from dummynet) is increases SSH > send throughput from 1 megabyte/sec to 225 megabytes/sec provided a > large enough socket buffer. I cannot reproduce that. I applied a modified version of this patch to OpenSSH-portable for Windows: https://github.com/Jan-E/openssh-portable/commit/e5c711f66aa3a14980dc7e0ca8504d0bda614fa6 For anyone that wants to try it: https://phpdev.toolsforresearch.com/expect.7z The scripting is done with Expect for Windows: https://github.com/zetamatta/expect/issues/8#issuecomment-416445736 -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Daniel Jelinski via curl-library (Sun, 26 Aug 2018 09:09:58 +0200): >The patched version is CPU-bound on lower latencies; when the latency >goes higher, SSH window becomes the limiting factor. I read that HPN >SSH should do better on high latency links, didn't try it out. A lot of PR's for HPN-SSH came from @allanjude. Interesting stuff: http://www.allanjude.com/bsd/AsiaBSDCon2017_-_SSH_Performance.pdf -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Daniel Jelinski via curl-library (Sun, 26 Aug 2018 09:09:58 +0200): >added 100ms: original 310kB/sec, patched 9900kB/sec Impressive results! >The patched version is CPU-bound on lower latencies; when the latency >goes higher, SSH window becomes the limiting factor. I read that HPN >SSH should do better on high latency links, didn't try it out. I had also been looking at HPN SSH. If you try that you should increase the buffersize as well: https://github.com/rapier1/openssh-portable/pull/11 >https://github.com/djelinski/libssh2/commit/c321b5d3a0ed964b291c710179dde7385e514ef7 >It's not ready for inclusion; it requires cleanup, and breaks SSL >backends other than openSSL with AES-CTR support. There is an open issue with AES-CTR in HPN SSH: https://github.com/rapier1/openssh-portable/issues/13#issuecomment-279778429 I did not run any tests yet. I will see if I can build it for Windows under Cygwin. https://github.com/rapier1/hpn-ssh/tree/master/contrib/cygwin -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
niedz., 26 sie 2018 o 03:36 Jan Ehrhardt via curl-library napisał(a): > Do you have any stats about the performanceimprovement? Workbench: I am running curl against openssh 7.2 shipped with Ubuntu. The server is running on the same machine as the client. I am uploading 1GB file to /dev/null on the server. My laptop is running a 2nd gen i3 CPU without hardware AES support. I'm tweaking network latency using tc. Curl speed was taken from curl-reported average when the transfer took less than a minute. For slower transfers I took a representative value from the momentary upload speed. Results with 16kB curl upload buffer: no delay added: original 12MB/sec, patched 33 MB/sec added 20 ms: original 390kB/sec, patched 26.6 MB/sec added 100ms: original 81 kB/sec, patched 9700 kB/sec Results with 64kB curl upload buffer: no delay added: original 27MB/sec, patched 35 MB/sec added 20 ms: original 1400kB/sec, patched 30MB/sec added 100ms: original 310kB/sec, patched 9900kB/sec The patched version is CPU-bound on lower latencies; when the latency goes higher, SSH window becomes the limiting factor. I read that HPN SSH should do better on high latency links, didn't try it out. Separately I have a work-in-progress patch that improves the CPU usage; I was able to reach 80MB/s transfers with that patch applied. You can see it here: https://github.com/djelinski/libssh2/commit/c321b5d3a0ed964b291c710179dde7385e514ef7 It's not ready for inclusion; it requires cleanup, and breaks SSL backends other than openSSL with AES-CTR support. If you have openSSL with AES support, it might be worth a try. --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Daniel Jelinski via curl-library (Sun, 26 Aug 2018 01:07:14 +0200): >sob., 11 sie 2018 o 01:05 Daniel Stenberg napisa?(a): >> It would require that libssh2 provides such an API, which it currently >> doesn't >> (and I don't know anyone working on it). > >Sent a PR to libssh2 for that: https://github.com/libssh2/libssh2/pull/264 Do you have any stats about the performanceimprovement? -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
On Sat, Aug 25, 2018, 4:10 PM Daniel Jeliński via curl-library < curl-library@cool.haxx.se> wrote: > sob., 11 sie 2018 o 01:05 Daniel Stenberg napisał(a): > > It would require that libssh2 provides such an API, which it currently > doesn't > > (and I don't know anyone working on it). > > Sent a PR to libssh2 for that: https://github.com/libssh2/libssh2/pull/264 > > > I still haven't checked what libssh provides or how it performs in > comparison. > > Just checked, looks about the same here. > > --- > Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library > Etiquette: https://curl.haxx.se/mail/etiquette.html --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
