Re: CURL Keberos, how to build or download DLL?
Christian Schmitz via curl-library in gmane.comp.web.curl.library (Sun, 19 Apr 2020 17:33:33 +0200): >I looked into the pre-built CURL DLLs for Windows. >But those I found on the website are all without Keberos. >The download website doesn't list the features enabled for each build. The curl by the curl project itself has Kerberos support: curl 7.69.1 (x86_64-pc-win32) libcurl/7.69.1 OpenSSL/1.1.1f (Schannel) zlib/1.2.11 brotli/1.0.7 WinIDN libssh2/1.9.0 nghttp2/1.40.0 Release-Date: 2020-03-11 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP brotli libz https://curl.haxx.se/windows/ The downloads do come with a DLL (in the bin directory). Here is a snippet of the code that builds it: https://ci.appveyor.com/project/curlorg/curl-for-win/builds/31857141#L10131 Github project: https://github.com/curl/curl-for-win You will probably want a .lib file to use the .dll in your own projects. Try generating one with procedures like https://stackoverflow.com/a/37760062/872051 -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Looking for a HTTP test server which doesn't send Content-Length
Andreas Falkenhahn via curl-library in gmane.comp.web.curl.library (Sat, 19 Oct 2019 18:40:49 +0200): >Please excuse this slightly off-topic question but I think this is the best >place to ask about it ;) > >So unfortunately, my old test case HTTP server which sent HTTP responses >without >the Content-Length field seems to be gone. Does anybody happen to have a URL >for a HTTP server which responds without setting Content-Length? Any server that uses an index.php which does not set the Content-Length. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: [RELEASE] curl 7.66.0
Hi Christoph and Daniel, Christoph M. Becker (Wed, 11 Sep 2019 14:20:36 +0200): >On 11.09.2019 at 13:36, Daniel Stenberg via curl-library wrote: > >> On Wed, 11 Sep 2019, Daniel Stenberg via curl-library wrote: >> Usng the winbuid system I am running into the error that nmake cannot find vquic\ngtcp2.obj. I tried a lot of command-line argumeents. Something has changed, but what? >> >> Does https://github.com/curl/curl/pull/4323 fix the build for you? > >FWIW, it fixes the build for me. And for me as well. Thanks to Bernhard Walle! -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: [RELEASE] curl 7.66.0
Daniel Stenberg (Wed, 11 Sep 2019 08:00:54 +0200 (CEST)): > o curl_version_info: offer quic (and h3) library info [38] > o see docs/KNOWN_BUGS (https://curl.haxx.se/docs/knownbugs.html) Usng the winbuid system I am running into the error that nmake cannot find vquic\ngtcp2.obj. I tried a lot of command-line argumeents. Something has changed, but what? -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: winbuild: Add MultiSSL support
Jan Ehrhardt via curl-library in gmane.comp.web.curl.library (Fri, 12 Apr 2019 23:07:42 +0200): >PR will follow soon. I just wanted to check here first. PR: https://github.com/curl/curl/pull/3772 >Daniel Stenberg via curl-library in gmane.comp.web.curl.library (Fri, 12 >Apr 2019 22:39:37 +0200 (CEST)): > >>The "/DWINSSL_ENABLED=1" piece looks wrong as we've worked hard recently to >>chase away all references to "winssl" and now prefer and use "Schannel". > >It is still there in the repo: >https://github.com/curl/curl/blob/master/configure.ac#L1521 >This is why I added the flag to the makefile command. I checked: "/DWINSSL_ENABLED=1" is not needed. "/DCURL_WITH_MULTI_SSL" is enough. Happy weekend. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: winbuild: Add MultiSSL support
Daniel Stenberg via curl-library in gmane.comp.web.curl.library (Fri, 12 Apr 2019 22:39:37 +0200 (CEST)): >On Fri, 12 Apr 2019, Jan Ehrhardt via curl-library wrote: > >> makefile.m32 and ./configure (when cross-compiling on *nix for Windows) >> already support MultiSSL. However the makefiles in /winbuild still prohibit >> it. It requires a small adjustment to enable it there too: >> https://github.com/Jan-E/cURL-winlibs/commit/3997fc006b3a3650f4b9bdb5b1f508ced04d00fb > >Thanks, but why not submit this as a pull request? PR will follow soon. I just wanted to check here first. >The "/DWINSSL_ENABLED=1" piece looks wrong as we've worked hard recently to >chase away all references to "winssl" and now prefer and use "Schannel". It is still there in the repo: https://github.com/curl/curl/blob/master/configure.ac#L1521 This is why I added the flag to the makefile command. >For the rest of the changes, they seem okay to me but I don't build on >Windows. > >> See the discussion on https://github.com/winlibs/cURL/issues/13 as well. >> Maybe PHP will not implement MultiSSL, but that should be no obstacle to >> enable it in Curl itself. > >OpenSSL itself does not read/use the Windows certificate store. (Comment based >on the discussion in said github issue). Yes, that is what I thought as well. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
winbuild: Add MultiSSL support
makefile.m32 and ./configure (when cross-compiling on *nix for Windows) already support MultiSSL. However the makefiles in /winbuild still prohibit it. It requires a small adjustment to enable it there too: https://github.com/Jan-E/cURL-winlibs/commit/3997fc006b3a3650f4b9bdb5b1f508ced04d00fb See the discussion on https://github.com/winlibs/cURL/issues/13 as well. Maybe PHP will not implement MultiSSL, but that should be no obstacle to enable it in Curl itself. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Using Windows certificate store when compiling with openssl
Thomas Glanzmann via curl-library (Fri, 7 Sep 2018 12:57:24 +0200): >Hello Jan, > >> x64 builds with OpenSSL just use the Windows certificate store. That >> applies even to cross-compiled x64 builds. Try it yourself >> https://phpdev.toolsforresearch.com/curl-mingw32-7.61.1.zip > >I tried, but it failed for me on w2k8r2. I tried both the 32bit and x64: Ouch. A lot of programs come with their own curl-ca-bundle.crt: Atlassian Sourcetree, PortableGit by Github, R (statistics). Apparently one of them found it necessary to put a curl-ca-bundle.crt in C:\Windows\System32. That one is automatically used by curl x64. Sorry, it was not the Windows certificate store. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Using Windows certificate store when compiling with openssl
Daniel Stenberg via curl-library (Fri, 7 Sep 2018 09:30:11 +0200 (CEST)): >I think it would be great to offer ability that to OpenSSL users on Windows. It is already there in x64 builds of curl with OpenSSL! -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Using Windows certificate store when compiling with openssl
Juan Isoza via curl-library (Fri, 7 Sep 2018 08:57:48 +0200): >But curl build for windows with openssl need a --insecure parameters or a >custom certificate file. x64 builds with OpenSSL just use the Windows certificate store. That applies even to cross-compiled x64 builds. Try it yourself https://phpdev.toolsforresearch.com/curl-mingw32-7.61.1.zip Built with Daniel Jelinski's libssh2 patch https://github.com/Jan-E/MyMpeg/blob/master/patches/ssh2_sftp_writeback.diff -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Jan Ehrhardt via curl-library (Sun, 26 Aug 2018 16:03:16 +0200): >Daniel Jelinski via curl-library (Sun, 26 Aug 2018 09:09:58 +0200): >>added 100ms: original 310kB/sec, patched 9900kB/sec > >Impressive results! Do you have a compiled version somewhere? I tried to build my own with the 3 patches: 1. winsock 2. oploadbuffer 512 KB 3. sftp writeback in libssh2 Dissappointing results for a 274MB sftp upload against a remote CoreFTP server: - bash / ssh / lftp: 68 seconds - openssh portable sftp: 50 seconds - curl triple patched: 734 seconds ping of the remote machine: 3-4 ms. Used cipher in openssh portable sftp: AES128-CBC. In production we have an old version of the CoreFTP server. On our dev machine I am going to install the most recent CoreFTP server version. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Jan Ehrhardt via curl-library (Sun, 26 Aug 2018 16:28:47 +0200): >Daniel Jelinski via curl-library (Sun, 26 Aug 2018 09:09:58 +0200): >>The patched version is CPU-bound on lower latencies; when the latency >>goes higher, SSH window becomes the limiting factor. I read that HPN >>SSH should do better on high latency links, didn't try it out. > >A lot of PR's for HPN-SSH came from @allanjude. Interesting stuff: >http://www.allanjude.com/bsd/AsiaBSDCon2017_-_SSH_Performance.pdf In May 2017 Allan Jude posted a new HPN patch here: https://lists.freebsd.org/pipermail/freebsd-current/2017-May/066015.html Quote: > In my benchmarks with 100ms of latency (from dummynet) is increases SSH > send throughput from 1 megabyte/sec to 225 megabytes/sec provided a > large enough socket buffer. I cannot reproduce that. I applied a modified version of this patch to OpenSSH-portable for Windows: https://github.com/Jan-E/openssh-portable/commit/e5c711f66aa3a14980dc7e0ca8504d0bda614fa6 For anyone that wants to try it: https://phpdev.toolsforresearch.com/expect.7z The scripting is done with Expect for Windows: https://github.com/zetamatta/expect/issues/8#issuecomment-416445736 -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Daniel Jelinski via curl-library (Sun, 26 Aug 2018 09:09:58 +0200): >The patched version is CPU-bound on lower latencies; when the latency >goes higher, SSH window becomes the limiting factor. I read that HPN >SSH should do better on high latency links, didn't try it out. A lot of PR's for HPN-SSH came from @allanjude. Interesting stuff: http://www.allanjude.com/bsd/AsiaBSDCon2017_-_SSH_Performance.pdf -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Daniel Jelinski via curl-library (Sun, 26 Aug 2018 09:09:58 +0200): >added 100ms: original 310kB/sec, patched 9900kB/sec Impressive results! >The patched version is CPU-bound on lower latencies; when the latency >goes higher, SSH window becomes the limiting factor. I read that HPN >SSH should do better on high latency links, didn't try it out. I had also been looking at HPN SSH. If you try that you should increase the buffersize as well: https://github.com/rapier1/openssh-portable/pull/11 >https://github.com/djelinski/libssh2/commit/c321b5d3a0ed964b291c710179dde7385e514ef7 >It's not ready for inclusion; it requires cleanup, and breaks SSL >backends other than openSSL with AES-CTR support. There is an open issue with AES-CTR in HPN SSH: https://github.com/rapier1/openssh-portable/issues/13#issuecomment-279778429 I did not run any tests yet. I will see if I can build it for Windows under Cygwin. https://github.com/rapier1/hpn-ssh/tree/master/contrib/cygwin -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: libssh2 optimization [was: Re: Windows users! Help us test upload performance tuning?]
Daniel Jelinski via curl-library (Sun, 26 Aug 2018 01:07:14 +0200): >sob., 11 sie 2018 o 01:05 Daniel Stenberg napisa?(a): >> It would require that libssh2 provides such an API, which it currently >> doesn't >> (and I don't know anyone working on it). > >Sent a PR to libssh2 for that: https://github.com/libssh2/libssh2/pull/264 Do you have any stats about the performanceimprovement? -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Jan Ehrhardt via curl-library (Wed, 15 Aug 2018 22:51:34 +0200): >Network connection: about 200 Mbits/s up, 200ms latency (speedtest.net) >Test file: 178 MB mp4 >Protocol: sftp > >iPad app 32 KB libssh2 buffer: 2147 seconds >iPad app 320 KB libssh2 buffer: 737 seconds >Filezilla (out-of-the-box): 967 seconds >curl x64 512 KB upload buffer: 1154 seconds We had a real oops when we tested the same 178 MB over plain FTP: curl x64 512KB upload buffer FTP: 16 seconds https upload php uploadprogress : 489 seconds % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 176M0 0 100 176M 0 11.0M 0:00:16 0:00:16 --:--:-- 13.6M start:2.062000 total:16.01500a0 Unencrypted uploads are legally (the HIPAA act) prohibited, but the price for encryption is really high. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Daniel Stenberg via curl-library (Tue, 14 Aug 2018 09:39:21 +0200 (CEST)): >I think its time we run some tests in an orderly fashion with different upload >buffer sizes and collect some numbers... Feedback on some informal upload tests from Phoenix, Arizona to Amsterdam, NL. High speed connection. Upload around 200 Mbits/s according to speedtest.net, but a latency of 200ms. My tester uploaded a 178MB mp4 file first with the unpatched iPad app (libssh2, not curl), then with a patched iPad app, then used Filezilla and last, but not least a patched curl.exe. Network connection: about 200 Mbits/s up, 200ms latency (speedtest.net) Test file: 178 MB mp4 Protocol: sftp iPad app 32 KB libssh2 buffer: 2147 seconds iPad app 320 KB libssh2 buffer: 737 seconds Filezilla (out-of-the-box): 967 seconds curl x64 512 KB upload buffer: 1154 seconds % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 176M0 0 100 176M 0 156k 0:19:14 0:19:14 --:--:-- 195k 100 176M0 0 100 176M 0 156k 0:19:14 0:19:14 --:--:-- 156k start:2.437000 total:1154.047000 Speed sometimes as low as 125K, said my tester. At the same time another user from the same network was uploading a 289 MB mp4 file in 1106 seconds, using bash/lftp. At the same speed that would have been 681 seconds for the tester's 178 MB file. bash/lftp may be some 8% faster than the patched iPad app. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Daniel Stenberg via curl-library (Tue, 14 Aug 2018 12:02:46 +0200 (CEST)): > Size Seconds Improvement > > 16 KB2.522- > 64 KB1.281x 1.97 > 128 KB 1.095x 2.30 > 256 KB 0.938x 2.69 > 512 KB 0.860x 2.93 Thanks for the stats. It indicates that my choice for 320 KB in the iOS app (using sftp) is quite good. >(Amazingly enough, HTTP/1.1 being 2.33 times faster than HTTP/2 ...) I remembered reading something about this, but it doesn't seem related (and was resolved anyway). But maybe it gives a clue: https://github.com/icing/mod_h2/issues/130 -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Daniel Stenberg via curl-library (Tue, 14 Aug 2018 09:39:21 +0200 (CEST)): >I think its time we run some tests in an orderly fashion with different upload >buffer sizes and collect some numbers... I did not test if there is a difference on *nix. Did you? Anyway, I agree with the fact that some testing has to be done. By more than 2 or 3 people... -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Daniel Stenberg via curl-library (Mon, 13 Aug 2018 23:37:56 +0200 (CEST)): >On Mon, 13 Aug 2018, Jan Ehrhardt via curl-library wrote: > >> Daniel Stenberg (Thu, 9 Aug 2018 16:55:38 +0200 (CEST)): >> -#define UPLOAD_BUFSIZE CURL_MAX_WRITE_SIZE >> +#define UPLOAD_BUFSIZE (512*1024) >> >> @Daniel Stenberg: is there a reason that is stopping you from changing this >> in curl? > >Yes. (Pretty much explained already in the TODO: >https://curl.haxx.se/docs/todo.html#Modified_buffer_size_approach) Clear. But what strikes me: only SFTP is mentioned, but my tests indicated that increasing this value also influenced the FTP upload speeds. From 10 to 5 seconds on XP in Daniel Jelinski's testcurl uploads. And down to less than a second on Win 7 and Win 10. >1. This buffer is allocated as part of the regular easy handle and increasing >that allocation this much in one go I fear will cause problems to some >existing applications out there. Especially those doing a huge amount of >parallel transfers. I can imagine this. In my iOS app (using libssh2, not curl), there are no parallel transfers, but only single transfers of typically 200MB+ files. >2. There's basically only SFTP that needs this huge buffer so it seems >wasteful to always use such a large buffer if it isn't necessary. I would like >to see it conditionally enlarged if SFTP is indeed used. Sure? What about FTP uploads? See https://curl.haxx.se/mail/lib-2018-08/0150.html >3. I think it might be a good idea to let the application control the SFTP >size, so that the 1,000 parallel transfer applications can use a smaller >buffer and the single connection apps can use a larger. The iOS NMSSH library did just that. The buffersize is configurable from the calling app. >Besides all that, it would also be interesting to work out what an "ideal" >size might be. Is 512KB large enough? Is 256KB enough for 99% of the cases? >Would 1MB help in the extremely high speed high latency cases? In my iOS tests I settled for 320KB, but that was was for a 4-5 ms latency connection. I have no idea whet the effect would be for high latency connections. And the app crashed with 512KB, most likely due to memory allocation issues. Debugging on iOS is no easy job, so I did nit investigate further. >Would it be sensible to attempt to detect the situation and dynamicly grow the >size if deemed fine? Ouch. Seems a hell of a job to auto-detect an optimal size. Better make it configurable from outside (API, command line option). -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows and CA certificates
Daniel Jelinski via curl-library (Tue, 7 Aug 2018 23:11:25 +0200): >I recently started using HTTPS functionality with libcurl + openSSL; I >noticed that by default this combo does not use Windows certificates, >but instead wants to load them from CA bundle. I happened to notice that recent X64 builds with OpenSSL 1.0.2 (and probably higher as well) actually do use the Windows certificates. I first noticed this with a cross-compiled X64 build on Ubuntu 16.04, but later confirmed it for native X64 builds (VC15, VC14, VC11 and even VC9 x64). No problems with Elliptic-curve ciphers and/or TLS v1.2. Example with a VC9 x64 build: C:\>curl --version curl 7.61.0 (x86_64-pc-win32) libcurl/7.61.0 OpenSSL/1.0.2o zlib/1.2.8 WinIDN libssh2/1.8.0 nghttp2/1.33.0 Release-Date: 2018-07-11 Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS IDN IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL libz HTTP2 HTTPS-proxy C:\>curl --head https://player.toolsforresearch.com/ HTTP/2 200 date: Mon, 13 Aug 2018 20:26:10 GMT server: Apache x-content-type-options: nosniff x-frame-options: SAMEORIGIN strict-transport-security: max-age=31536000 expect-ct: enforce,max-age=30 vary: Accept-Encoding x-xss-protection: 1; mode=block referrer-policy: no-referrer, strict-origin-when-cross-origin content-type: text/html Example with a cross-compiled X64 build, zipped in https://phpdev.toolsforresearch.com/curl-mingw32-7.61.0.zip C:\>curl-x86_64-w64-mingw32-static.exe --head https://nghttp2.org HTTP/2 200 date: Mon, 13 Aug 2018 20:18:24 GMT content-type: text/html last-modified: Tue, 08 May 2018 13:53:22 GMT etag: "5af1abd2-19d8" accept-ranges: bytes content-length: 6616 x-backend-header-rtt: 0.002717 strict-transport-security: max-age=31536000 server: nghttpx via: 2 nghttpx x-frame-options: SAMEORIGIN x-xss-protection: 1; mode=block x-content-type-options: nosniff -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Daniel Stenberg (Thu, 9 Aug 2018 16:55:38 +0200 (CEST)): -#define UPLOAD_BUFSIZE CURL_MAX_WRITE_SIZE +#define UPLOAD_BUFSIZE (512*1024) @Daniel Stenberg: is there a reason that is stopping you from changing this in curl? -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Daniel Stenberg via curl-library (Thu, 9 Aug 2018 16:55:38 +0200 (CEST)): >-#define UPLOAD_BUFSIZE CURL_MAX_WRITE_SIZE >+#define UPLOAD_BUFSIZE (512*1024) This extra patch also had major speed improvements for Daniel Jelinski's testcurl.bat. I have now included a fully static curl-vc9-x86-double-patched.exe in https://phpdev.toolsforresearch.com/curl-mingw32-7.61.0.zip The two cross-compiled curl's in the zip have both patches as well. Microsoft Windows XP [Version 5.1.2600] generating test file... running vanilla... start:0,531000 total:11,469000 running patched... start:0,546000 total:11,328000 running double patched... start:0,516000 total:5,266000 Microsoft Windows [Version 6.1.7601] generating test file... running vanilla... start:0,546000 total:9,578000 running patched... start:0,546000 total:2,698000 running double patched... start:0,546000 total:0,812000 Microsoft Windows [Version 10.0.17134.165] generating test file... running vanilla... start:0.547000 total:5.125000 running patched... start:0.515000 total:1.828000 running double patched... start:0.532000 total:0.813000 Measured on the same machine (a quadruple boot Lenovo X220). Thanks to this discussion I have also doubled the upload speed in my iOS app. The 274MB mp4 file uploaded now in 70 seconds in stead of 2:25 minutes: https://github.com/NMSSH/NMSSH/pull/136#issuecomment-412398204 -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Daniel Jelinski via curl-library (Fri, 10 Aug 2018 22:59:22 +0200): >Ok, so let's put Linux to a test. Reality check from a CentOS 6 machine % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 274M0 0 100 274M 0 667k 0:07:00 0:07:00 --:--:-- 667k start:0.191403 total:420.697969 >0. Patch CURL: -#define UPLOAD_BUFSIZE CURL_MAX_WRITE_SIZE +#define UPLOAD_BUFSIZE (512*1024) % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 274M0 0 100 274M 0 1297k 0:03:36 0:03:36 --:--:-- 1297k start:0.230242 total:216.560168 If I use lftp on the target Windows 2008 R2 machine to get the file from the CentOS 6 machine: c:\>ptime bash /utils/pcx6get.sh um001348.opt.mp4 === bash /utils/pcx6get.sh um001348.opt.mp4 === Execution time: 13.201 s rsync running on the target Windows 2008 R2 server, syncing from the CentOS, is even a tiny bit faster: 287754814 100% 23.35MB/s0:00:11 (xfer#1, to-check=0/1) sent 30 bytes received 287790026 bytes 23023204.48 bytes/sec total size is 287754814 speedup is 1.00 Execution time: 12.183 s -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Jan Ehrhardt via curl-library (Fri, 10 Aug 2018 17:25:51 +0200): >running curl-mingw64 sftp > % Total% Received % Xferd Average Speed TimeTime Time Current > Dload Upload Total SpentLeft Speed >100 274M0 0 100 274M 0 298k 0:15:42 0:15:42 --:--:-- 313k >100 274M0 0 100 274M 0 298k 0:15:42 0:15:42 --:--:-- 298k >start:0,64 total:942,153000 Gisle sent me a link to his monolitic compiled version. About the same speed with my 274MB mp4 file: running curl-monolitic sftp % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 274M0 0 100 274M 0 292k 0:16:00 0:16:00 --:--:-- 299k 100 274M0 0 100 274M 0 292k 0:16:00 0:16:00 --:--:-- 292k start:0,468000 total:960,763000 Too bad. -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Jan Ehrhardt via curl-library (Fri, 10 Aug 2018 03:29:21 +0200):
>Gisle Vanem via curl-library (Thu, 9 Aug 2018 18:31:12 +0200):
>>Jan Ehrhardt wrote:
>>
>>>> Wow dude! 2 times faster than FileZilla now.
>>>>
>>>> Time decreased from 33.153s to 6.4 sec (same random 10 MByte file).
>>>> Versus approx. 5.3 sec for curl/FTP.
>>>
>>> Using SFTP?
>>
>>Yes:
>>curl.exe -k -# --write-out "speed: %%{speed_upload} bytes/sec, total-time:
>>%%{time_total}" ^
>> sftp://xyz -T c:\TEMP\curl-test.file
>>speed: 1649348,000 bytes/sec, total-time: 6,063000
>
>Can you make your compiled version available somewhere? I tried different
>combinations of VC9, VC11, VC14 and x86 / x64 and cannot get it higher than
>300k
>at the moment.
I even cross-compiled curl.exe with GCC 7.3.1 on a Ubuntu 16.04 machine, linking
libssh2 1.8.0 (also compiled with GCC 7.3.1):
https://phpdev.toolsforresearch.com/curl-mingw32-7.61.0.zip
It has both patches: the winsock patch and the sftp uploadbuffer one. Uploading
over sftp went up from 145k to about 300KB/s, but nowhere near the 4+ MB/s of
the other methods.
Interesting about the cross-compiled versions: they include a static OpenSSL
1.0.2o module (also cross-compiled by GCC), that somehow reads the system
certificate store of the Windows machines.
--
Jan
Microsoft Windows [Version 6.1.7601]
running lftp
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== E:\utils\bash.exe /utils/bash.sh ===
Execution time: 69.934 s
running vanilla...
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== curl -w"start:%{time_starttransfer} total:%{time_total}\n" (removed)
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
100 274M0 0 100 274M 0 5062k 0:00:55 0:00:55 --:--:-- 5200k
start:0,39 total:55,506000
Execution time: 55.667 s
running patched...
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== curl -w"start:%{time_starttransfer} total:%{time_total}\n" (removed)
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
100 274M0 0 100 274M 0 6825k 0:00:41 0:00:41 --:--:-- 6231k
start:0,375000 total:41,169000
Execution time: 41.320 s
running curl-mingw64 patched...
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== curl -w"start:%{time_starttransfer} total:%{time_total}\n" (removed)
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
100 274M0 0 100 274M 0 6550k 0:00:42 0:00:42 --:--:-- 6043k
start:0,374000 total:42,90
Execution time: 43.048 s
running curl-mingw64 sftp
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== curl -w"start:%{time_starttransfer} total:%{time_total}\n" (removed)
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
100 274M0 0 100 274M 0 298k 0:15:42 0:15:42 --:--:-- 313k
100 274M0 0 100 274M 0 298k 0:15:42 0:15:42 --:--:-- 298k
start:0,64 total:942,153000
Execution time: 942.211 s
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Gisle Vanem via curl-library (Thu, 9 Aug 2018 18:31:12 +0200):
>Jan Ehrhardt wrote:
>
>>> Wow dude! 2 times faster than FileZilla now.
>>>
>>> Time decreased from 33.153s to 6.4 sec (same random 10 MByte file).
>>> Versus approx. 5.3 sec for curl/FTP.
>>
>> Using SFTP?
>
>Yes:
>curl.exe -k -# --write-out "speed: %%{speed_upload} bytes/sec, total-time:
>%%{time_total}" ^
> sftp://xyz -T c:\TEMP\curl-test.file
>speed: 1649348,000 bytes/sec, total-time: 6,063000
Can you make your compiled version available somewhere? I tried different
combinations of VC9, VC11, VC14 and x86 / x64 and cannot get it higher than 300k
at the moment.
--
Jan
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Gisle Vanem via curl-library (Thu, 9 Aug 2018 17:48:49 +0200): >Daniel Stenberg wrote: > >> /* The upload buffer size, should not be smaller than CURL_MAX_WRITE_SIZE, >> as >> it needs to hold a full buffer as could be sent in a write callback */ >> -#define UPLOAD_BUFSIZE CURL_MAX_WRITE_SIZE >> +#define UPLOAD_BUFSIZE (512*1024) > >Wow dude! 2 times faster than FileZilla now. > >Time decreased from 33.153s to 6.4 sec (same random 10 MByte file). >Versus approx. 5.3 sec for curl/FTP. Using SFTP? -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Daniel Stenberg via curl-library (Thu, 9 Aug 2018 16:55:38 +0200 (CEST)): >On Thu, 9 Aug 2018, Jan Ehrhardt via curl-library wrote: > >> curl plain ftp patched 41 seconds >> curl patched sftp 1925 seconds > >Oh what a sad number there... =( > >A quick little experiment could be to try upping the upload buffer size >significantly: snip patch Patch applied. On another connection (slower than back at home) it now reports % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 274M0 0 100 274M 0 871k 0:05:22 0:05:22 --:--:-- 900k 100 274M0 0 100 274M 0 871k 0:05:22 0:05:22 --:--:-- 871k 322 seconds. Lot faster. -- Jant --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Gisle Vanem via curl-library (Thu, 9 Aug 2018 13:51:50 +0200):
>Jan Ehrhardt wrote:
>
>>> 33.153s vs 5.4s for a 10 MByte file.
>>
>> Did you time how long Filezilla takes for the same action? Filezilla
>> squeezes quite a lot over sftp-connections...
>
>11.4 sec!!
snip
>I must be using the wrong compiler and SSL-lib :-)
>But, it's certainly possible to make SFTP faster.
Same results here. Yesterday I had a discussion with a customer, uploading
a 274MB file from an iPad over sftp. That took 2:27 minutes. I repeated the
upload now with Filezilla, lftp and curl
TL;DR
iPad sftp (NMSSH2) 147 seconds
Filezilla sftp 67 seconds
lftp sftp70 seconds
curl plain ftp vanilla 56 seconds
curl plain ftp patched 41 seconds (the 2nd is my own compiled version: 42s)
curl patched sftp 1925 seconds
--
Jan
Microsoft Windows [Version 6.1.7601]
running lftp
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== E:\utils\bash.exe /utils/bash.sh ===
Execution time: 70.587 s
running vanilla...
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== curl -w"start:%{time_starttransfer} total:%{time_total}\n" removed
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
100 274M0 0 100 274M 0 4982k 0:00:56 0:00:56 --:--:-- 4960k
start:0,374000 total:56,394000
Execution time: 56.547 s
running patched...
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== curl -w"start:%{time_starttransfer} total:%{time_total}\n" removed
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
100 274M0 0 100 274M 0 6812k 0:00:41 0:00:41 --:--:-- 6420k
start:0,375000 total:41,247000
Execution time: 41.408 s
running curl patched...
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== curl -w"start:%{time_starttransfer} total:%{time_total}\n" removed
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
100 274M0 0 100 274M 0 6617k 0:00:42 0:00:42 --:--:-- 6923k
start:0,374000 total:42,463000
Execution time: 42.699 s
running patched sftp
ptime 1.0 for Win32, Freeware - http://www.pc-tools.net/
Copyright(C) 2002, Jem Berkes
=== curl -w"start:%{time_starttransfer} total:%{time_total}\n" removed
% Total% Received % Xferd Average Speed TimeTime Time Current
Dload Upload Total SpentLeft Speed
100 274M0 0 100 274M 0 145k 0:32:05 0:32:05 --:--:-- 150k
100 274M0 0 100 274M 0 145k 0:32:05 0:32:05 --:--:-- 145k
start:0,421000 total:1925,349000
Execution time: 1925.451 s
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Windows users! Help us test upload performance tuning?
Gisle Vanem via curl-library (Thu, 9 Aug 2018 01:42:18 +0200): >Jan Ehrhardt wrote: > >> I ended up with a Windows port of lftp, launched from a bash script. Curl >> sftp >> did resume, but was terribly slow. > >I also just tested with 'curl sftp//:' with the latest libssh2 >and the new 'SIO_IDEAL_SEND_BACKLOG_QUERY' option. 'sftp://' is >still 6 times slower than ftp against the same server in Denmark. > >33.153s vs 5.4s for a 10 MByte file. Did you time how long Filezilla takes for the same action? Filezilla squeezes quite a lot over sftp-connections... -- Jan --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
