Re: Curl's network performance is slower than IDM.
Patrick Schlangen 于2020年6月19日周五 上午4:31写道： > Hi, > > interesting thread. > > Am 18.06.2020 um 12:09 schrieb Daniel Stenberg via curl-library < > curl-library@cool.haxx.se>: > > The question is then of course still what IDM does that makes it that much > faster or perhaps what curl should do to improve. But without more details > and knowledge of IDM's set of tricks, I'm not sure we will achieve that > right now. > > > I’ve downloaded that program out of curiosity and compared it against curl > on macOS. > > curl is running natively on macOS, IDM in a Windows 10 VMware Fusion > virtual machine on the same machine. I only tested a couple of times with > the file (http://speedtest-ny.turnkeyinternet.net/1000mb.bin) and timed > the total download time (using `time` for curl and using a stopwatch for > IDM). > > Interestingly, IDM showed max 13.5 MB/s with an average total download > time of 93 seconds, while curl showed max 12.0 MB/s and took 86 seconds for > the same file on average. So actually curl was faster but displayed the > lower max transfer rate. > > Important: Start your stopwatch already when you are in the dialog shown > in my attached screenshot. They work with a little trick here: They start > downloading in the background before you even press "Start Download"! (See > network activity in my screenshot...) > > @Hongyi: I’d recommend to retry the tests while measuring the time it > takes to download the file instead of relying on the displayed download > speed. Either curl could be too pessimistic in the displayed speed or IDM > could be too optimistic. (In the end the selling point of their product is > to be faster than the competitors.) > > Disclaimer: I didn’t spend much time, and from a statistical point it’s > probably not valid to draw any conclusions from my experiments. > I've noted that the real URL used by IDM for the file ( http://speedtest-ny.turnkeyinternet.net/1000mb.bin) is the following: http://ipv4.download.thinkbroadband.com/1GB.zip Then I retried with the above URL for several times to benchmark the total duration for each downloading progress. Anyway, the ultimate purpose is the short time-duration for the same job. The results are as follows: 1. IDM with single connection setting vs curl: They basically show the same time-durations for my case: around 58 ~ 59 secs. 2. IDM with multiple connections/multi-threading setting vs PyIDM ( https://github.com/pyIDM/PyIDM): They still basically show the same time-durations for my case: around 45 ~ 49 secs. Regards -- Hongyi Zhao --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Curl's network performance is slower than IDM.
On Fri, 19 Jun 2020, Hongyi Zhao wrote:
Another consideration came to my mind the speed computation method used in
two tools. In detail:
speed = downloaded_bytes / time_duration
Showing download speed is somewhat more complicated than so. For example,
imagine a download that goes like this:
First 60 seconds at 100% speed
Then 60 seconds at 5% speed
Finally, it recovers and goes back to 100% speed.
What "transfer speed" do you show in the various phases? In curl's case, we
show two speeds: we should the full "downloaded_bytes / time_duration" average
one ("Average Dload") and then we show the "Current Speed", which is the
average speed over the last 5 seconds. That makes it show 100% "Current Speed"
at the end but the "Average Dload" will never get up to 100% again after the
first phase.
Other tools will do it differently and some people will have strong opinions
on what's wrong and what's right and what data to include and not within the
"downloaded bytes".
What I want to confirm is whether the different criterions for data
package statistics are used for speed computation between IDM curl.
Given what Patrick explained, I would say that's pretty much concluded now:
they're done differently.
--
/ daniel.haxx.se | Commercial curl support up to 24x7 is available!
| Private help, bug fixes, support, ports, new features
| https://www.wolfssl.com/contact/
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Cross-compiled generating error 77 in application
Thanks for the advice, it is what I needed to know. Compiling with the following fixes resolves my problem. --without-ssl --with-schannel Peter On Fri, Jun 19, 2020 at 8:37 AM Ray Satiro via curl-library < curl-library@cool.haxx.se> wrote: > On 6/19/2020 3:17 AM, Ray Satiro wrote: > > If you use the Schannel backend (native WIndows SSL) instead of > > OpenSSL then you do not need a separate bundle of certificates, the > > Windows OS certificate store is used by default. The next version of > > curl (7.71, to be released next week) will support that for OpenSSL > > but not by default, you would use CURLSSLOPT_NATIVE_CA [5] to enable it. > > > To clarify the changes in the next version, it appears the curl tool w/ > OpenSSL on Windows will fall back on the native CA store when no > certificate bundle is found. [1] That may override hardcoded paths in > libcurl. I will seek clarification on that. > > [1]: > https://github.com/curl/curl/blob/fa4fbc5/src/tool_operate.c#L2415-L2428 > > --- > Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library > Etiquette: https://curl.haxx.se/mail/etiquette.html --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Cross-compiled generating error 77 in application
On 6/19/2020 3:17 AM, Ray Satiro wrote: If you use the Schannel backend (native WIndows SSL) instead of OpenSSL then you do not need a separate bundle of certificates, the Windows OS certificate store is used by default. The next version of curl (7.71, to be released next week) will support that for OpenSSL but not by default, you would use CURLSSLOPT_NATIVE_CA [5] to enable it. To clarify the changes in the next version, it appears the curl tool w/ OpenSSL on Windows will fall back on the native CA store when no certificate bundle is found. [1] That may override hardcoded paths in libcurl. I will seek clarification on that. [1]: https://github.com/curl/curl/blob/fa4fbc5/src/tool_operate.c#L2415-L2428 --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Cross-compiled generating error 77 in application
On 6/18/2020 1:27 PM, Peter Bushnell via curl-library wrote:
Have added curl source to build a statically compiled libcurl.a to a
multi-platform application which cross-compiles binaries for on Linux
for Windows and Mac. The application on Linux and Mac work as
expected, but when calling curl_easy_perform with a HTTPS URL a 77
error code is returned. During configure the following ca bundle path
is set.
/etc/ssl/certs/ca-certificates.crt
When configuring for Windows the same cert is set which can obviously
not be found in a Windows environment. Hoping that I only need to set
--with-ca-path or --with-ca-bundle but I've not been able to work out
what it should be set to. The advice I've seen online is for people to
download the cert which allows my cross-compiled curl.exe to work with
SSL, but does not seem to work if placed in the same directory as the
application. Tried setting the CA bundle path to ./ without luck and
also tried SSPI but that did not make a difference.
Setting the following allows calls to HTTPS to work, this shows that
libcurl has been successfully compiled, linked and can work but
obviously this is not desired behaviour.
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0);
Compiling with the following configure options:
./configure --disable-shared --enable-static --disable-ldap
--disable-sspi --without-librtmp --disable-ftp --disable-file
--disable-dict --disable-telnet --disable-tftp --disable-rtsp
--disable-pop3 --disable-imap --disable-smtp --disable-gopher
--disable-smb --host=x86_64-w64-mingw32
Configure output:
configure: Configured to build curl/libcurl:
Host setup: x86_64-w64-mingw32
Install prefix: /mnt/d/github/pyrk-win64/depends/x86_64-w64-mingw32
Compiler: x86_64-w64-mingw32-gcc
CFLAGS: -pipe -O2 -pthread
CPPFLAGS: -isystem
/mnt/d/github/pyrk-win64/depends/x86_64-w64-mingw32/include -isystem
/mnt/d/github/pyrk-win64/depends/x86_64-w64-mingw32/include
LDFLAGS: -L/mnt/d/github/pyrk-win64/depends/x86_64-w64-mingw32/lib
-L/mnt/d/github/pyrk-win64/depends/x86_64-w64-mingw32/lib
LIBS: -lssl -lcrypto -lssl -lcrypto -lgdi32 -lws2_32
curl version: 7.70.0
SSL: enabled (OpenSSL)
SSH: no (--with-{libssh,libssh2})
zlib: no (--with-zlib)
brotli: no (--with-brotli)
GSS-API: no (--with-gssapi)
TLS-SRP: enabled
resolver: POSIX threaded
IPv6: enabled
Unix sockets: no (--enable-unix-sockets)
IDN: no (--with-{libidn2,winidn})
Build libcurl: Shared=no, Static=yes
Built-in manual: enabled
--libcurl option: enabled (--disable-libcurl-option)
Verbose errors: enabled (--disable-verbose)
Code coverage: disabled
SSPI: no (--enable-sspi)
ca cert bundle: /etc/ssl/certs/ca-certificates.crt
ca cert path: no
ca fallback: no
LDAP: no (--enable-ldap / --with-ldap-lib /
--with-lber-lib)
LDAPS: no (--enable-ldaps)
RTSP: no (--enable-rtsp)
RTMP: no (--with-librtmp)
Metalink: no (--with-libmetalink)
PSL: no (libpsl not found)
Alt-svc: no (--enable-alt-svc)
HTTP2: disabled (--with-nghttp2)
HTTP3: disabled (--with-ngtcp2, --with-quiche)
ESNI: no (--enable-esni)
Protocols: HTTP HTTPS
Features: SSL IPv6 AsynchDNS NTLM TLS-SRP HTTPS-proxy
Any help on this would be appreciated.
On Windows the curl command line tool (curl.exe) will check environment
variables like CURL_CA_BUNDLE and if not found then search paths.
[1][2][3] libcurl doesn't do that, if you have no compile-time path then
you would need to detect the location yourself and use CURLOPT_CAINFO
[4] to set it.
If you use the Schannel backend (native WIndows SSL) instead of OpenSSL
then you do not need a separate bundle of certificates, the Windows OS
certificate store is used by default. The next version of curl (7.71, to
be released next week) will support that for OpenSSL but not by default,
you would use CURLSSLOPT_NATIVE_CA [5] to enable it.
[1]: https://curl.haxx.se/docs/sslcerts.html
[2]:
https://github.com/curl/curl/blob/curl-7_70_0/src/tool_operate.c#L2261-L2339
[3]:
https://github.com/curl/curl/blob/curl-7_70_0/src/tool_doswin.c#L598-L644
[4]: https://curl.haxx.se/libcurl/c/CURLOPT_CAINFO.html
[5]: https://curl.haxx.se/libcurl/c/CURLOPT_SSL_OPTIONS.html
---
Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library
Etiquette: https://curl.haxx.se/mail/etiquette.html
