Re: Windows users! Help us test upload performance tuning?
Hello! Please note that we would *love* your assistance here if you're a Windows user and can offer a few moments of your time to run a few tests on a few Windows versions and tell us the outcome! Here's a simple way to help us make curl better without doing any coding at all! =) We want to have more user's experience and results from tests to determine how we should make curl make uploads on windows as fast as possible. See Daniel's mail below for details. Please report your results to the list, to me or to Daniel Jeliński and we can make a summary in a few days. Looks good. 200/15 MBit cable connection. Windows XP Professional SP3: Microsoft Windows XP [Version 5.1.2600] generating test file... running vanilla... start:0.734000 total:10.671000 running patched... start:0.563000 total:10.157000 Windows 7 Professional SP1: Microsoft Windows [Version 6.1.7601] generating test file... running vanilla... start:0.546000 total:10.109000 running patched... start:0.53 total:2.621000 --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: Compile warning on Fedora-26
Daniel Stenberg wrote: On Wed, 6 Sep 2017, Ben Greear wrote: Is there a good fix for this warning, or do I just need to turn off strict warnings? ... vtls/openssl.c: In function ‘ossl_connect_step1’: vtls/openssl.c:2091:5: warning: ‘SSLv3_client_method’ is deprecated [-Wdeprecated-declarations] req_method = SSLv3_client_method(); Your OpenSSL headers should define OPENSSL_NO_SSL3 so that libcurl won't try to use SSLv3. When OPENSSL_NO_SSL3 is defined, there's no warning. Yes, but the real cause is: OpenSSL 1.1.0 has deprecated the function SSLv3_client_method() in favor of TLS_client_method(). TLS_client_method() also works for SSLv3. See https://www.openssl.org/docs/man1.1.0/ssl/SSLv3_client_method.html : "TLS_method(), TLS_server_method(), TLS_client_method(): These are the general-purpose version-flexible SSL/TLS methods. The actual protocol version used will be negotiated to the highest version mutually supported by the client and the server. The supported protocols are SSLv3, TLSv1, TLSv1.1 and TLSv1.2. Applications should use these methods, and avoid the version-specific methods described below." Regards, Michael --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: PolarSSL test 301 failure after bcca842e
Since commit bcca842e from last week "polarssl: fix hangs" my PolarSSL autobuilds have been failing test 301 with "curl returned 56, when expecting 0". Error 56 is CURLE_RECV_ERROR. Has anyone else seen this? I have fixed this with commit a7e4348. Thank you for reporting this! Regards, Michael --- Unsubscribe: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: CURLOPT_TCP_NODELAY by default
If no objections show up, I'll merge tomorrow. Please replace "TLS_NODELAY" with "TCP_NODELAY" in the commit message Regards, Michael --- List admin: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: External NTLM authentication
Do you want to authenticate against the reverse proxy or some server beyond that? The latter case will never work with NTLM afaik unless the reverse proxy is transparent. I am building the reverse proxy :-) It's like this: Client <--> Reverse Proxy (uses libcurl) <--> Server The client wants to authenticate against the server. Both know how to do NTLM authentication. The reverse proxy's job (in this example) is to just forward the requests and responses. To my knowledge NTLM does not care if the connection is closed between the 2nd and 3rd authentication stage and libcurl should properly handle that. The answer to the CHALLENGE packet is just sent in a different connection and *that* one is the one getting authenticated and is later used for the data transfer. No, the NTLM type 2 message and the type 3 message must use the same TCP connection. See the section "Keeping the connection alive" at https://www.innovation.ch/personal/ronald/ntlm.html Regards, Michael --- List admin: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
External NTLM authentication
Hi, in my company, we use libcurl to forward requests in a reverse proxy setup. We want to forward requests that use NTLM authentication. These requests already have "Authentication: NTLM ..." headers, so libcurl's CURLAUTH_NTLM feature is not suitable for such requests. The problem is this: When libcurl is sending a request with a body, and libcurl gets an early "401 Unauthorized" response from the server, then libcurl stops sending the body and closes the connection. The code is in http.c, about line 3150 and starts with "if(k->httpcode >= 300) ...". Usually this is a reasonable thing to do, but for NTLM, the connection must not be closed because NTLM authenticates connections, not requests. Note that if the built-in NTLM authentication is enabled, libcurl does not close the connection in this case. As a workaround, I have tried to call "curl_easy_pause(handle, CURLPAUSE_RECV)" in the read function, and then call "curl_easy_pause(handle, CURLPAUSE_CONT)" when the body has been read completely. But there is a race condition: libcurl may get the response code from the server before the read function is called for the first time. It seems to me that I need a new option for libcurl that disables the handling of early 401 responses from the server. I can write a patch, but I'm not sure what name would fit. IGNORE_EARLY_401, or HTTP_EXTERNAL_CONN_AUTH, or ...? What do you think? Regards, Michael --- List admin: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Re: getting SessionHandle values
> looking at the SessionHandle struct, and wondering: how do people retrieve current values from the structure? There is no official API to retrieve the current settings. curl_easy_getinfo() is not the counterpart of curl_easy_setopt(), and there is no curl_easy_getopt(). For options like CURLOPT_HTTPHEADER and CURLOPT_CONNECT_TO, applications need to keep the list until the transfer completes, because libcurl will not copy the list. Therefore applications that use these options must keep a reference to the list anyway. Regards, Michael --- List admin: https://cool.haxx.se/list/listinfo/curl-library Etiquette: https://curl.haxx.se/mail/etiquette.html
Mailing list admin tool stopped working
Hi, the admin tool for the mailing lists at http://cool.haxx.se/mailman/listinfo/curl-library stopped working: "Bug in Mailman... we're sorry, we hit a bug" Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: CURLINFO_TLS_SESSION with OPENSSL returns SSL_CTX instead of SSL
Should we consider providing another pointer for the OpenSSL backend? Sure, I'm open for discussions and solutions! Yes, please consider this. I propose to add a new CURLINFO constant, because the existing API must not be changed (backwards compatibility). We could call it CURLINFO_TLS_SESSION_EXTENDED or CURLINFO_TLS_SESSION_V2. It should work the same way as CURLINFO_TLS_SESSION, but with SSL* instead of SSL_CTX* for OpenSSL. At first glance, this may look like a hack, but actually it is a clean, backwards compatible solution for a real problem. Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Patch: No special error handling for Content-Encoding: compress
Hi,
Currently, libcurl rejects responses with Content-Encoding: compress
when CURLOPT_ACCEPT_ENCODING is set to . I think that libcurl should
treat the Content-Encoding compress the same as other
Content-Encodings that it does not support, e.g. bzip2. I have
attached a patch for this.
Regards,
Michael
diff --git a/lib/http.c b/lib/http.c
index b9e8047..a1eef81 100644
--- a/lib/http.c
+++ b/lib/http.c
@@ -3582,14 +3582,6 @@ CURLcode Curl_http_readwrite_headers(struct SessionHandle *data,
k-auto_decoding = GZIP;
start += 6;
}
-else if(checkprefix(compress, start)) {
- k-auto_decoding = COMPRESS;
- start += 8;
-}
-else if(checkprefix(x-compress, start)) {
- k-auto_decoding = COMPRESS;
- start += 10;
-}
else
/* unknown! */
break;
@@ -3622,9 +3614,6 @@ CURLcode Curl_http_readwrite_headers(struct SessionHandle *data,
else if(checkprefix(gzip, start)
|| checkprefix(x-gzip, start))
k-auto_decoding = GZIP;
- else if(checkprefix(compress, start)
- || checkprefix(x-compress, start))
-k-auto_decoding = COMPRESS;
}
else if(checkprefix(Content-Range:, k-p)) {
/* Content-Range: bytes [num]-
diff --git a/lib/http_chunks.c b/lib/http_chunks.c
index 80c0f95..7e91b37 100644
--- a/lib/http_chunks.c
+++ b/lib/http_chunks.c
@@ -218,7 +218,6 @@ CHUNKcode Curl_httpchunk_read(struct connectdata *conn,
(ssize_t)piece);
break;
- case COMPRESS:
default:
failf (conn-data,
Unrecognized content encoding type.
diff --git a/lib/transfer.c b/lib/transfer.c
index 32e8702..718139b 100644
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -756,7 +756,6 @@ static CURLcode readwrite_data(struct SessionHandle *data,
result = Curl_unencode_gzip_write(conn, k, nread);
break;
- case COMPRESS:
default:
failf (data, Unrecognized content encoding type.
libcurl understands `identity', `deflate' and `gzip'
diff --git a/lib/urldata.h b/lib/urldata.h
index 0655922..b1c2056 100644
--- a/lib/urldata.h
+++ b/lib/urldata.h
@@ -660,7 +660,6 @@ struct SingleRequest {
#define IDENTITY 0 /* No encoding */
#define DEFLATE 1 /* zlib deflate [RFC 1950 1951] */
#define GZIP 2 /* gzip algorithm [RFC 1952] */
-#define COMPRESS 3 /* Not handled, added for completeness */
#ifdef HAVE_LIBZ
zlibInitState zlib_init; /* possible zlib init state;
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
SSL session cache for all easy handles in a multi handle
Hi, From the documentation of curl_share_setopt: Cached DNS hosts will be shared across the easy handles using this shared object. Note that when you use the multi interface, all easy handles added to the same multi handle will share DNS cache by default without this having to be used! Multi handles also provide a connection cache, but they do not provide an SSL session cache. Is this on purpose (security), or is this just not implemented yet? Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Connecting to multiple hosts that have the same hostname
What about adding the connection cache to the share object? The SSL session IDs are already part of it, so adding the connection cache seems reasonable. I always intended to have the connection cache to also be shareable with the share interface and I think it would fit the existing paradigms very well. Thank you for your answer. Maybe I will create a patch for this. For me, that's OK. Of course, a CURLOPT_RESOLVE that only affects one single easy handle would be better :-) I completely understand that. I wouldn't mind introducing some sort of mechanism that allows this. A callback function for the name resolution would be a powerful mechanism. Similar to CURLOPT_OPENSOCKETFUNCTION, but a bit on a higher level. This new callback function should be able to either return an address, or to tell libcurl to proceed with the normal name resolution. I still think that libcurl's connection cache should also look at the addresses when reusing connections. This is useful for CURLOPT_RESOLVE (it's possible to add and remove name to address mappings), and it would also be useful for the callback function proposed above. Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Connecting to multiple hosts that have the same hostname
As suggested in many mails on this mailing list, I have tried to solve this using CURLOPT_RESOLVE. I have found two problems: Let me suggeset yet another way with existing functionality: Use the share interface, and tell each easy handle that uses those host names to use its own share object with its own DNS cache. That way, the CURLOPT_RESOLVE setting would (should!) only affect that particular DNS cache. Thank you for the suggestion. Using the share interface solves the DNS problem, but I think that the connection cache problem remains unsolved because all easy handles in a multi handle use the same connection cache. Because libcurl looks at the hostname when reusing connections (not at the IP address), an existing connection to a wrong host may get reused. What about adding the connection cache to the share object? The SSL session IDs are already part of it, so adding the connection cache seems reasonable. The drawback is of course that the handles now use one DNS cache each and don't benefit from the previous resolves from the other handles involved. For me, that's OK. Of course, a CURLOPT_RESOLVE that only affects one single easy handle would be better :-) Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Connecting to multiple hosts that have the same hostname
Hi, I have a challenging scenario for libcurl. I want to connect with https to a cluster of ADFS hosts (Active Directory Federation Services). These hosts have the same hostname, but different IP addresses. For example: - host.example.org, IP: 10.0.0.1 - host.example.org, IP: 10.0.0.2 When establishing a connection, I want to control to which host libcurl should connect to (10.0.0.1 or 10.0.0.2) because the hosts have separate user sessions. Furthermore ADFS needs SNI, so it is necessary to use the hostname in the URL (https://host.example.org). It is not possible to just use the IP address (https://10.0.0.1/ or https://10.0.0.2/). As suggested in many mails on this mailing list, I have tried to solve this using CURLOPT_RESOLVE. I have found two problems: - CURLOPT_RESOLVE is not a local setting because it pre-populates the DNS cache. All easy handles that use the same multi handle share a DNS cache. Setting different IP addresses for the same hostname using CURLOPT_RESOLVE may therefore lead to race conditions. - A wrong existing connection may get reused. libcurl only looks at the hostname, and does not remember that the connection has been set up using a special IP address with CURLOPT_RESOLVE. (I have not tested this, but I have looked at libcurl's source code.) It is possible to work around these problems by using CURLOPT_RESOLVE with an artificial hostname in the URL, e.g. https://host.example.org-10.0.0.1/; - but then the server drops the connection, because the artificial hostname is also used for SNI. A libcurl option to control the SNI hostname (and also the hostname used for the server certificate checks) would make this workaround viable. A clean solution needs to address the problems mentioned above: - CURLOPT_RESOLVE should only affect a single easy handle - A new option to control the connection cache's behavior (reuse based on hostname only or reuse based on hostname + IP address). This may be implemented in a more general way with a label (string) that gets attached to a connection, and libcurl is only allowed to reuse connections when the labels match. What do you think? All ideas and suggestions are welcome. Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Connecting to multiple hosts that have the same hostname
Interesting challenge. Can you just use different curl multi handles? --Daniel Yes, I think this would work. But tracking more than one multi handle is a difficult task. libcurl is optimized for the add many easy handles to one multi handle approach. Regards, Michael On Wed, May 27, 2015 at 1:10 PM, Michael Kaufmann m...@michael-kaufmann.ch wrote: Hi, I have a challenging scenario for libcurl. I want to connect with https to a cluster of ADFS hosts (Active Directory Federation Services). These hosts have the same hostname, but different IP addresses. For example: - host.example.org, IP: 10.0.0.1 - host.example.org, IP: 10.0.0.2 When establishing a connection, I want to control to which host libcurl should connect to (10.0.0.1 or 10.0.0.2) because the hosts have separate user sessions. Furthermore ADFS needs SNI, so it is necessary to use the hostname in the URL (https://host.example.org). It is not possible to just use the IP address (https://10.0.0.1/ or https://10.0.0.2/). As suggested in many mails on this mailing list, I have tried to solve this using CURLOPT_RESOLVE. I have found two problems: - CURLOPT_RESOLVE is not a local setting because it pre-populates the DNS cache. All easy handles that use the same multi handle share a DNS cache. Setting different IP addresses for the same hostname using CURLOPT_RESOLVE may therefore lead to race conditions. - A wrong existing connection may get reused. libcurl only looks at the hostname, and does not remember that the connection has been set up using a special IP address with CURLOPT_RESOLVE. (I have not tested this, but I have looked at libcurl's source code.) It is possible to work around these problems by using CURLOPT_RESOLVE with an artificial hostname in the URL, e.g. https://host.example.org-10.0.0.1/; - but then the server drops the connection, because the artificial hostname is also used for SNI. A libcurl option to control the SNI hostname (and also the hostname used for the server certificate checks) would make this workaround viable. A clean solution needs to address the problems mentioned above: - CURLOPT_RESOLVE should only affect a single easy handle - A new option to control the connection cache's behavior (reuse based on hostname only or reuse based on hostname + IP address). This may be implemented in a more general way with a label (string) that gets attached to a connection, and libcurl is only allowed to reuse connections when the labels match. What do you think? All ideas and suggestions are welcome. Regards, Michael --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html -- We must remember that those mortals we meet in parking lots, offices, elevators, and elsewhere are that portion of mankind God has given us to love and to serve. It will do us little good to speak of the general brotherhood of mankind if we cannot regard those who are all around us as our brothers and sisters.” --Spencer W. Kimball --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Improve the documentation for CURLOPT_SEEKFUNCTION
Hi, the documentation for CURLOPT_SEEKFUNCTION should mention that the seek function is also needed if an HTTP server closes an existing connection while curl is trying to reuse it. I have attached a patch. Feel free to improve it :-) Regards, Michael diff --git a/docs/libcurl/opts/CURLOPT_SEEKFUNCTION.3 b/docs/libcurl/opts/CURLOPT_SEEKFUNCTION.3 index 8747a17..94a52d7 100644 --- a/docs/libcurl/opts/CURLOPT_SEEKFUNCTION.3 +++ b/docs/libcurl/opts/CURLOPT_SEEKFUNCTION.3 @@ -43,8 +43,11 @@ shown above. This function gets called by libcurl to seek to a certain position in the input stream and can be used to fast forward a file in a resumed upload (instead of reading all uploaded bytes with the normal read -function/callback). It is also called to rewind a stream when doing a HTTP PUT -or POST with a multi-pass authentication method. The function shall work like +function/callback). It is also called to rewind a stream when data has already +been sent to the server and needs to be sent again. This may happen when doing +a HTTP PUT or POST with a multi-pass authentication method, or when an +existing HTTP connection is reused too late and the server closes the +connection. The function shall work like fseek(3) or lseek(3) and it gets SEEK_SET, SEEK_CUR or SEEK_END as argument for \fIorigin\fP, although libcurl currently only passes SEEK_SET. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
