Re: 503 error
On Thu, 9 Aug 2012, 黎小辉 wrote: (I brought back the original and more descriptive subject) My question is what should I do if 503 return from server in my app? You need to figure out what in the request that makes the server respond like this. I would suggest you modify your libcurl request (with the use of curl_easy_setopt options) to look identical to one of the successful requests. -- / daniel.haxx.se--- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
bug in schannel connection shutdown?
Hi,
I'm using libcurl 7.27.0 with schannel on windows with MSVC2008,
with https.
My code runs allocates and deallocates handles on demand, and
adds/removes them all to a single multi handle, so always
curl_easy_init()
curl_easy_setopt()...
curl_multi_add_handle()
curl_multi_perform()...
curl_multi_remove_handle()
curl_easy_cleanup()
If I then call curl_multi_cleanup() (when shutting down the entire
program), I get accesses to free()d memory in schannel connection
cleanup. I don't get such issues on linux with gnutls.
If I set CURLOPT_FORBID_REUSE, connections are closed earlier (I guess
either in curl_multi_perform() or curl_easy_cleanup()), and the issue
doesn't occur.
The attached file reproduces the issue.
Frank
--
Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it. - Brian W. Kernighan
#include curl/curl.h
int main(int argc, char *argv[])
{
CURLcode ret;
CURL *hnd;
CURLM *mhnd;
int running_handles;
curl_global_init(CURL_GLOBAL_ALL);
hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_URL, https://www.google.com/;);
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
#if 0
curl_easy_setopt(hnd, CURLOPT_FORBID_REUSE, 1L);
#endif
mhnd = curl_multi_init();
curl_multi_add_handle(mhnd, hnd);
curl_multi_perform(mhnd, running_handles);
while(running_handles != 0)
{
curl_multi_perform(mhnd, running_handles);
}
curl_multi_remove_handle(mhnd, hnd);
curl_easy_cleanup(hnd);
curl_multi_cleanup(mhnd);
curl_global_cleanup();
return 0;
}
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Re:curl-library Digest, Vol 84, Issue 16
Hi , Daniel StenbergThanks for your reply!Below is the http request of the VLC player:GET / HTTP/1.1 Host: iptv.tjurk.com:5953 User-Agent: VLC/2.0.3 LibVLC/2.0.3 Range: bytes=0- // my app have no this request Connection: close // my app have no this request Authorization: Basic c3BvcjU6MjIyNA== Icy-MetaData: 1 // my app have no this request, I don't know how to add this http request HTTP/1.1 200 OK Server: nginx Date: Thu, 09 Aug 2012 10:53:59 GMT Content-Type: application/octet-stream Transfer-Encoding: chunked Connection: close Cache-Control: no-cache --And the following is my app's http requests and server responses:* About to connect() to iptv.tjurk.com port 5953 (#0) * Trying 188.165.228.55... * Connected to iptv.tjurk.com (188.165.228.55) port 5953 (#0) * Server auth using Basic with user 'spor5' GET / HTTP/1.1 Authorization: Basic c3BvcjU6MjIyNA== User-Agent: HbbTV1.1.1(;); CE-HTML; SPARK Host: iptv.tjurk.com:5953 Accept: */* HTTP/1.1 200 OK Server: nginx Date: Thu, 09 Aug 2012 10:36:57 GMT Content-Type: application/octet-stream Transfer-Encoding: chunked Connection: keep-alive Keep-Alive: timeout=2 Cache-Control: no-cache * Closing connection #0 * About to connect() to iptv.tjurk.com port 5953 (#0) * Trying 188.165.228.55... * Connected to iptv.tjurk.com (188.165.228.55) port 5953 (#0) * Server auth using Basic with user 'spor5' GET / HTTP/1.1 Authorization: Basic c3BvcjU6MjIyNA== Range: bytes=0- User-Agent: HbbTV1.1.1(;); CE-HTML; SPARK Host: iptv.tjurk.com:5953 Accept: */* transferMode.dlna.org: Interactive HTTP/1.1 200 OK Server: nginx Date: Thu, 09 Aug 2012 10:36:58 GMT Content-Type: application/octet-stream Transfer-Encoding: chunked Connection: keep-alive Keep-Alive: timeout=2 Cache-Control: no-cache * About to connect() to iptv.tjurk.com port 5953 (#0) * Trying 188.165.228.55... * Connected to iptv.tjurk.com (188.165.228.55) port 5953 (#0) * Server auth using Basic with user 'spor5' GET / HTTP/1.1 Authorization: Basic c3BvcjU6MjIyNA== User-Agent: HbbTV1.1.1(;); CE-HTML; SPARK Host: iptv.tjurk.com:5953 Accept: */* HTTP/1.1 503 Service Temporarily Unavailable Server: nginx Date: Thu, 09 Aug 2012 10:37:02 GMT Content-Type: text/html Content-Length: 383 383 File Length extracted from Content Length : 383 Connection: keep-alive Keep-Alive: timeout=2 * Closing connection #0 * About to connect() to iptv.tjurk.com port 5953 (#0) * Trying 188.165.228.55... * Connected to iptv.tjurk.com (188.165.228.55) port 5953 (#0) * Server auth using Basic with user 'spor5' GET / HTTP/1.1 Authorization: Basic c3BvcjU6MjIyNA== User-Agent: HbbTV1.1.1(;); CE-HTML; SPARK Host: iptv.tjurk.com:5953 Accept: */* HTTP/1.1 503 Service Temporarily Unavailable Server: nginx Date: Thu, 09 Aug 2012 10:37:03 GMT Content-Type: text/html Content-Length: 383 383 File Length extracted from Content Length : 383 Connection: keep-alive Keep-Alive: timeout=2 * Closing connection #0 * Closing connection #0 --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: Re:curl-library Digest, Vol 84, Issue 16
You can add custom headers to your request by creating a list of them with curl_slist_append and then passing that list to curl_easy_setopt(CURLOPT_HTTPHEADER). See the example at http://curl.haxx.se/libcurl/c/curl_slist_append.html. But looking at this I think the problem is that you're sending 5 requests in a row on the same connection - the first two succeed, and then the server starts sending 503 Service Temporarily Unavailable. It seems to be telling you that it has already responded on this connection and it's not going to talk to you anymore. Connection: Close should help with this. Try adding curl_easy_setopt(handle, CURLOPT_FORBID_REUSE, 1). And make sure you actually want to be fetching the same url on the server over and over - you may just have an accidental infinite loop. From: curl-library [curl-library-boun...@cool.haxx.se] on behalf of 黎小辉 [lixiaohui9...@126.com] Sent: Thursday, August 09, 2012 7:07 AM To: curl-library@cool.haxx.se Subject: Re:curl-library Digest, Vol 84, Issue 16 Hi , Daniel Stenberg Thanks for your reply! Below is the http request of the VLC player: GET / HTTP/1.1 Host: iptv.tjurk.com:5953 User-Agent: VLC/2.0.3 LibVLC/2.0.3 Range: bytes=0- // my app have no this request Connection: close // my app have no this request Authorization: Basic c3BvcjU6MjIyNA== Icy-MetaData: 1 // my app have no this request, I don't know how to add this http request HTTP/1.1 200 OK Server: nginx Date: Thu, 09 Aug 2012 10:53:59 GMT Content-Type: application/octet-stream Transfer-Encoding: chunked Connection: close Cache-Control: no-cache -- And the following is my app's http requests and server responses: * About to connect() to iptv.tjurk.com port 5953 (#0) * Trying 188.165.228.55... * Connected to iptv.tjurk.com (188.165.228.55) port 5953 (#0) * Server auth using Basic with user 'spor5' GET / HTTP/1.1 Authorization: Basic c3BvcjU6MjIyNA== User-Agent: HbbTV1.1.1(;); CE-HTML; SPARK n! bsp; Host: iptv.tjurk.com:5953 Accept: */* HTTP/1.1 200 OK Server: nginx Date: Thu, 09 Aug 2012 10:36:57 GMT Content-Type: application/octet-stream Transfer-Encoding: chunked Connection: keep-alive Keep-Alive: timeout=2 Cache-Control: no-cache * Closing connection #0 * About to connect() to iptv.tjurk.com port 5953 (#0) * Trying 188.165.228.55... * Connected to iptv.tjurk.com (188.165.228.55) port 5953 (#0) * Server auth using Basic with user 'spor5' GET / HTTP/1.1 Authorization: Basic c3BvcjU6MjIyNA== Range: bytes=0- User-Agent: HbbTV1.1.1(;); CE-HTML; SPARK Host: iptv.tjurk.com:5953 Accept: */* transferMode.dlna.org: Interactive HTTP/1.1 200 OK Server: nginx Date: Thu, 09 Aug 2012 10:36:58 GMT Content-Type: application/octet-stream Transfer-Encoding: chunked Connection: keep-alive Keep-Alive: timeout=2 Cache-Control: no-cache * About to connect() to iptv.tjurk.com port 5953 (#0) * Trying 188.165.228.55... * Connected to iptv.tjurk.com (188.165.228.55) port 5953 (#0) * Server auth using Basic with user 'spor5' GET / HTTP/1.1 Authorization: Basic c3BvcjU6MjIyNA== User-Agent: HbbTV1.1.1(;); CE-HTML; SPARK Host: iptv.tjurk.com:5953 Accept: */* HTTP/1.1 503 Service Temporarily Unavailable Server: nginx Date: Thu, 09 Aug 2012 10:37:02 GMT Content-Type: text/html Content-Length: 383 383 File Length extracted from Content Length : 383 Connection: keep-alive Keep-Alive: timeout=2 * Closing connection #0 * About to connect() to iptv.tjurk.com port 5953 (#0) * Trying 188.165.228.55... * Connected to iptv.tjurk.com (188.165.228.55) port 5953 (#0) * Server auth using Basic with user 'spor5' GET / HTTP/1.1 Authorization: Basic c3BvcjU6MjIyNA== User-Agent: HbbTV1.1.1(;); CE-HTML; SPARK Host: iptv.tjurk.com:5953 Accept: */* HTTP/1.1 503 Service Temporarily Unavailable Server: nginx Date: Thu, 09 Aug 2012 10:37:03 GMT Content-Type: text/html Content-Length: 383 383! File Length extracted from Content Length : 383 Connection: keep-alive Keep-Alive: timeout=2 * Closing connection #0 * Closing connection #0 - This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.
RE: bug in schannel connection shutdown?
Frank, I'm using libcurl 7.27.0 with schannel on windows with MSVC2008, with https. ... If I then call curl_multi_cleanup() (when shutting down the entire program), I get accesses to free()d memory in schannel connection cleanup. I don't get such issues on linux with gnutls. The attached file reproduces the issue. I've reproduced the issue you reported and I am taking a look at it now. Thanks, Mark --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Wrong http request size calculation when uploading /proc/cpuinfo
Hi,
When using curl to upload files to a pastebin, I noticed that `curl -v
-F file=@/proc/cpuinfo http://...` will send an http request stating
that the file is empty, but it reads the contents anyway and the server
(apache in my case) returns 400 Bad Request because it didn't expect
the content.
AFAICT the problem is that stat() on files like this returns 0 in
st_size and AddFormData() doesn't handle this case in any special way.
Right now I'm using this code[1] to work around the problem. Basically I
just load the file into memory in case st_size is 0 and pass it as
CURLFORM_BUFFER{,PTR} instead of CURLFORM_FILE.
I hope this is enough for someone with more knowledge about libcurl
internals to fix this properly.
[1]: http://git.server-speed.net/users/flo/fb/tree/fb-helper.c?h=master#n270
--
Florian Pritz
signature.asc
Description: OpenPGP digital signature
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Wrong http request size calculation when uploading /proc/cpuinfo
On Thu, 9 Aug 2012, Florian Pritz wrote:
When using curl to upload files to a pastebin, I noticed that `curl -v -F
file=@/proc/cpuinfo http://...` will send an http request stating that the
file is empty, but it reads the contents anyway and the server (apache in my
case) returns 400 Bad Request because it didn't expect the content.
Yes. I took a decision many years ago (introduced in libcurl 7.12.1 in August
2004) that I would let libcurl stat() the file first to figure out the file
size (and use that in the request headers) and then read the file contents
later in a streaming manner when providing thr request body. That is necessary
to avoid having to read the entire thing into memory before sending it off -
which libcurl used to do before that.
The downside is that it doesn't handle growing or shrinking files properly.
Right now I'm using this code[1] to work around the problem. Basically I
just load the file into memory in case st_size is 0 and pass it as
CURLFORM_BUFFER{,PTR} instead of CURLFORM_FILE.
Sounds like a decent work-around. Another version could copy the file to a
temporary one and you'd then upload that. It would use less memory but require
temp file management in your app.
I hope this is enough for someone with more knowledge about libcurl
internals to fix this properly.
Sorry, but what exactly would a proper fix be to this problem you think?
I can think of one fix if we assume that the server is a HTTP/1.1, as then we
could send the whole request using chunked encoding - but libcurl can't
blindly assume 1.1 and some server won't like getting such posted in a chunked
manner.
--
/ daniel.haxx.se
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Wrong http request size calculation when uploading /proc/cpuinfo
On 09.08.2012 22:50, Daniel Stenberg wrote: On Thu, 9 Aug 2012, Florian Pritz wrote: When using curl to upload files to a pastebin, I noticed that `curl -v -F file=@/proc/cpuinfo http://...` will send an http request stating that the file is empty, but it reads the contents anyway and the server (apache in my case) returns 400 Bad Request because it didn't expect the content. [..] I hope this is enough for someone with more knowledge about libcurl internals to fix this properly. Sorry, but what exactly would a proper fix be to this problem you think? So far I have only seen this happen for files read from some pseudo file system like /proc and those tend to be small. I think you could load the file into ram and add a (configurable) upper limit. The default could be something like 50MiB. The user can increase as they see fit and 0 could mean unlimited. If you hit the limit you throw an error. It won't be perfect, but at least the error will give users a clue about what's going on. -- Florian Pritz signature.asc Description: OpenPGP digital signature --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Wrong http request size calculation when uploading /proc/cpuinfo
On Thu, Aug 09, 2012 at 11:13:13PM +0200, Florian Pritz wrote: So far I have only seen this happen for files read from some pseudo file system like /proc and those tend to be small. The same thing would happen, for example, on block devices, pipes or special character devices. I think you could load the file into ram and add a (configurable) upper limit. The default could be something like 50MiB. The user can increase as they see fit and 0 could mean unlimited. If you hit the limit you throw an error. It won't be perfect, but at least the error will give users a clue about what's going on. There some danger in doing that. If the user is trying to send data directly from a pipe or a slow network socket or /dev/video, then it could take a *long* time to reach 50 MiB at which time the program will just exit without doing anything. Meanwhile, all that data is gone forever. Dan --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: Wrong http request size calculation when uploading /proc/cpuinfo
From: curl-library [curl-library-boun...@cool.haxx.se] on behalf of Florian Pritz [bluew...@xinu.at] Sent: Thursday, August 09, 2012 5:13 PM To: curl-library@cool.haxx.se Subject: Re: Wrong http request size calculation when uploading /proc/cpuinfo So far I have only seen this happen for files read from some pseudo file system like /proc and those tend to be small. I think if the OS filesystem is not returning the right value from stat, that's not curl's problem. I would fix this by capping the amount of data sent to that returned by stat (and maybe printing a warning that not all the file data was sent, and why). So for /proc/cpuinfo it would just not send anything. You can always copy it to a regular temp file and then send that instead. Joe - This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Known Bug #37 + auth callback status
37. Having more than one connection to the same host when doing NTLM authentication (with performs multiple passes and authenticates a connection rather than a HTTP request), and particularly when using the multi interface, there's a risk that libcurl will re-use a wrong connection when doing the different passes in the NTLM negotiation and thus fail to negotiate (in seemingly mysterious ways). I just uploaded a test case to reproduce this problem (the top commit of https://github.com/JoeNotCharles/curl/commits/ntlm_connect - test2032). I'm going on vacation tomorrow for a week; when I get back I'm going to try to fix it. It would be great if somebody can use this reproduction to fix it before then - it should be pretty simple! (I believe that ConnectionExists needs to add a check if the request is NTLM and has already used a connection, it should re-use that connection if possible; currently it just uses the first connection that it can't rule out.) After this is fixed I'll get back to writing unit tests for the auth callback, so hopefully I'll be able to get it committed soon. Joe - This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Wrong http request size calculation when uploading /proc/cpuinfo
Am 10.08.2012 00:08, schrieb Joe Mason: I think if the OS filesystem is not returning the right value from stat, that's not curl's problem. I would fix this by capping the amount of data sent to that returned by stat (and maybe printing a warning that not all the file data was sent, and why). That sounds like the best solution so far, in my opinion. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: e351972 SSH: added agent based authentication
Hi, just saw that Tor's autobuilds are broken: http://curl.haxx.se/dev/log.cgi?id=2012081140-4441#prob1 probably this e351972 SSH: added agent based authentication either needs configure to check for the libssh2 version, or a test for the struct member? Gün. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
