Intermittent write errors while doing a PUT operation in 7.27.0
Hello,
We have been using the curl library since some time now. The version that we
are currently using is 7.21.6 and we are attempting to update our curl library
to 7.27.0. Platform is linux (centos/redhat)
We build the library using:
./configure --with-ssl --enable-ares --without-libidn --disable-ldap
And then our application links with libcurl.a (and libcares.a) statically.
The version of c-ares being used is the latest.
Our software uses the curl_easy API.
Our test code which wraps around curl, essentially does a PUT operation to an
HTTPS server in the cloud by doing the following approximately:
curl_easy_init();
while(i 100) {
curl_easy_reset();
curl_easy_setopt(url);
curl_easy_setopt(PUT operation);
// also set a few more setopt options..
curl_easy_perform();
i ++;
}
curl_easy_destroy();
When linked with any version of libcurl until and including 7.23.1, our test
code functions exactly as expected - it sends data to the server successfully
on every attempt. However from version 7.24.0 onwards, we notice that
intermittently one odd connection errors out (after a 60 second timeout due to
a setopt) waiting for a 200OK response from the server.
When viewing the backtrace for a specific failed connection, it seems to wait
forever with this backtrace signature. (The line number correspond to 7.27.0)
(gdb) bt
#0 0x003dca8dc053 in poll () from /lib64/libc.so.6
#1 0x00585fc0 in Curl_socket_check (readfd0=315, readfd1=-1,
writefd=-1, timeout_ms=1000) at select.c:216
#2 0x005b9316 in Transfer (conn=0x7fadb40012c8) at transfer.c:1384
#3 0x005ba2f6 in Curl_do_perform (data=0x7fadb8e8) at
transfer.c:2109
#4 0x005ba5ac in Curl_perform (data=0x7fadb8e8) at transfer.c:2233
#5 0x0057c355 in curl_easy_perform (curl=0x7fadb8e8) at easy.c:536
This seems to indicate that the thread is waiting for a 200OK from the server
after having performed a write.
An interesting observation is that I'm able to recreate this error very
consistently and the error happens at the exact same index in the pseudo code
while loop above. As mentioned, this is never seen on libcurl 7.23.1 and lower,
but it is always seen on all version 7.24.0 and above.
A code inspection of hte transfer.c code or the polling code in select.c did
not raise any alarm flags..
Any pointers on what might be going on here is appreciated..
Thanks,
Gautam---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
multi interface blocking with certain ftp servers
I have a problem with the libcurl multi interface (using libcurl
7.25.0 on openSUSE 12.2) blocking when talking to certain FTP
servers until a libcurl timeout is reached after a couple of
minutes. This is problematic because it freezes the whole
application and all other running transfers. Apparently the
problem is related to passive mode, i.e. happens after switching
to passive mode, but only for certain hosts as for the vast
majority it works fine.
After tracing it with gdb it seems that libcurl gets
stuck in the while loop in ftp_easy_statemach() calling
Curl_pp_easy_statemach() - Curl_socket_check() - poll()
repeatedly until a timeout in Curl_pp_easy_statemach() is
reached. A minimal example (adapted from multi-app.c) that
connects to a server triggering the problem is attached. It
produces the following verbose output:
8
ping
ping
ping
ping
ping
ping
ping
* About to connect() to ftp.caudium.net port 21 (#0)
* Trying 193.218.105.12...
ping
ping
ping
ping
* Connected to ftp.caudium.net (193.218.105.12) port 21 (#0)
* Connected to ftp.caudium.net (193.218.105.12) port 21 (#0)
ping
220-- Welcome to Pure-FTPd [privsep] [TLS] --
220-You are user number 2 of 50 allowed.
220-Local time is now 09:03. Server port: 21.
220-Only anonymous FTP is allowed here
220 You will be disconnected after 15 minutes of inactivity.
USER anonymous
ping
230-
230-
230- W E L C O M E A S S O C I A T I O N K A Z A R F T P S E R V E R
230-
230-
230-
230-This public FTP is repository of some OpenSource Projects and also a
230-public library.
230-
230-Please notice that bandwith is limited globaly not by sessions.
230-If you'd like to have more bandwith, please peer with us using Public IX.
230-To see where we can peer, please see http://as35189.net/, or have a look
230-at Panap, PouIX and GeIX IX pages.
230-
230-Also see our web site : http://kazar.net/
230-
230-Thanks
230-
230- Association KAZAR Team
230-
230 Anonymous user logged in
PWD
ping
257 / is your current location
* Entry path is '/'
CWD caudium
ping
250 OK. Current directory is /caudium
CWD source
ping
250 OK. Current directory is /caudium/source
EPSV
* Connect data stream passively
ping
229 Extended Passive mode OK (|||60138|)
* Trying 193.218.105.12...
* Connecting to 193.218.105.12 (193.218.105.12) port 60138
TYPE A
200 TYPE is now ASCII
LIST
* server response timeout
* Failure sending ABOR command: Timeout was reached
* Closing connection #0
8
--
Guido Berhoerster
/***
* _ _ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
*| (__| |_| | _ | |___
* \___|\___/|_| \_\_|
*
* Copyright (C) 1998 - 2011, Daniel Stenberg, dan...@haxx.se, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at http://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an AS IS basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
***/
/* This is an example application source code using the multi interface. */
#include stdio.h
#include string.h
/* somewhat unix-specific */
#include sys/time.h
#include unistd.h
/* curl stuff */
#include curl/curl.h
/*
* Download a HTTP file and upload an FTP file simultaneously.
*/
#define HANDLECOUNT 1 /* Number of simultaneous transfers */
#define FTP_HANDLE 0/* Index for the FTP transfer */
int main(void)
{
CURL *handles[HANDLECOUNT];
CURLM *multi_handle;
int still_running; /* keep number of running handles */
int i;
CURLMsg *msg; /* for picking up messages with the transfer status */
int msgs_left; /* how many messages are left */
/* Allocate one CURL handle per transfer */
for (i=0; iHANDLECOUNT; i++)
handles[i] = curl_easy_init();
/*
curl_easy_setopt(handles[FTP_HANDLE], CURLOPT_URL, ftp://sunsite.unc.edu/pub/Linux/games/;);
*/
curl_easy_setopt(handles[FTP_HANDLE], CURLOPT_URL, ftp://ftp.caudium.net/caudium/source/;);
curl_easy_setopt(handles[FTP_HANDLE], CURLOPT_VERBOSE, 1L);
/* init a multi stack */
multi_handle = curl_multi_init();
/* add the individual transfers */
for (i=0; iHANDLECOUNT; i++)
curl_multi_add_handle(multi_handle, handles[i]);
/* we
Leftovers after chunking while using the easy interface
Dear all, I use libcurl 7.21.7 to make requests to a web service. The web service answers with chunked encoding. As my process is sequential, I use the easy interface and not the multi one. This works perfectly except when I use it behind our corporate proxy. In this case, libcurl randomly returns before having received all the data. This seems to occur only with big chunks 0x2000 (8192) bytes. The next request sent to the server gets then mixed with the previous data. CURLOPT_VERBOSE gives Leftovers after chunking: nnn bytes. Could you help me to understand why libcurl returns leaving pending data, and what I should do to solve this problem ? Daniel Said -- Les donnees et renseignements contenus dans ce message sont personnels, confidentiels et secrets. Toute publication, utilisation ou diffusion, meme partielle, doit etre autorisee. Si vous n'etes pas le bon destinataire, nous vous demandons de ne pas lire, copier, utiliser ou divulguer cette communication. Nous vous prions de notifier cette erreur a l'expediteur et d'effacer immediatement cette communication de votre systeme. Any data and information contained in this electronic mail is personal, confidential and secret. Any total or partial publication, use or distribution must be authorized. If you are not the right addressee, we ask you not to read, copy, use or disclose this communication. Please notify this error to the sender and erase at once this communication from your system. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Question about NOCWD and CURLOPT_WILDCARDMATCH
On Wed, 17 Oct 2012, Mehmet Bozkurt wrote: Running with the NOCWD setting causes an CWD incorrectly (for my use case at least) being issued during a transfer with a wildcard pattern for the target file. I've found the lines in the libcurl code which changes the FTP filemethod mode to MULTICWD if it set to NOCWD. Commenting the line and running again works fine, but the change might cause a subtle or not so subtle bug that I haven't discovered yet. So what would be very much appreciated and time-saving is if anyone can give me some pointers to what I might be breaking with this change. Lovely! I would suggest that you show us the patch you've used to fix your case, and that you help us write up a test case for the problematic server/issue. Then we can work on getting that into the curl test suite and run everything and make sure no regressions occur with this fix or in the future. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Small example of using libcurl with HTML stream parser
On Wed, 17 Oct 2012, Michał Kowalczyk wrote: This program get a remote web page using libcurl functions calls. Then extracts values of the attribute href from this web page and print it to standard out. Thanks a lot! This example is now visible online at: http://curl.haxx.se/libcurl/c/href_extractor.html -- / daniel.haxx.se--- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Small example of using libcurl with HTML stream parser
Thanks a lot! I am glad that You placed my example on libcurl page. But in the comment is a wrong parser home page. Correct one is http://code.google.com/p/htmlstreamparser/ This example working only with this parser. Michael Kowalczyk 2012/10/18 Daniel Stenberg dan...@haxx.se On Wed, 17 Oct 2012, Michał Kowalczyk wrote: This program get a remote web page using libcurl functions calls. Then extracts values of the attribute href from this web page and print it to standard out. Thanks a lot! This example is now visible online at: http://curl.haxx.se/libcurl/c/**href_extractor.htmlhttp://curl.haxx.se/libcurl/c/href_extractor.html -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: How can i get the SSL context from a curl handle
Resending since I didn't get any response. Is there a way to extract the ssl context for the https connection initiated so that we could use SSL_read on the socket handle extracted from the curl handle. From: Mayank Kumar (mayankum) Sent: Tuesday, October 02, 2012 4:08 PM To: curl-library@cool.haxx.se Subject: How can i get the SSL context from a curl handle Here is what I am trying to do :- 1. Initiate a https connection using curl_easy_perform 2. Extract the socket from curl handle 3. Extract the ssl context from the curl handle and then use SSL_read to read the data from the socket since the other side will send encrypted data written using SSL_write. Not sure how to get 3 working. Thanks Mayank --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Intermittent write errors while doing a PUT operation in 7.27.0
On Thu, 18 Oct 2012, Gautam Gopinadhan wrote: When linked with any version of libcurl until and including 7.23.1, our test code functions exactly as expected - it sends data to the server successfully on every attempt. However from version 7.24.0 onwards, we notice that intermittently one odd connection errors out (after a 60 second timeout due to a setopt) waiting for a 200OK response from the server. This is plain HTTP? I guess it could be intersting to bisect your way all the way to the exact commit that changed the behavior for you... When viewing the backtrace for a specific failed connection, it seems to wait forever with this backtrace signature. (The line number correspond to 7.27.0) (gdb) bt #0 0x003dca8dc053 in poll () from /lib64/libc.so.6 #1 0x00585fc0 in Curl_socket_check (readfd0=315, readfd1=-1, writefd=-1, timeout_ms=1000) at select.c:216 It gets stuck forever in a check that is supposed to time-out after 1000 ms? Are you sure? #2 0x005b9316 in Transfer (conn=0x7fadb40012c8) at transfer.c:1384 #3 0x005ba2f6 in Curl_do_perform (data=0x7fadb8e8) at transfer.c:2109 #4 0x005ba5ac in Curl_perform (data=0x7fadb8e8) at transfer.c:2233 #5 0x0057c355 in curl_easy_perform (curl=0x7fadb8e8) at easy.c:536 This seems to indicate that the thread is waiting for a 200OK from the server after having performed a write. ... and you're saying that the data is coming without libcurl noticing it? An interesting observation is that I'm able to recreate this error very consistently and the error happens at the exact same index in the pseudo code while loop above. As mentioned, this is never seen on libcurl 7.23.1 and lower, but it is always seen on all version 7.24.0 and above. I've never seen a report like this before. Between 7.23.1 and 7.24.0 we did 205 commits... -- / daniel.haxx.se--- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Build error: with_ssh2=static, VC9, makefile.vc
I am building libcurl_a.lib from the commandline, using VC9: nmake /f makefile.vc mode=static VC=9 MACHINE=x86 WITH_DEVEL=../../win32build USE_IDN=yes WITH_WINSSL=static WITH_SSH2=static WITH_ZLIB=static This fails with: libcurl_a.lib(agent.obj) : error LNK2019: unresolved external symbol __imp_FindWindowA referenced in function agent_connect_pageant libcurl_a.lib(agent.obj) : error LNK2019: unresolved external symbol __imp_SendMessageA referenced in function agent_transact_pageant At first I solved this by recompiling libssh2 without agent_ops_pageant. Recompiling libssh2 showed me that the missing symbols can be found in user32.dll: __imp__FindWindowA 100171f0 user32:USER32.dll Moreover, libssh2 includes user32.lib while linking. So, I changed line 135 in winbuild/MakefileBuild.vc from -SSH2_LIBS = libssh2_a_debug.lib into +SSH2_LIBS = libssh2_a_debug.lib user32.lib And libcurl builds fine again. Then I checked the preceding line 131 as well: -SSH2_LIBS = libssh2.lib +SSH2_LIBS = libssh2_a.lib user32.lib libssh2_a.lib is better than libssh2.lib, because libssh2 produces an _a.lib as standard, just like curl. And the user32.lib is also needed for with_ssh2=dll. Jan --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
