Re: Fetching More than 500 files from Server
On Tue, 19 Jun 2012, Swamy Mudhbasalar wrote: Please let me know what may be causing this issue. You're ignoring my requests to follow the netiquette and you're not providing more info. I'm done. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Fetching More than 500 files from Server
Hi, We have added one more line to our code to reduce the connection established. curl_easy_setopt(ptCurlHandle, CURLOPT_COOKIE, pstrSessionCookie) After adding the CURLOPT_COOKIE, the connection issue was solved but data fetched is zero bytes Please let me know what may be causing this issue. Regards, SAM -- Message: 3 Date: Thu, 14 Jun 2012 10:33:08 +0200 (CEST) From: Daniel Stenberg dan...@haxx.se To: libcurl development curl-library@cool.haxx.se Subject: Re: Fetching More than 500 files from Server Message-ID: alpine.deb.2.00.1206141023270.19...@tvnag.unkk.fr Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed On Thu, 14 Jun 2012, Swamy Mudhbasalar wrote: Please stop top-posting and full-quoting. If you intend to reply to emails here, switch off the digest-mode! LibCurl Version : 7.21.0 We have made 13 releases since that, so it's not unthinkable that a newer version works better or at least different. It would be very interesting to learn if you see anything similar with a recent version. curl_easy_setopt(ptCurlHandle, CURLOPT_ERRORBUFFER, acErrorBuffer); curl_easy_setopt(ptCurlHandle, CURLOPT_URL, pstrURL); curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEFUNCTION, VmwGetCB); curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEDATA, (void *)tContext); curl_easy_setopt(ptCurlHandle, CURLOPT_USERAGENT, pstrUserAgent); curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYPEER, 0L); curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYHOST, 0L); curl_easy_setopt(ptCurlHandle, CURLOPT_USERPWD, pstrUserPass); If these are truly the only options you use then the connection: header in the request is truly a mystery to me... You could consider using CURLOPT_VERBOSE to get to see about libcurl's choices to keep connections open or close them etc. In which sense is this connection then still alive ? How do you see it being kept in 30 minutes? When I run the netstat command from the command prompt, i can see many connection established. For every file there is one connection established. Even ESTABLISHED! That sounds so completely broken. When libcurl concludes that a connection can't be kept for further re-use it will _close_ that socket. If it doesn't conclude that, it will keep the connection in its connection pool (and possibly close an older one to make room) but that pool will only fit a finite number of connections and that will be much less than 500 by default... All this connection will get closed after some time approximately 30 mins. So who or what is closing them after 30 minutes? -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Fetching More than 500 files from Server
On Thu, 14 Jun 2012, Swamy Mudhbasalar wrote: Please stop top-posting and full-quoting. If you intend to reply to emails here, switch off the digest-mode! LibCurl Version : 7.21.0 We have made 13 releases since that, so it's not unthinkable that a newer version works better or at least different. It would be very interesting to learn if you see anything similar with a recent version. curl_easy_setopt(ptCurlHandle, CURLOPT_ERRORBUFFER, acErrorBuffer); curl_easy_setopt(ptCurlHandle, CURLOPT_URL, pstrURL); curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEFUNCTION, VmwGetCB); curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEDATA, (void *)tContext); curl_easy_setopt(ptCurlHandle, CURLOPT_USERAGENT, pstrUserAgent); curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYPEER, 0L); curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYHOST, 0L); curl_easy_setopt(ptCurlHandle, CURLOPT_USERPWD, pstrUserPass); If these are truly the only options you use then the connection: header in the request is truly a mystery to me... You could consider using CURLOPT_VERBOSE to get to see about libcurl's choices to keep connections open or close them etc. In which sense is this connection then still alive ? How do you see it being kept in 30 minutes? When I run the netstat command from the command prompt, i can see many connection established. For every file there is one connection established. Even ESTABLISHED! That sounds so completely broken. When libcurl concludes that a connection can't be kept for further re-use it will _close_ that socket. If it doesn't conclude that, it will keep the connection in its connection pool (and possibly close an older one to make room) but that pool will only fit a finite number of connections and that will be much less than 500 by default... All this connection will get closed after some time approximately 30 mins. So who or what is closing them after 30 minutes? -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Fetching More than 500 files from Server
On 13 Jun 2012, at 05:38, Swamy Mudhbasalar wrote: 1. In all HTTP request sent and recieved observed Connection: close ? 2. How to make Connection : Alive and use the same connection to fetch all the files? What need to added specificalling in the code to do this ? 3.Why connections are not getting closed and they remain idle for 30 mins? If the response contains Connection: close, it would be a good idea to cleanup that curl easy handle. The server is dropping a strong hint not to use HTTP keepalive. If you want something more robust / faster, you can use 2 parallel connections (which will need extra programming). -- Tim Bannister – is...@jellybaby.net smime.p7s Description: S/MIME cryptographic signature --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Fetching More than 500 files from Server
On Wed, 13 Jun 2012, Swamy Mudhbasalar wrote: We are try to fetch files from the ESX server 4.0 through HTTP request using curl library in C program. Using which libcurl version? Using which API? On what operating system? I am using a single easy curl handle to fetch all the files to take advantage of persistent connection. But, When tried to fetch more than 500 files from the ESX 4.0 using curl library fails. ... Each time while fetching a file a new connection is established. So when more than 500 connection is established and fetching of files stops. The established connections are not getting closed and they remain idle for 30 mins. Why is there new ones created and if new ones are created, how come the old ones are still kept around? That's not something libcurl makes happen. libcurl tries very hard to re-use connections as much as possible. HTTP Client request Sent: Host: dev-vcenter-4-2 User-Agent: gSOAP/2.7 Content-Type: text/xml; charset=utf-8 Content-Length: 504 Connection: close Cookie: vmware_soap_session=C762CF10-B1DF-427A-9E0C-9EB851FFDE3D;$Domain=dev-vcenter-4-2 Cookie: ;$Path=/sdk;$Domain=dev-vcenter-4-2 SOAPAction: urn:vim25/4.1 ... Connection: close is something you make libcurl send, as it wouldn't select to send that by itself. Connection: close In which sense is this connection then still alive ? How do you see it being kept in 30 minutes? I have a hard time to accept this problem description. 1. In all HTTP request sent and recieved observed Connection: close ? That's caused by your program's use of libcurl. 2. How to make Connection : Alive and use the same connection to fetch all the files? What need to added specificalling in the code to do this ? HTTP 1.1 doesn't need that header to be persistant, it is so by default! 3.Why connections are not getting closed and they remain idle for 30 mins? You haven't explained with enough details for us to understand what exactly you're seeing and how you're using libcurl so this really isn't possible to answer to without a lot of guessing. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Fetching More than 500 files from Server
Hello,
Using which libcurl version? Using which API? On what operating system?
LibCurl Version : 7.21.0
Operating System : Microsoft Windows Server 2003
Below APIs are used.
main()
{
ptCurlHandle = curl_easy_init();
while(NoofFiles != 0)
{
DownloadFiles();
NoofFiles--;
}
curl_easy_cleanup(ptCurlHandle);
}
Void DownloadFiles()
{
curl_easy_setopt(ptCurlHandle, CURLOPT_ERRORBUFFER, acErrorBuffer);
curl_easy_setopt(ptCurlHandle, CURLOPT_URL, pstrURL);
curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEFUNCTION, VmwGetCB);
curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEDATA, (void *)tContext);
curl_easy_setopt(ptCurlHandle, CURLOPT_USERAGENT, pstrUserAgent);
curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYHOST, 0L);
curl_easy_setopt(ptCurlHandle, CURLOPT_USERPWD, pstrUserPass);
curl_easy_perform(ptCurlHandle);
}
The ptCurlHandle is a gobal variable. curl_easy_init is called initially.
DownloadFiles() is called in a loop, the loop terminates when all
files are downloaded.
pstrURL will contains the path of the file to be downloaded.
We are try to fetch files of Linux machine (On Which ESX is installed)
from Window machines.
In which sense is this connection then still alive ? How do you see it
being
kept in 30 minutes?
When I run the netstat command from the command prompt, i can see many
connection established. For every file there is one connection
estalished. It can also be seen in vSphere client.
For example:
netsats
Active Connections
Proto Local Address Foreign AddressState
TCPdev-vcbproxydev-vcenter-4-2ESTABLISHED
TCPdev-vcbproxy dev-vcenter-4-2ESTABLISHED
.
.
.
TCPdev-vcbproxy dev-vcenter-4-2ESTABLISHED
All this connection will get closed after some time approximately 30 mins.
Since we are fetching more than 500 files we can see more than 500
Active Connections.
Regards,
SAM
===
On 6/13/12, curl-library-requ...@cool.haxx.se
curl-library-requ...@cool.haxx.se wrote:
Send curl-library mailing list submissions to
curl-library@cool.haxx.se
To subscribe or unsubscribe via the World Wide Web, visit
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-library
or, via email, send a message with subject or body 'help' to
curl-library-requ...@cool.haxx.se
You can reach the person managing the list at
curl-library-ow...@cool.haxx.se
When replying, please edit your Subject line so it is more specific
than Re: Contents of curl-library digest...
Today's Topics:
1. Re: Fetching More than 500 files from Server (Daniel Stenberg)
2. RE: SFTP File already completely downloaded but the file is
empty (NEDJARI Hafed)
3. Re: curl_schannel.c and realloc() (Marc Hoersken)
4. Re: curl_schannel.c and realloc() (Daniel Stenberg)
5. RE: SFTP File already completely downloaded but the file is
empty (Daniel Stenberg)
6. RE: Windows SSPI Schannel implementation ready (Steve Holme)
7. Re: Windows SSPI Schannel implementation ready (Yang Tse)
8. Re: Windows SSPI Schannel implementation ready (Marc Hoersken)
--
Message: 1
Date: Wed, 13 Jun 2012 12:00:32 +0200 (CEST)
From: Daniel Stenberg dan...@haxx.se
To: libcurl development curl-library@cool.haxx.se
Subject: Re: Fetching More than 500 files from Server
Message-ID: alpine.deb.2.00.1206131155050.14...@tvnag.unkk.fr
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
On Wed, 13 Jun 2012, Swamy Mudhbasalar wrote:
We are try to fetch files from the ESX server 4.0 through HTTP request
using
curl library in C program.
Using which libcurl version? Using which API? On what operating system?
I am using a single easy curl handle to fetch all the files to take
advantage of persistent connection. But, When tried to fetch more than 500
files from the ESX 4.0 using curl library fails.
...
Each time while fetching a file a new connection is established. So when
more than 500 connection is established and fetching of files stops. The
established connections are not getting closed and they remain idle for 30
mins.
Why is there new ones created and if new ones are created, how come the old
ones are still kept around? That's not something libcurl makes happen.
libcurl
tries very hard to re-use connections as much as possible.
HTTP Client request Sent:
Host: dev-vcenter-4-2
User-Agent: gSOAP/2.7
Content-Type: text/xml; charset=utf-8
Content-Length: 504
Connection: close
Cookie:
vmware_soap_session=C762CF10-B1DF-427A-9E0C-9EB851FFDE3D;$Domain=dev-vcenter-4-2
Cookie: ;$Path=/sdk;$Domain=dev-vcenter-4-2
SOAPAction: urn:vim25/4.1
Fetching More than 500 files from Server
Hello, Problem Description: We are try to fetch files from the ESX server 4.0 through HTTP request using curl library in C program. I am using a single easy curl handle to fetch all the files to take advantage of persistent connection. But, When tried to fetch more than 500 files from the ESX 4.0 using curl library fails. Cause : Each time while fetching a file a new connection is established. So when more than 500 connection is established and fetching of files stops. The established connections are not getting closed and they remain idle for 30 mins. Observation : HTTP Client request Sent: Host: dev-vcenter-4-2 User-Agent: gSOAP/2.7 Content-Type: text/xml; charset=utf-8 Content-Length: 504 Connection: close Cookie: vmware_soap_session=C762CF10-B1DF-427A-9E0C-9EB851FFDE3D;$Domain=dev-vcenter-4-2 Cookie: ;$Path=/sdk;$Domain=dev-vcenter-4-2 SOAPAction: urn:vim25/4.1 HTTP Server Reponse Recieved : HTTP/1.1 200 OK Date: Thu, 7 Jun 2012 13:36:29 GMT Set-Cookie: vmware_soap_session=C762CF10-B1DF-427A-9E0C-9EB851FFDE3D; Path=/; Cache-Control: no-cache Connection: close Content-Type: text/xml; charset=utf-8 Content-Length: 1934 Questions : 1. In all HTTP request sent and recieved observed Connection: close ? 2. How to make Connection : Alive and use the same connection to fetch all the files? What need to added specificalling in the code to do this ? 3.Why connections are not getting closed and they remain idle for 30 mins? Regards, SAM --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
