Re: Fetching More than 500 files from Server
On 13 Jun 2012, at 05:38, Swamy Mudhbasalar wrote: 1. In all HTTP request sent and recieved observed Connection: close ? 2. How to make Connection : Alive and use the same connection to fetch all the files? What need to added specificalling in the code to do this ? 3.Why connections are not getting closed and they remain idle for 30 mins? If the response contains Connection: close, it would be a good idea to cleanup that curl easy handle. The server is dropping a strong hint not to use HTTP keepalive. If you want something more robust / faster, you can use 2 parallel connections (which will need extra programming). -- Tim Bannister – is...@jellybaby.net smime.p7s Description: S/MIME cryptographic signature --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Reading SSL certs from buffer
Hello all. Have question about feature that i've implemented in curl locally for my purposes. I have CYASSL + CURL. In my project there is a need to set Certs not from file but using a buffer. I haven't found that possible in current implementation. I have implemented it using list approach like with slist. So several buffers can be joined together. List node stores pointer to buffer (not the data itself pointed by), length of data and format. Guys, if you interested in such a feature i can prepare a patches and send it. Thanks you in advance. -- WBR Dmitry E. Ponomarev --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: curl_schannel.c and realloc()
On Wed, 13 Jun 2012, Marc Hoersken wrote: If this is the case, we can probably just set CURL_SCHANNEL_BUFFER_INIT_SIZE to BUFSIZE in order to reduce the number of realloc() calls. I would personally keep the code to gracefully handle the need for more buffer space. I was first going to agree and then a second thought struck me. Why would it ever need to handle more data? If it gets called asking for 20K of data, there's nothing in the API that says the function must return that much. We can safely just make BUFSIZE the maximum amount of data the schannel_recv() function can return without it breaking any properly written code! It would simplify the code without breaking anything... Then, as a follow-up improvement the code could probably use the 'buf' buffer immediately instead of separately storing received data in a malloc'ed buffer that is then memcpy()'ed to 'buf'. It would save one malloc/free of a 16K buffer, and also skip memcpy()ing all data that is received. (It will be copied at least once afterwards anyway.) -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Fetching More than 500 files from Server
On Wed, 13 Jun 2012, Swamy Mudhbasalar wrote: We are try to fetch files from the ESX server 4.0 through HTTP request using curl library in C program. Using which libcurl version? Using which API? On what operating system? I am using a single easy curl handle to fetch all the files to take advantage of persistent connection. But, When tried to fetch more than 500 files from the ESX 4.0 using curl library fails. ... Each time while fetching a file a new connection is established. So when more than 500 connection is established and fetching of files stops. The established connections are not getting closed and they remain idle for 30 mins. Why is there new ones created and if new ones are created, how come the old ones are still kept around? That's not something libcurl makes happen. libcurl tries very hard to re-use connections as much as possible. HTTP Client request Sent: Host: dev-vcenter-4-2 User-Agent: gSOAP/2.7 Content-Type: text/xml; charset=utf-8 Content-Length: 504 Connection: close Cookie: vmware_soap_session=C762CF10-B1DF-427A-9E0C-9EB851FFDE3D;$Domain=dev-vcenter-4-2 Cookie: ;$Path=/sdk;$Domain=dev-vcenter-4-2 SOAPAction: urn:vim25/4.1 ... Connection: close is something you make libcurl send, as it wouldn't select to send that by itself. Connection: close In which sense is this connection then still alive ? How do you see it being kept in 30 minutes? I have a hard time to accept this problem description. 1. In all HTTP request sent and recieved observed Connection: close ? That's caused by your program's use of libcurl. 2. How to make Connection : Alive and use the same connection to fetch all the files? What need to added specificalling in the code to do this ? HTTP 1.1 doesn't need that header to be persistant, it is so by default! 3.Why connections are not getting closed and they remain idle for 30 mins? You haven't explained with enough details for us to understand what exactly you're seeing and how you're using libcurl so this really isn't possible to answer to without a lot of guessing. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: SFTP File already completely downloaded but the file is empty
Many thanks for you quickly reply. As the file I try to download is not empty,
I have to redirect my research to libssh2.
Do you think I can believe to get an answer from the libssh2 Team ?
For me, you can close my request to libcurl development.
Hafed
-Message d'origine-
De : curl-library-boun...@cool.haxx.se
[mailto:curl-library-boun...@cool.haxx.se] De la part de Daniel Stenberg
Envoyé : mercredi 13 juin 2012 00:41
À : libcurl development
Objet : RE: SFTP File already completely downloaded but the file is empty
On Tue, 12 Jun 2012, NEDJARI Hafed wrote:
* File already completely downloaded
... this originates from this source code:
/* Setup the actual download */
if(data-req.size == 0) {
/* no data to transfer */
Curl_setup_transfer(conn, -1, -1, FALSE, NULL, -1, NULL);
infof(data, File already completely downloaded\n);
state(conn, SSH_STOP);
break;
}
It looks like libssh2_sftp_stat_ex() returned and said the file size is zero
bytes... I suggest you research that a bit closer.
--
/ daniel.haxx.se
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: curl_schannel.c and realloc()
Hi Daniel, 2012/6/13 Daniel Stenberg dan...@haxx.se On Wed, 13 Jun 2012, Marc Hoersken wrote: If this is the case, we can probably just set CURL_SCHANNEL_BUFFER_INIT_SIZE to BUFSIZE in order to reduce the number of realloc() calls. I would personally keep the code to gracefully handle the need for more buffer space. I was first going to agree and then a second thought struck me. Why would it ever need to handle more data? If it gets called asking for 20K of data, there's nothing in the API that says the function must return that much. We can safely just make BUFSIZE the maximum amount of data the schannel_recv() function can return without it breaking any properly written code! It would simplify the code without breaking anything... That's a good plan. Then, as a follow-up improvement the code could probably use the 'buf' buffer immediately instead of separately storing received data in a malloc'ed buffer that is then memcpy()'ed to 'buf'. It would save one malloc/free of a 16K buffer, and also skip memcpy()ing all data that is received. (It will be copied at least once afterwards anyway.) The problem is that we still need to buffer the received encrypted and unencrypted data since such data can already arive during the initial SSL/TLS handshake handled by the step-functions. Therefore we need the buffer to have the data available for the next read. And we also need this buffer between reads, because there might be an incomplete encrpyted SSL/TLS packet in the queue. Also there might be more data decrypted by DecryptMessage than the user wants to read, so we need to store that already decrypted data, too. Basically we have to do all the buffering around the DecryptMessage function as it is not guaranteed that it decrypts the exact amount of data we want to read or that all encrypted data passed into it is really decrypted afterwards. Best regards, Marc --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: curl_schannel.c and realloc()
On Wed, 13 Jun 2012, Marc Hoersken wrote: Then, as a follow-up improvement the code could probably use the 'buf' buffer immediately instead of separately storing received data in a malloc'ed buffer that is then memcpy()'ed to 'buf'. The problem is that we still need to buffer the received encrypted and unencrypted data since such data can already arive during the initial SSL/TLS handshake handled by the step-functions. Therefore we need the buffer to have the data available for the next read. And we also need this buffer between reads, because there might be an incomplete encrpyted SSL/TLS packet in the queue. Also there might be more data decrypted by DecryptMessage than the user wants to read, so we need to store that already decrypted data, too. Oh right. Sorry for suggsting such things without actually checking out the code carefully enough to say if it is indeed possible to do. Thanks for the details! -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: SFTP File already completely downloaded but the file is empty
On Wed, 13 Jun 2012, NEDJARI Hafed wrote: Many thanks for you quickly reply. As the file I try to download is not empty, I have to redirect my research to libssh2. Yes, and I would then recommend that you get the latest libssh2, build it with --enable-debug and then post a full debug trace (showing the STAT call that gets the zero file size) to the libssh2-devel list for analysis. Do you think I can believe to get an answer from the libssh2 Team ? I'm the maintainer of libssh2 as well. I believe you'll get some attention to your problem yes, but until we can repeat your problem in our ends we rely on you to get more details and to do the actual debugging... -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: Windows SSPI Schannel implementation ready
Hi all, Get well soon. Thank you for the well wishes - I got a good night's sleep but am still feeling pretty crap this morning :( I'm dosed up on Ibuprofen at the moment to try and reduce the fever but I might have to nip out in a bit and grab some paracetamol based cold and flue type remedies. As Gün sugested SSL-Windows-native or WinSSL without version number information. But, only when actually using Windows SSL implementation. Unless I'm wrong, that is when schannel is in use not simply when SSPI is used. I'm not opposed to not including the version number - this would be consistent to what WinIDN displays, however, I also think including the version number would also be consistent with all the other libraries that we display. I know this is a community effort but perhaps some direction from Daniel and whether the version number, as a rule of thumb, should be included for all items in the version string or not is needed here. I also think, as per the discussion I started 6 weeks ago which I thought we had decided to do, hence my work here, was that the package name WinSSPI, Windows SSPI or SSPI-Windows-native should be displayed for the other features that SSPI offers not just the SChannel SSL support - again this is synonymous to the other Security Providers that curl uses and provides consistency. The inclusion of SSPI in curl has obviously changed a fair amount since it was first included as a feature and that was why I recommended including it in the version string back in April and moving to a scenario where SSPI isn't listed in the features list ;-) For API compatibility we have decided to keep it in the features list as well as listing it in the package / version string. If this is something that you had a strong opinion on, and I'm still not sure if it is or not, then why didn't you let us know 6 weeks ago before I spent two days doing the rework and learning curl's makefiles when I didn't need to. Mostly library version number given for a system library. I don't have an issue with that, whilst others Guenter, Marc, Gisle et all don't seem to mind either. The only possible objection I have is... that version.lib is currently being linked to statically and if that is a problem for running curl on 12-year old+ versions of Windows then we *should* consider dynamically including it like we do with secur32.dll for example. S. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Steve Holme steve_ho...@hotmail.com wrote: I'm not opposed to not including the version number - this would be consistent to what WinIDN displays, [...] Ok, then we have consensus then. I also think, as per the discussion I started 6 weeks ago which I thought we had decided to do, hence my work here, was that the package name WinSSPI, Windows SSPI or SSPI-Windows-native should be displayed for the other features that SSPI offers not just the SChannel SSL support - again this is synonymous to the other Security Providers that curl uses and provides consistency. I asked for a patch april 23. http://curl.haxx.se/mail/lib-2012-04/0259.html It has not been provided until june 10. http://curl.haxx.se/mail/lib-2012-06/0111.html One of the reasons for which I personally dislike big patches is that these usually hide changes which are not properly discussed, or discussed with such lengthy threads that no one knows finally what's going on, making it necessary to fully analyze resulting patch in order to know what changes it really introduces. This is what I'm doing now. I might have further objections, so don't be surprised if I mention or fix something else. Given that it seems we've reached consensus on that you can live without the numeric part of the string, and that I can live with some schannel specific identifier, I'm pushing right now a patch with the following commit message: schannel: remove version number and identify its use with 'schannel' literal Version number is removed in order to make this info consistent with how we do it with other MS and Linux system libraries for which we don't provide this info. Identifier changed from 'WinSSPI' to 'schannel' given that this is the actual provider of the SSL/TLS support. libcurl can still be built with SSPI and without SCHANNEL support. -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Hello everyone, 2012/6/13 Yang Tse yangs...@gmail.com: One of the reasons for which I personally dislike big patches is that these usually hide changes which are not properly discussed, or discussed with such lengthy threads that no one knows finally what's going on, making it necessary to fully analyze resulting patch in order to know what changes it really introduces. This is what I'm doing now. I might have further objections, so don't be surprised if I mention or fix something else. This is why I posted the full commit history during the early development stages and especially separated even squashed commits to avoid mixing up changes to different areas. While you say that an updated patch wasn't provided until June, the other changes have been around since April and the full commit history including separate small patches was available from the beginning. The new changes introducing the updated version information were also separated and can be easily identified in the history. Given that it seems we've reached consensus on that you can live without the numeric part of the string, and that I can live with some schannel specific identifier, I'm pushing right now a patch with the following commit message: schannel: remove version number and identify its use with 'schannel' literal Version number is removed in order to make this info consistent with how we do it with other MS and Linux system libraries for which we don't provide this info. Identifier changed from 'WinSSPI' to 'schannel' given that this is the actual provider of the SSL/TLS support. libcurl can still be built with SSPI and without SCHANNEL support. Actually schannel is not the correct identifier. It's either Schannel or Secure Schannel. Please take a look at the MSDN documentation before doing such changes: http://msdn.microsoft.com/en-us/library/windows/desktop/ms678421.aspx I see that you changed the winbuild scripts to automatically set USE_SSPI to yes and USE_SCHANNEL to true if USE_SSL is false. Why did you do that? That makes it impossible to build a Windows version without SSL support. Before your changes Schannel was only enabled if SSPI was enabled and OpenSSL was not. Besides that, I liked the idea of having the low level Windows libaries in a separate WINLIBS variable in the makefile. Why did you revert this cleanup change, too? Best regards, Marc --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Hi, Am 13.06.2012 16:40, schrieb Yang Tse: One of the reasons for which I personally dislike big patches is that these usually hide changes which are not properly discussed, or discussed with such lengthy threads that no one knows finally what's going on, making it necessary to fully analyze resulting patch in order to know what changes it really introduces. This is what I'm doing now. I might have further objections, so don't be surprised if I mention or fix something else. but then we did make a mistake: we saw this huge patch coming, and it was then wrong to encorage Marc to work further in his fork until its ready - instead we should have granted him commit rights and told him to push his changes one by one right after the last release. just my 2ct. Gün. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: Windows SSPI Schannel implementation ready
Hi Yang, I'm not opposed to not including the version number - this would be consistent to what WinIDN displays, [...] Ok, then we have consensus then. On the version number yes, on the SChannel literal no as this should be SSPI - you wouldn't list either libssl32 or libeay32 for OpenSSL !! I also think, as per the discussion I started 6 weeks ago which I thought we had decided to do, hence my work here, was that the package name WinSSPI, Windows SSPI or SSPI-Windows-native should be displayed for the other features that SSPI offers not just the SChannel SSL support - again this is synonymous to the other Security Providers that curl uses and provides consistency. I asked for a patch april 23. http://curl.haxx.se/mail/lib-2012-04/0259.html As Marc has already mentioned the commit history of the original work has been available for a long time now. My rework has been available since the 22nd April. The work you reverted on the 23 April was made because of some points over SSPI vs SSO #defines and not the version number rework itself - these issues were then addressed on the forums between the 13 May and the 16 May - no additional input was provided by you during that conversation. Given that it seems we've reached consensus on that you can live without the numeric part of the string, and that I can live with some schannel specific identifier, I'm pushing right now a patch with the following commit message: schannel: remove version number and identify its use with 'schannel' literal This is the exact opposite of what I have been saying - Windows SSPI is a provider of security features like GNUTLS is and should be recognised as such. Like you also said we don't list the individual features in the package / version string so why list SChannel on its own? Identifier changed from 'WinSSPI' to 'schannel' given that this is the actual provider of the SSL/TLS support. libcurl can still be built with SSPI and without SCHANNEL support. I will have a look at the change you are putting in but from reading your reply here, you seem to have completely ignored everything I have said on this matter and any contribution I have made. I have provided good argument for including Windows SSPI as the package name and for the inclusion in the version string for both SChannel based SSL and for without. I have had agreement from others here and to that degree I can only conclude that you simple do what you want when you don't agree with what has been said. In that respect I can only thank you for wasting my time on this - 2 days of development and several hours of emails. Steve --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Folks, to shorten this never-ending discussion I did just change the string to SSL-Windows-native for 3 reasons: 1) this describes how SSL is supported in a way the average user understands; schannel is something which users might not even know about what it is at all. 2) It is conform with what we show as version when we build with native IDN support. 3) At least we have 2 of us agree on this (me and Marc). TODOs: 1) we should make sure this string appears *only* if SSL functionality is provided via schannel, sspi, whatever ... 2) it should remain possible to build with SSPI auth stuff and without native SSL stuff - we need still *two* defines 3) SSPI listed as feature if enabled must remain (as Daniel stated), and it should also be possible to build with SSL-native but without SSPI auth stuff. 4) we should should make sure that this SSL-native support is called equal allover the project files, f.e. WINSSL (- makefiles) 5) we need to teach configure to build with SSL-native; I propose: --winssl OT, but in the same context: we need also such an option for IDN-native, I propose here: --winidn please lets concentrate on these things now so that we can soon start to add some autobuilds for the new stuff! Gün. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
On Wed, 13 Jun 2012, Guenter wrote: 5) we need to teach configure to build with SSL-native; I propose: --winssl I assume you meant that to be --with-winssl? That seems more like regular configure style. (It is also done more or less automatically with the use of autoconf macros.) we need also such an option for IDN-native, I propose here: --winidn --with-winidn? please lets concentrate on these things now so that we can soon start to add some autobuilds for the new stuff! Thanks for this Guenter. I am still interested to hear what issues or debates we have left to sort out after Guenter's points have been addressed. I'm thinking we could try to have a vote about the issues that we can't reach consensus on. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Am 13.06.2012 20:24, schrieb Daniel Stenberg: On Wed, 13 Jun 2012, Guenter wrote: 5) we need to teach configure to build with SSL-native; I propose: --winssl I assume you meant that to be --with-winssl? That seems more like yup, of course! sorry ... regular configure style. (It is also done more or less automatically with the use of autoconf macros.) we need also such an option for IDN-native, I propose here: --winidn --with-winidn? sure. Gün. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: Windows SSPI Schannel implementation ready
Hi Guenter, to shorten this never-ending discussion I did just change the string to SSL-Windows-native for 3 reasons: [...] 3) At least we have 2 of us agree on this (me and Marc). Three (me) - I was simply against the package name as SChannel, although Marc, Yang and I didn't have a problem with the string WinSSPI although what you suggested is consistent with the IDN string and people that know me, know I have an issue with consistency ;-) TODOs: 1) we should make sure this string appears *only* if SSL functionality is provided via schannel, sspi, whatever ... I still think it is worth having the string there when SSPI is included when it compiled in security context mode / SSO (auth stuff as you call it) as it is using the same library. For example, you don't remove the libssh string when it isn't compiled with zlib ;-) I slight bad analogy but hopefully you understand my point there. Now without checking email history yet again, and to be honest I am just about out of patients on this, considering the amount of time and effort I have already put into it, I thought Marc and Daniel agreed here. 2) it should remain possible to build with SSPI auth stuff and without native SSL stuff - we need still *two* defines I'm a little confused here... are you talking about makefile options or the USE_* #defines or something else? In Visual Studio I had curl building with 1) Just the security context / SSO capabilities and 2) These plus SChannel SSL with last night's version of the code base - I haven't rebuilt todays changes so I don't know if something has changed to affect that. USE_WINDOWS_SSPI should build just the security context / SSO capabilities. USE_WINDOWS_SSPI and USE_SCHANNEL should additionally build in the SChannel SSL stuff. Kind Regards Steve --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Daniel Stenberg dan...@haxx.se wrote: I assume you meant that to be --with-winssl? That seems more like regular configure style. (It is also done more or less automatically with the use of autoconf macros.) Done. Just pushed. -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Marc Hoersken i...@marc-hoersken.de wrote: Identifier changed from 'WinSSPI' to 'schannel' given that this is the actual provider of the SSL/TLS support. libcurl can still be built with SSPI and without SCHANNEL support. Actually schannel is not the correct identifier. It's either Schannel or Secure Schannel. Please take a look at the MSDN documentation before doing such changes: http://msdn.microsoft.com/en-us/library/windows/desktop/ms678421.aspx Personally I don't care how we call it as long as the literal that we use to identify that SSL is provided by MS libraries does not include the 'SSPI' literal. For the reasons I've given in my previous mail and commit message. OTOH every half baked Windows user, and all power-users, knows that schannel is the 'thingy' to hack in the registry to modify SSL behavior. But I have no problem with calling it 'SSL-Windows-native'. I see that you changed the winbuild scripts to automatically set USE_SSPI to yes and USE_SCHANNEL to true if USE_SSL is false. Why did you do that? That makes it impossible to build a Windows version without SSL support. Before your changes Schannel was only enabled if SSPI was enabled and OpenSSL was not. Fixed now. define USE_WINSSL to build with both SSPI and schannel support. Besides that, I liked the idea of having the low level Windows libaries in a separate WINLIBS variable in the makefile. Why did you revert this cleanup change, too? Because in order to use debug and release versions of windows libs it is easier to keep it as it was previously. -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: Windows SSPI Schannel implementation ready
Yang, OTOH every half baked Windows user, and all power-users, knows that schannel is the 'thingy' to hack in the registry to modify SSL behavior. But I have no problem with calling it 'SSL-Windows-native'. Personally I find that quite offensive. I have been programming Windows since v2.0 and have used SSPI on a number of occasions for Security Contexts but not actually used SChannel. I might have read about it and glossed over it but as a software engineer and power user I wasn't aware of its usage fully until Marc's work. I have stated time and time again that the literal should include SSPI because, as I understand it, and please correct me if I am wrong and point me to the MSDN documentation but SSPI is the interface to all the services that it provides. Kind Regards Steve --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Steve Holme steve_ho...@hotmail.com wrote: I'm not opposed to not including the version number - this would be consistent to what WinIDN displays, [...] Ok, then we have consensus then. On the version number yes, on the SChannel literal no as this should be SSPI - you wouldn't list either libssl32 or libeay32 for OpenSSL !! Nope. libcurl can be built with SSPI support using other SSL libraries as it has been done up to nowadays. And this is one of the things that must be kept around. I also think, as per the discussion I started 6 weeks ago which I thought we had decided to do, hence my work here, was that the package name WinSSPI, Windows SSPI or SSPI-Windows-native should be displayed for the other features that SSPI offers not just the SChannel SSL support - again this is synonymous to the other Security Providers that curl uses and provides consistency. I asked for a patch april 23. http://curl.haxx.se/mail/lib-2012-04/0259.html As Marc has already mentioned the commit history of the original work has been available for a long time now. My rework has been available since the 22nd April. The work you reverted on the 23 April was made because of some points over SSPI vs SSO #defines and not the version number rework itself - these issues were then addressed on the forums between the 13 May and the 16 May - no additional input was provided by you during that conversation. Given that no answer, neither with patch nor without it, to my april 23 mail requesting a patch was given by any of you. I lost any interest in any development that was going outside of libcurl repo. Assuming that the time would come when you would bring all that work either as a patch or somehow and that would be the moment to review it. schannel: remove version number and identify its use with 'schannel' literal This is the exact opposite of what I have been saying - Windows SSPI is a provider of security features like GNUTLS is and should be recognised as such. Like you also said we don't list the individual features in the package / version string so why list SChannel on its own? Because it is the one thanks to which SSL works on Windows. As an exercise remove schannel.dll and you'll see that SSL stops working. MS loves to put layers and layers in between things. SSPI is a 'user' of schannel, the same as many other windows parts. Identifier changed from 'WinSSPI' to 'schannel' given that this is the actual provider of the SSL/TLS support. libcurl can still be built with SSPI and without SCHANNEL support. I will have a look at the change you are putting in but from reading your reply here, you seem to have completely ignored everything I have said on this matter and any contribution I have made. I really appreciate every one's work. The fact that it might need some adjustments doesn't imply the contrary. I have provided good argument for including Windows SSPI as the package name and for the inclusion in the version string for both SChannel based SSL and for without. I have had agreement from others here and to that degree I can only conclude that you simple do what you want when you don't agree with what has been said. In that respect I can only thank you for wasting my time on this - 2 days of development and several hours of emails. Hmmm, when you calm down and read this when some time passes by you might realize how groundless this last paragraph is. Good luck and get well soon. -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
On 13 Jun 2012, at 16:04, Marc Hoersken wrote: Actually schannel is not the correct identifier. It's either Schannel or Secure Schannel. Please take a look at the MSDN documentation before doing such changes: http://msdn.microsoft.com/en-us/library/windows/desktop/ms678421.aspx I've never done any secure network coding on Windows at this level, but where you put “Secure Schannel” did you mean to type “Secure Channel”? cf. http://msdn.microsoft.com/en-us/library/windows/desktop/aa380123(v=vs.85).aspx I think users would expect to see the name “Schannel”. SSPI is Microsoft's proprietary variant of GSSAPI. -- Tim Bannister – is...@jellybaby.net smime.p7s Description: S/MIME cryptographic signature --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Steve Holme steve_ho...@hotmail.com wrote: OTOH every half baked Windows user, and all power-users, knows that schannel is the 'thingy' to hack in the registry to modify SSL behavior. But I have no problem with calling it 'SSL-Windows-native'. Personally I find that quite offensive. I have been programming Windows since v2.0 and have used SSPI on a number of occasions for Security Contexts but not actually used SChannel. I might have read about it and glossed over it but as a software engineer and power user I wasn't aware of its usage fully until Marc's work. Why do you take my comment as something personal? I really don't understand this. I have stated time and time again that the literal should include SSPI because, as I understand it, and please correct me if I am wrong and point me to the MSDN documentation but SSPI is the interface to all the services that it provides. Hey, I'm not saying SSPI isn't the outer interface. I'm saying that we can build a libcurl version wich uses SSPI and makes no use of schannel. You want a MS link which speaks about schannel without mentioning SSPI? http://support.microsoft.com/kb/245030 -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
CyaSSL problem in ldap.c
From gcc/MingW:
In file included from ../../CyaSSL-2.2.0/cyassl/openssl/ssl.h:31:0,
from urldata.h:115,
from ldap.c:68:
../../CyaSSL-2.2.0/cyassl/ssl.h:422:5: error: expected identifier before '('
token
Reason being that OCSP_REQUEST+OCSP_RESPONSE are enum values
in CyaSSL and defines in wincrypt.h included via winldap.h in ldap.c.
A fix is to #undef-ine them before ldap.c hits them again via urldata.h:
diff --git a/lib/urldata.h b/lib/urldata.h
index 20e339b..590d79a 100644
--- a/lib/urldata.h
+++ b/lib/urldata.h
@@ -112,6 +112,12 @@
#endif
#ifdef USE_CYASSL
+/*
+ * These are enum values in cyassl/ssl.h and defines in wincrypt.h
+ * included elsewhere (ldap.c).
+ */
+#undef OCSP_REQUEST
+#undef OCSP_RESPONSE
#include cyassl/openssl/ssl.h
#endif
--gv
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: curl_schannel.c and realloc()
On Wed, 13 Jun 2012, Marc Hoersken wrote: I was first going to agree and then a second thought struck me. Why would it ever need to handle more data? If it gets called asking for 20K of data, there's nothing in the API that says the function must return that much. We can safely just make BUFSIZE the maximum amount of data the schannel_recv() function can return without it breaking any properly written code! It would simplify the code without breaking anything... That's a good plan. Would you be able to write up a patch for this? As I don't build for or use Windows myself I will only mess up some little detail if I try to write this blindly ... -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
schannel and cacert verification
Hi guys, Am I right when I assume that the new schannel code uses the Windows cert store when a server's SSL certificate gets verified? -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: schannel and cacert verification
That's correct. Desktop windows has multiple cert stores - there is a machine store and a user store. The user store is what you see when you open up the Certificates view from IE. By default I think all code uses this store too. Mark -Original Message- From: curl-library-boun...@cool.haxx.se [mailto:curl-library-boun...@cool.haxx.se] On Behalf Of Daniel Stenberg Sent: Wednesday, June 13, 2012 3:18 PM To: libcurl hacking Subject: schannel and cacert verification Hi guys, Am I right when I assume that the new schannel code uses the Windows cert store when a server's SSL certificate gets verified? -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
2012/6/13 Tim Bannister is...@jellybaby.net: On 13 Jun 2012, at 16:04, Marc Hoersken wrote: Actually schannel is not the correct identifier. It's either Schannel or Secure Schannel. Please take a look at the MSDN documentation before doing such changes: http://msdn.microsoft.com/en-us/library/windows/desktop/ms678421.aspx I've never done any secure network coding on Windows at this level, but where you put “Secure Schannel” did you mean to type “Secure Channel”? cf. http://msdn.microsoft.com/en-us/library/windows/desktop/aa380123(v=vs.85).aspx Yes, that was actually a typo on my end. Sorry for that. Best regards, Marc --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: curl_schannel.c and realloc()
Hi Daniel, 2012/6/13 Daniel Stenberg dan...@haxx.se: On Wed, 13 Jun 2012, Marc Hoersken wrote: I was first going to agree and then a second thought struck me. Why would it ever need to handle more data? If it gets called asking for 20K of data, there's nothing in the API that says the function must return that much. We can safely just make BUFSIZE the maximum amount of data the schannel_recv() function can return without it breaking any properly written code! It would simplify the code without breaking anything... That's a good plan. Would you be able to write up a patch for this? As I don't build for or use Windows myself I will only mess up some little detail if I try to write this blindly ... yes, of course I will be able to provide a patch for this. But I would like to give it a little more time until the other SSPI discussions are settled. Or do you have any other recommendations? Best regards, Marc --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: curl_schannel.c and realloc()
On Wed, 13 Jun 2012, Marc Hoersken wrote: yes, of course I will be able to provide a patch for this. But I would like to give it a little more time until the other SSPI discussions are settled. Or do you have any other recommendations? I'm perfectly fine with that! -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: schannel and cacert verification
Hi there, -Original Message- From: curl-library-boun...@cool.haxx.se [mailto:curl-library-boun...@cool.haxx.se] On Behalf Of Daniel Stenberg Sent: Wednesday, June 13, 2012 3:18 PM To: libcurl hacking Subject: schannel and cacert verification Hi guys, Am I right when I assume that the new schannel code uses the Windows cert store when a server's SSL certificate gets verified? 2012/6/13 Salisbury, Mark mark.salisb...@hp.com: That's correct. Desktop windows has multiple cert stores - there is a machine store and a user store. The user store is what you see when you open up the Certificates view from IE. By default I think all code uses this store too. yes, the current implementation uses the cert stores mentioned by Mark. WinINET, WinHTTP, CryptoAPI, Schannel and all the other related Windows Security libraries are able to use those by default if some flags are specified. Additionally it is also possible to add custom client certificate and server certificate validation to those layers by using the lower-lever functionality provided by CryptoAPI and related libraries. This is why I added those features to the TODO list at the top of curl_schannel.c. As shown in the other thread, Mark already started on the validation feature as it is required for WinCE since the default validation options are not available there. Best regards, Marc --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: further schannel improvements
Salisbury, Mark mark.salisb...@hp.com wrote: Back in April I announced I had some experience creating a schannel implementation for libcurl and I would like to help contribute to the effort that Marc had started. Marc took a first review of these patches yesterday, I've rebased them on top of the more recent changes and I think they are ready to go. I'd appreciate any feedback or questions anyone has on my patches. First round, 0001-SSPI-related-code-Unicode-support-for-WinCE.patch introduces win32_multibyte.[ch], It would be interesting if these got renamed to something which starts with curl_ to ensure that our files never in life clash with anything from anywhere else. win32_multibyte.h lacks header inclusion guards such as HEADER_CURL_WIN32_MULTIBYTE_H Makes functions _curl_win32_UTF8_to_wchar() and _curl_win32_wchar_to_UTF8() available throughout libcurl, according to libcurl coding standards these should be renamed to Curl_win32_UTF8_to_wchar() and Curl_win32_wchar_to_UTF8() 0002-schannel-SSL-Use-standard-Curl-read-write-methods.patch curl_schannel.c introduces C++ style comment. 0003-schannel-SSL-Added-helper-methods-to-simplify-code.patch Contains spaces between '*' and pointer variable name in declaration of InitSecBuffer() and InitSecBufferDesc() 0004-schannel-SSL-Made-send-method-handle-unexpected-case.patch Ok, 0005-schannel-SSL-Handle-extra-data-buffer-during-handsha.patch Introduces C++ style comment. Replace while (true) statement with for (;;) to avoid compiler warnings. 0006-schannel-SSL-certificate-validation-on-WinCE.patch Introduces spaces between if and opening parenthesis. Introduces at least onr line longer than 79 chars. 0007-schannel-SSL-Implemented-SSL-shutdown.patch Alignment of arguments given to InitializeSecurityContext() should be lined up below the first argument following opening parenthesis. 0008-strerror.c-setup.h-fixed-a-couple-of-compile-issues-.patch I Might have missed something but from a first inspection, other than the mentioned above, it follows libcurl's coding standards pretty well. No other testing done at all. Thanks, -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: further schannel improvements
Hello everyone, 2012/6/13 Salisbury, Mark mark.salisb...@hp.com: Hello, Back in April I announced I had some experience creating a schannel implementation for libcurl and I would like to help contribute to the effort that Marc had started. Marc took a first review of these patches yesterday, I've rebased them on top of the more recent changes and I think they are ready to go. I'd appreciate any feedback or questions anyone has on my patches. Things to note: * Supporting Windows CE and XP/Vista/Win7 is one of my main requirements (I work for HP on devices which run various flavors of windows - Linux too - which use libcurl). Because libcurl relies on posix methods that WinCE doesn't define, to actually build libcurl for WinCE one needs to find a Windows CE posix library. I'm using wceelibcex right now (with some improvements). * I tried to build using the suggested nmake process. It didn't work for me - include paths weren't valid relative to where I was building. I tried building with a VS 2008 command shell. (do other people have problems running nmake too?) For me it was easy to create a VS 2008 project to build / test with. I have not modified the build files but I did add one new file - win32_multibyte.c - so I'd appreciate help getting this into the necessary makefiles. I'd like to thank Marc H for his efforts here. I think he's done a great job. I'd love for more people to test and provide feedback on this new schannel implementation. thanks Mark. I do also appreciate the work you spend on these patches to improve my Schannel implementation and especially adding write buffering as well as SSL/TLS shutdown behaviour. During git am I get the following warning: d:/Dev/curl/.git/rebase-apply/patch:182: trailing whitespace. connssl-cred-cred_handle, NULL, warning: 1 line adds whitespace errors. Besides that, I noticed that for example the following lines are longer than 79 chars: 88, 95, 143, 147, 149, 151, 152, 160, 184, 358, 407, 410, 411, 421, 452, 495, 528, 689, 783, 784, 786, and more after line 800. During compilation with mingw32 I get the following warnings: gcc -I. -I../include -g -O2 -Wall -fno-strict-aliasing -DBUILDING_LIBCURL -DUSE_WINDOWS_SSPI -DUSE_SCHANNEL -c curl_schannel.c curl_schannel.c: In function 'schannel_connect_step2': curl_schannel.c:412:32: warning: pointer targets in passing argument 2 of 'Curl_read_plain' differ in signedness [-Wpointer-sign] sendf.h:58:10: note: expected 'char *' but argument is of type 'unsigned char *' curl_schannel.c: In function 'schannel_recv': curl_schannel.c:913:21: warning: pointer targets in passing argument 2 of 'Curl_read_plain' differ in signedness [-Wpointer-sign] sendf.h:58:10: note: expected 'char *' but argument is of type 'unsigned char *' It would be great if you could fix those small issues. Aside from those warnings and smaller issues, applying those patches breaks Schannel support for me. Performing the following simple test command, curl just sits there and prints nothing: src\curl.exe https://www.google.de Doing the same command with verbose output enabled, works: src\curl.exe -v https://www.google.de I had similiar problems during the development of the first implementation and it always seemed to be a corner-case there curl was running too fast and it didn't receive all the required data for a decryption operation. I will try to debug this, but it was always very hard and required quite a lot of work to tune the receive buffering to handle those cases. You may have unintentionally broken something there. Best regards, Marc --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: CyaSSL problem in ldap.c
Gisle Vanem gva...@broadpark.no wrote: Reason being that OCSP_REQUEST+OCSP_RESPONSE are enum values in CyaSSL and defines in wincrypt.h included via winldap.h in ldap.c. A fix is to #undef-ine them before ldap.c hits them again via urldata.h: CyaSSL has already fixed this problem in development version http://www.yassl.com/forums/topic229-cyassl-220-names-clash-in-sslh.html But given that there's no CyaSSL fixed and released version available, I've pushed your fix to our repo. commit: 0d0893f2b9 Thanks, -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Reading SSL certs from buffer
I haven't. But I actually it is one function call. I'm sure that OpenSSL has that ability. Also I can check other SSL libs for it to workout. But I assume you are interested in such a feature, aren't you? In worst case we can do that only for those ssl libs that supports certs from buffer. For those which do not we can return some error code when setopt get called. I'm asking all these since I have limited resources and need carefully estimate that work and be surfe it is could be usefull not only for me. Thank you in advance. WBR, Dmitry Ponomarev On Jun 14, 2012, at 1:23 AM, Daniel Stenberg dan...@haxx.se wrote: On Wed, 13 Jun 2012, Дмитрий Пономарёв wrote: In my project there is a need to set Certs not from file but using a buffer. I haven't found that possible in current implementation. I have implemented it using list approach like with slist. So several buffers can be joined together. List node stores pointer to buffer (not the data itself pointed by), length of data and format. Guys, if you interested in such a feature i can prepare a patches and send it. Sounds very interesting! Have you looked anything at all at how hard/easy that is to do for any other SSL library? -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Fetching More than 500 files from Server
Hello,
Using which libcurl version? Using which API? On what operating system?
LibCurl Version : 7.21.0
Operating System : Microsoft Windows Server 2003
Below APIs are used.
main()
{
ptCurlHandle = curl_easy_init();
while(NoofFiles != 0)
{
DownloadFiles();
NoofFiles--;
}
curl_easy_cleanup(ptCurlHandle);
}
Void DownloadFiles()
{
curl_easy_setopt(ptCurlHandle, CURLOPT_ERRORBUFFER, acErrorBuffer);
curl_easy_setopt(ptCurlHandle, CURLOPT_URL, pstrURL);
curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEFUNCTION, VmwGetCB);
curl_easy_setopt(ptCurlHandle, CURLOPT_WRITEDATA, (void *)tContext);
curl_easy_setopt(ptCurlHandle, CURLOPT_USERAGENT, pstrUserAgent);
curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(ptCurlHandle, CURLOPT_SSL_VERIFYHOST, 0L);
curl_easy_setopt(ptCurlHandle, CURLOPT_USERPWD, pstrUserPass);
curl_easy_perform(ptCurlHandle);
}
The ptCurlHandle is a gobal variable. curl_easy_init is called initially.
DownloadFiles() is called in a loop, the loop terminates when all
files are downloaded.
pstrURL will contains the path of the file to be downloaded.
We are try to fetch files of Linux machine (On Which ESX is installed)
from Window machines.
In which sense is this connection then still alive ? How do you see it
being
kept in 30 minutes?
When I run the netstat command from the command prompt, i can see many
connection established. For every file there is one connection
estalished. It can also be seen in vSphere client.
For example:
netsats
Active Connections
Proto Local Address Foreign AddressState
TCPdev-vcbproxydev-vcenter-4-2ESTABLISHED
TCPdev-vcbproxy dev-vcenter-4-2ESTABLISHED
.
.
.
TCPdev-vcbproxy dev-vcenter-4-2ESTABLISHED
All this connection will get closed after some time approximately 30 mins.
Since we are fetching more than 500 files we can see more than 500
Active Connections.
Regards,
SAM
===
On 6/13/12, curl-library-requ...@cool.haxx.se
curl-library-requ...@cool.haxx.se wrote:
Send curl-library mailing list submissions to
curl-library@cool.haxx.se
To subscribe or unsubscribe via the World Wide Web, visit
http://cool.haxx.se/cgi-bin/mailman/listinfo/curl-library
or, via email, send a message with subject or body 'help' to
curl-library-requ...@cool.haxx.se
You can reach the person managing the list at
curl-library-ow...@cool.haxx.se
When replying, please edit your Subject line so it is more specific
than Re: Contents of curl-library digest...
Today's Topics:
1. Re: Fetching More than 500 files from Server (Daniel Stenberg)
2. RE: SFTP File already completely downloaded but the file is
empty (NEDJARI Hafed)
3. Re: curl_schannel.c and realloc() (Marc Hoersken)
4. Re: curl_schannel.c and realloc() (Daniel Stenberg)
5. RE: SFTP File already completely downloaded but the file is
empty (Daniel Stenberg)
6. RE: Windows SSPI Schannel implementation ready (Steve Holme)
7. Re: Windows SSPI Schannel implementation ready (Yang Tse)
8. Re: Windows SSPI Schannel implementation ready (Marc Hoersken)
--
Message: 1
Date: Wed, 13 Jun 2012 12:00:32 +0200 (CEST)
From: Daniel Stenberg dan...@haxx.se
To: libcurl development curl-library@cool.haxx.se
Subject: Re: Fetching More than 500 files from Server
Message-ID: alpine.deb.2.00.1206131155050.14...@tvnag.unkk.fr
Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII
On Wed, 13 Jun 2012, Swamy Mudhbasalar wrote:
We are try to fetch files from the ESX server 4.0 through HTTP request
using
curl library in C program.
Using which libcurl version? Using which API? On what operating system?
I am using a single easy curl handle to fetch all the files to take
advantage of persistent connection. But, When tried to fetch more than 500
files from the ESX 4.0 using curl library fails.
...
Each time while fetching a file a new connection is established. So when
more than 500 connection is established and fetching of files stops. The
established connections are not getting closed and they remain idle for 30
mins.
Why is there new ones created and if new ones are created, how come the old
ones are still kept around? That's not something libcurl makes happen.
libcurl
tries very hard to re-use connections as much as possible.
HTTP Client request Sent:
Host: dev-vcenter-4-2
User-Agent: gSOAP/2.7
Content-Type: text/xml; charset=utf-8
Content-Length: 504
Connection: close
Cookie:
vmware_soap_session=C762CF10-B1DF-427A-9E0C-9EB851FFDE3D;$Domain=dev-vcenter-4-2
Cookie: ;$Path=/sdk;$Domain=dev-vcenter-4-2
SOAPAction: urn:vim25/4.1
