curl_multi_socket + libev: monitoring data?
In the example evhiperfifo.c, a named pipe is created and monitored. When new URLs are sent to the pipe, line by line, then for every URL a cURL easy handle is created. Now, in want to change this process a little bit: * I want to have URLs read from a database. So, naturally, there would be no monitoring of a named pipe. * I want to only have a certain number of easy handles. This is in order not to exceed the system's limits, and in order not to make individual transfers too slow. My idea: Every time after a call to curl_multi_socket_action(), a MySQL database is checked for new URLs. Then, for each URL, a new easy handle is created, until a certain maximum number of handles has been reached. Background info: There is a separate process that times out URLs in the database, in order to prevent the queue from flowing over. Questions: * Is there any problem if program flow is paused right after calling curl_multi_socket_action? * During connection to database (possibly slow), downloads will continue in the background under UNIX, right? --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: after 7.26.0
Hi Steve, 2012/6/10 Steve Holme steve_ho...@hotmail.com Hi Marc, * Cloned the repository from https://github.com/mback2k/curl.git * Switched to the schannel-rebase-00fddba672 branch That's a way to do it. But if you want to push to your own repo instead of needing to generate e-mail patches, you could have created a fork on github.com. Anyway, you can just send your commits as a patch to the mailinglist. Oooo, Forking sounds a bit scary - I think I'll stick with the email patches ;-) Oh, it's absolutely not. Did you take a look at the guide on github.com? https://help.github.com/articles/fork-a-repo Will I have direct commit / push access to that branch or should I be working differently? Yes, you will have commit access to your local clone, but you won't be able to push to my personal fork on github.com. No problem - I wasn't sure whether you wanted the changes as patches or whether you wanted to give me access to your fork. I cannot give you direct write access to my fork. For that I would have to add your private SSK key to my github.com account. First of all it is not possible to add the same key to different accounts, so you would have to generate a separate key. Second of all you would have access to all my personal repositories. I am sorry, but that's not an option. ;-) A better approach would to fork Daniel's repo and then pull my changes into it. Now that's starting to sound really scary ;-) I've just added support for APOP in a local branch here, which I hope to push over the weekend and once that is out of the way / whilst I wait for the autobuilds to run I will hopefully be able to re do my version info changes into your branch. Cool! Just to give you an update... * I pushed all my changes to the main curl repo in order to add support for APOP * I have updated your repo and branch schannel-rebase-00fddba672 with my version changes here * I have tested the following configurations to make sure --version displays the correct information: 1) No SSPI 2) No SSPI and OpenSSL 3) SSPI but no SSL 4) SSPI with OpenSSL 5) SSPI with SChannel SSL * The results were: 1) curl 7.27.0-DEV (x86_64-pc-win32) libcurl/7.27.0-DEV Features: AsynchDNS Largefile 2) curl 7.27.0-DEV (x86_64-pc-win32) libcurl/7.27.0-DEV OpenSSL/1.0.0e Features: AsynchDNS Largefile NTLM SSL 3) curl 7.27.0-DEV (x86_64-pc-win32) libcurl/7.27.0-DEV sspi/6.1.7601 Features: AsynchDNS GSS-Negotiate Largefile NTLM 4) curl 7.27.0-DEV (x86_64-pc-win32) libcurl/7.27.0-DEV OpenSSL/1.0.0e sspi/6.1.7601 Features: AsynchDNS GSS-Negotiate Largefile NTLM SSL 5) curl 7.27.0-DEV (x86_64-pc-win32) libcurl/7.27.0-DEV sspi/6.1.7601 Features: AsynchDNS GSS-Negotiate Largefile NTLM SSL That looks great! Thanks for your work on this. I now need to: * Separate the work into two patches and email them to you Just a quick question before I do that though... Should we have the version string as sspi as I have done or as WinSSPI ? I would be like to see it show up as either SSPI or WinSSPI, as it's an uppercase abbreviation. Just my 2 cent. I appreciate this maybe still to do... but a little bit of feedback as I found having to add USE_WINDOWS_SSPI; USE_SSL; USE_SCHANNEL to the pre-processor section of my Visual Studio project a little confusing to enable SSPI and SChannel based SSL. I thought that by having USE_SCHANNEL that would automatically define USE_SSL. So in the end you mean that USE_SSL should automatically be defined if USE_SCHANNEL is defined? Best regards, Marc --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: after 7.26.0
Hi Marc, Oooo, Forking sounds a bit scary - I think I'll stick with the email patches ;-) Oh, it's absolutely not. Did you take a look at the guide on github.com? https://help.github.com/articles/fork-a-repo Not yet - but I will do when I get 5 minutes ;-) No problem - I wasn't sure whether you wanted the changes as patches or whether you wanted to give me access to your fork. Second of all you would have access to all my personal repositories. I am sorry, but that's not an option. ;-) No problem - I wasn't sure how GitHub manages access to forks. Just a quick question before I do that though... Should we have the version string as sspi as I have done or as WinSSPI ? I would be like to see it show up as either SSPI or WinSSPI, as it's an uppercase abbreviation. Just my 2 cent. I had some additional thoughts about this overnight and I've gone with WinSSPI. I thought that by having USE_SCHANNEL that would automatically define USE_SSL. So in the end you mean that USE_SSL should automatically be defined if USE_SCHANNEL is defined? Yep - When I perform an OpenSSL based curl build I don't have to define USE_SSL as well. Looking at setup.h USE_SSL gets defined at line 583 if USE_SSLEAY or any of the other SSL libraries are defined. As such I think || defined(USE_SCHANNEL) should be added to that list ;-) Kind Regards Steve --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: after 7.26.0
Hi Steve, 2012/6/10 Steve Holme steve_ho...@hotmail.com: Just a quick question before I do that though... Should we have the version string as sspi as I have done or as WinSSPI ? I would be like to see it show up as either SSPI or WinSSPI, as it's an uppercase abbreviation. Just my 2 cent. I had some additional thoughts about this overnight and I've gone with WinSSPI. Yep, I think that's a good decision. I thought that by having USE_SCHANNEL that would automatically define USE_SSL. So in the end you mean that USE_SSL should automatically be defined if USE_SCHANNEL is defined? Yep - When I perform an OpenSSL based curl build I don't have to define USE_SSL as well. Looking at setup.h USE_SSL gets defined at line 583 if USE_SSLEAY or any of the other SSL libraries are defined. As such I think || defined(USE_SCHANNEL) should be added to that list ;-) Ok, this sounds reasonable. Can you put the minimal change into your patches and give it a try, please? Thanks! Best regards, Marc --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: after 7.26.0
Hi again, I thought that by having USE_SCHANNEL that would automatically define USE_SSL. So in the end you mean that USE_SSL should automatically be defined if USE_SCHANNEL is defined? Yep - When I perform an OpenSSL based curl build I don't have to define USE_SSL as well. Looking at setup.h USE_SSL gets defined at line 583 if USE_SSLEAY or any of the other SSL libraries are defined. As such I think || defined(USE_SCHANNEL) should be added to that list ;-) Ok, this sounds reasonable. Can you put the minimal change into your patches and give it a try, please? Thanks! No problem. S. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Patch: winbuild fixes and formatting
Hello everyone, in order to update the winbuild makefiles for the SSPI and Schannel changes, I did some cleanup and fixed typos, formatting and a problem with environment variables being not passed from one makefile to another. It would be great if someone could put this patch into the master branch, before I rebase the schannel stuff onto it for final merging. Best regards, Marc 0001-winbuild-Fixed-environment-variables-being-lost.patch Description: Binary data --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: Patch: winbuild fixes and formatting
Hi Marc, On Sun, 10 Jun 2012, Marc Hoersken wrote: in order to update the winbuild makefiles for the SSPI and Schannel changes, I did some cleanup and fixed typos, formatting and a problem with environment variables being not passed from one makefile to another. Thank you. You mentioned it was coming in private email so... It would be great if someone could put this patch into the master branch, before I rebase the schannel stuff onto it for final merging. I've given it a quick review and pushed as commit: 72c7c1d64e1016 Kind Regards Steve --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Patch: winbuild fixes and formatting
2012/6/10 Marc Hoersken i...@marc-hoersken.de: Hi Steve, 2012/6/10 Steve Holme steve_ho...@hotmail.com: Hi Marc, On Sun, 10 Jun 2012, Marc Hoersken wrote: in order to update the winbuild makefiles for the SSPI and Schannel changes, I did some cleanup and fixed typos, formatting and a problem with environment variables being not passed from one makefile to another. Thank you. You mentioned it was coming in private email so... It would be great if someone could put this patch into the master branch, before I rebase the schannel stuff onto it for final merging. I've given it a quick review and pushed as commit: 72c7c1d64e1016 Thank you very much. I am not going to rebase the schannel implementation, including your changes, on top of this commit. Best regards, Marc Sorry, I meant now and not not. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Patch: winbuild fixes and formatting
Hi Steve, 2012/6/10 Steve Holme steve_ho...@hotmail.com: Hi Marc, On Sun, 10 Jun 2012, Marc Hoersken wrote: in order to update the winbuild makefiles for the SSPI and Schannel changes, I did some cleanup and fixed typos, formatting and a problem with environment variables being not passed from one makefile to another. Thank you. You mentioned it was coming in private email so... It would be great if someone could put this patch into the master branch, before I rebase the schannel stuff onto it for final merging. I've given it a quick review and pushed as commit: 72c7c1d64e1016 Thank you very much. I am not going to rebase the schannel implementation, including your changes, on top of this commit. Best regards, Marc --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: Patch: winbuild fixes and formatting
Hiya, Thank you very much. I am not going to rebase the schannel implementation, including your changes, on top of this commit. Sorry, I meant now and not not. Hehehe - I did wonder ;-) S. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: curl_multi_socket + libev: monitoring data?
On Sun, 10 Jun 2012, Felix E. Klee wrote: * Is there any problem if program flow is paused right after calling curl_multi_socket_action? Not that I can think of, no. * During connection to database (possibly slow), downloads will continue in the background under UNIX, right? No, not unless you make them do so. libcurl's multi API is doing all the concurrent transfers simultaneously in the *same* thread. If you do something blocking you will block all the other transfers done in that thread. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Hello again, sorry, but I have to submit two more patches for the mingw32 build: - sspi: Fixed incompatible parameter pointer type in Curl_sspi_version - mingw32: Fixed warning of USE_SSL being redefined And just for the record, the following things are left to be done for the Schannel implementation: - implement write buffering - implement SSL/TLS shutdown - implement client certificate authentication - implement custom server certificate validation - implement cipher/algorithm option Best regards, Marc 2012/6/10 Marc Hoersken i...@marc-hoersken.de: Hello everyone, with the support of Steve and Guenter, I was able to ready my Windows SSPI Schannel implementation. Attached you will find a zip-file containing separate patch files for each commit. The other file attached is a whole patch file containing the commit history. I squashed the commit history and removed obsolete commits in order to have a clean history. I also updated the RELEASE-NOTES and FEATURES docs as well as adding myself to the THANKS list: https://github.com/mback2k/curl/commit/e439268e3aee707a816b313d7fe7f06be364505f The corresponding branch can be found here: https://github.com/mback2k/curl/tree/schannel-ready A comparision between the current master and this branch can be found here: https://github.com/mback2k/curl/compare/schannel-ready The patche(s) should easily apply to the current master 72c7c1d64e: https://github.com/bagder/curl/commit/72c7c1d64e101675520387c0d758b63f71d4c48a It would be great if everyone could help do a little final build testing before this is merged. We need tests using mingw32 cross-compilation, native-compilation and the VC winbuild scripts. I would appreciate any help on this matter, thanks in advance! Best regards, Marc 0027-sspi-Fixed-incompatible-parameter-pointer-type-in-Cu.patch Description: Binary data 0028-mingw32-Fixed-warning-of-USE_SSL-being-redefined.patch Description: Binary data --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Patch: curl_ntlm_msgs.c: Fixed passwdlen not being used and recalculated
Hello everyone, I just spotted this small mistake while doing mingw32-make builds. gcc -I. -I../include -g -O2 -Wall -fno-strict-aliasing -DBUILDING_LIBCURL -DUSE_WINDOWS_SSPI -DUSE_SCHANNEL -c curl_ntlm_msgs.c curl_ntlm_msgs.c: In function 'Curl_ntlm_create_type1_message': curl_ntlm_msgs.c:402:10: warning: variable 'passwdlen' set but not used [-Wunused-but-set-variable] The variable is calculated like this: if(passwdp) passwdlen = strlen(passwdp); But then the length value was recalculated instead of the variable being used: ntlm-identity.UserLength = (unsigned long)userlen; if((ntlm-identity.Password = (unsigned char *)strdup(passwdp)) == NULL) return CURLE_OUT_OF_MEMORY; ntlm-identity.PasswordLength = (unsigned long)strlen(passwdp); if((ntlm-identity.Domain = malloc(domlen + 1)) == NULL) return CURLE_OUT_OF_MEMORY; I changed that to the following: ntlm-identity.UserLength = (unsigned long)userlen; if((ntlm-identity.Password = (unsigned char *)strdup(passwdp)) == NULL) return CURLE_OUT_OF_MEMORY; ntlm-identity.PasswordLength = (unsigned long)passwdlen; if((ntlm-identity.Domain = malloc(domlen + 1)) == NULL) return CURLE_OUT_OF_MEMORY; Attached you will find the corresponding patch against the master branch. Best regards, Marc 0001-curl_ntlm_msgs.c-Fixed-passwdlen-not-being-used-and-.patch Description: Binary data --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Hello again, shame on me, but I have to submit another small patch with just a single character being added: - Fixed missing pointer cast in Curl_sspi_version Best regards, Marc 0029-sspi-Fixed-missing-pointer-cast-in-Curl_sspi_version.patch Description: Binary data --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Windows SSPI Schannel implementation ready
Hi Marc, I'll look soonish into all this. An ultra fast patch skimming shows there are line ending problems (changed from UNIX to DOS) in at least all lines of include/curl/curl.h and lib/setup.h -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Why there is undefined symbol __cxa_begin_cleanup in libcurl.so
I recommend that you try such a standard build too: export PATH=/opt/android-ndk-r8-standalone/bin:$PATH ./configure --host=arm-linux-androideabi --enable-debug --with-zlib --enable-ipv6 and then report back if you still see same issues as before ... Gün. Hi Gün I can build curl successfully with these parameters, after replacing config.guess and config.sub in /usr with the files mentioned in your previous mail. libcurl.so still has the __cxa_begin_cleanup while libcurl.a doesn't have it. So i use the latter to link to my executable. What i feel strange is: src/.libs/curl can be linked with libcurl.so, but my executable can not, with the hidden symbol error reported. It maybe caused by some compiling flags in my project. I'll check it later. Thank very much for your help! -sonmerfield --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
