Re: VMS fixes needed to allow building
Tom and John, Issues mentioned here have been addressed with following commits: https://github.com/bagder/curl/commit/17c088fb45 https://github.com/bagder/curl/commit/8ae9ee0cf2 Daily snapshot (http://curl.haxx.se/snapshots/) of March 14 should include these. Thanks, -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: VMS fixes needed to allow building.
On Wed, Mar 13, John E. Malmberg wb8...@qsl.net wrote: I have a git clone in one directory tree now and it has the command procedures needed to build the .h files. I have not tested them. I can not run autoconf on VMS to generate the configure and lib/curl_config_in. [...] I pulled down the March 13th, 2013 snapshot and unpacked it. It is also missing the VMS specific build files in the packages/vms directory that I can see in the git repository. Fixed now with commit: https://github.com/bagder/curl/commit/308dc625f7 March 14 daily snapshot should include them now. Something I've noticed about the build process... build_curlbuild_h.com attempts to generate file curlbuild.h. But the file it generates is not a proper one. It has dependencies on several HAVE_* preprocessor macros which is wrong for curlbuild.h. Given that it is not possible to run autoconf, I suggest you make direct use of curlbuild.h if it exists (daily and release tarballs), and simply copy curlbuild.h.dist to curlbuild.h when it doesn't exist (git pull). This will also ensure that whatever changes are done to curlbuild.h.dist are automatically taken in account for VMS builds, without someone having to figure out that VMS builds get broken. -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Avoid write callbacks
Is there a way to avoid callbacks, while still supporting incremental retrieval of data? In a nutshell, I'm looking for an iterator-like interface. I could use the multi interface with a custom write function which writes to an in-memory buffer. I would run the event loop until that buffer contains some data. My iterator would return data from the buffer as long as some is available. It would run some iterations of the event loop to get more data. But I wonder if there is an easier way to accomplish this. --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
RE: CURLE_OPERATION_TIMEDOUT using curl_multi_socket_action
I will work on writing a sample code and will mail it by tomorrow.
Find the sample program attached which illustrates the problem I am facing.
I am using Mozilla's NSPR: http://www.mozilla.org/projects/nspr/
#include sys/epoll.h
#include curl/curl.h
#include prtpool.h
#include prlock.h
#include prcvar.h
#define EASY_COUNT 100
PRThreadPool *gPool;
CURLM *gMulti;
int gEpollFd;
PRLock *gIoQueueLock;
PRCondVar *gIoQueueWait;
PRLock *gMultiLock;
int gIoQueueCount;
struct ConnectionInfo {
CURL *easy;
curl_socket_t socket;
int action;
PRJob *job;
} gEasy[EASY_COUNT];
size_t curl_writer(void *ptr, size_t size, size_t nmemb, void *param) {
return size*nmemb;
}
void setup_easy(CURL *easy) {
curl_easy_setopt(easy, CURLOPT_TIMEOUT, 300L);
curl_easy_setopt(easy, CURLOPT_CONNECTTIMEOUT, 30L);
curl_easy_setopt(easy, CURLOPT_FOLLOWLOCATION, 0L);
curl_easy_setopt(easy, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(easy, CURLOPT_WRITEFUNCTION, curl_writer); //Ignore output
from server
curl_easy_setopt(easy, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(easy, CURLOPT_URL, https://www.google.com/ncr;);
curl_easy_setopt(easy, CURLOPT_SSL_SESSIONID_CACHE, 0L);
}
void curl_calls(void *arg) {
ConnectionInfo *info = (ConnectionInfo*) arg;
int still_running;
PR_Lock(gMultiLock);
if (info-socket == CURL_SOCKET_BAD) { //Job is running for the first time.
//Do some operation which takes a couple of seconds.
PR_Sleep(PR_SecondsToInterval(1));
curl_multi_add_handle(gMulti, info-easy);
}
curl_multi_socket_action(gMulti, info-socket, info-action, still_running);
PR_Unlock(gMultiLock);
if (info-action CURL_POLL_INOUT) {
epoll_event event;
event.events = EPOLLRDHUP | EPOLLONESHOT
| (info-action CURL_POLL_IN ? EPOLLIN : 0)
| (info-action CURL_POLL_OUT ? EPOLLOUT : 0);
event.data.ptr = info;
epoll_ctl(gEpollFd, EPOLL_CTL_ADD, info-socket, event);
PR_Lock(gIoQueueLock);
gIoQueueCount++;
info-job = PR_QueueJob_Timer(gPool, PR_SecondsToInterval(300), curl_calls,
info, PR_FALSE);
PR_NotifyCondVar(gIoQueueWait);
PR_Unlock(gIoQueueLock);
}
}
int socket_callback(CURL *easy, curl_socket_t socket, int action, void
*multi_data, void *socket_data) {
ConnectionInfo *info;
curl_easy_getinfo(easy, CURLINFO_PRIVATE, info);
info-action = action;
info-socket = socket;
return 0;
}
void epoll_calls(void *arg) {
PR_Lock(gIoQueueLock);
while(true) {
PR_WaitCondVar(gIoQueueWait, PR_INTERVAL_NO_TIMEOUT);
epoll_event events[10];
int n;
do {
PR_Unlock(gIoQueueLock);
if ((n = epoll_wait(gEpollFd, events, 10, 3000)) 0) {
PR_Lock(gIoQueueLock);
for (int i = 0; i n; ++i) {
ConnectionInfo *info = (ConnectionInfo*) events[i].data.ptr;
PR_CancelJob(info-job);
epoll_ctl(gEpollFd, EPOLL_CTL_DEL, info-socket, NULL);
info-job = PR_QueueJob(gPool, curl_calls, info, PR_FALSE);
}
gIoQueueCount -= n;
PR_Unlock(gIoQueueLock);
}
PR_Lock(gIoQueueLock);
} while (gIoQueueCount 0);
}
}
int main() {
curl_global_init(CURL_GLOBAL_ALL);
gEpollFd = epoll_create(1);
gIoQueueLock = PR_NewLock();
gIoQueueWait = PR_NewCondVar(gIoQueueLock);
gMultiLock = PR_NewLock();
gPool = PR_CreateThreadPool(5, 5, 1024*1024);
gMulti = curl_multi_init();
curl_multi_setopt(gMulti, CURLMOPT_SOCKETFUNCTION, socket_callback);
PR_QueueJob(gPool, epoll_calls, NULL, PR_FALSE);
for (int i = 0; i EASY_COUNT; ++i) {
gEasy[i].socket = CURL_SOCKET_BAD;
gEasy[i].easy = curl_easy_init();
curl_easy_setopt(gEasy[i].easy, CURLOPT_PRIVATE, gEasy + i);
setup_easy(gEasy[i].easy);
gEasy[i].job = PR_QueueJob(gPool, curl_calls, gEasy + i, PR_FALSE);
}
PR_JoinThreadPool(gPool);
return 0;
}---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: VMS fixes needed to allow building
Date: Tue, 12 Mar 2013 22:02:45 -0500 From: John E. Malmberg wb8...@qsl.net Subject: Re: VMS fixes needed to allow building Message-ID: khoq6v$q6d$1...@ger.gmane.org On 3/12/2013 7:17 PM, Tom Grace wrote: Using extern int vms_show = 0 also creates a globaldef. globaldef is only for use with code intended to be compiled with the very obsolete VAXC compiler. It should not be used for anything new or current. Yes your exactly right, I should of said that the use of extern int vms_show = 0 ; defines a global symbol internally just like the old-form globaldef . (in addition to the psect ref) So that the linker will be happy. So does the form, but it is a weak ? globaldef. int vms_show; /* outside a function*/ so is the recommendation? one extern int vms_show = 0; and multiple extern int vms_show This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at http://www.amdocs.com/email_disclaimer.asp --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Strange performance discrepancy with proxy
We have recently integrated libcurl into our application to provide the underlying HTTP transport (replacing WinINET, which is a huge improvement, thanks!). The application is using the multi interface, though for the most part on a single request is made at a given time. I'm seeing strange behavior where requests are _much_ faster when routed through a proxy (in this case the Fiddler diagnostic tool). Note that this is not a caching proxy, and all the requests are POSTs anyway, so I don't think that is the explanation. If I run the app with an HTTP_PROXY environment variable defined it is much faster than without - the code is the same in both cases. I've instrumented the code as well to display the progression of requests, and it appears to be the same in both cases except the proxied requests return much faster. I'd love to have the same performance regardless of whether or not a proxy is used, so is there something I'm potentially missing that might speed up the non- proxied case as well? thanks matt --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: Avoid write callbacks
On Wed, Mar 13, 2013 at 01:01:35PM +0100, Florian Weimer wrote: Is there a way to avoid callbacks, while still supporting incremental retrieval of data? Unless you're writing to a FILE*, libcurl gives no other option for writing received data without using callbacks. But, you can certainly wrap libcurl in such a way that the use of callbacks is hidden from the application. In a nutshell, I'm looking for an iterator-like interface. I could use the multi interface with a custom write function which writes to an in-memory buffer. I would run the event loop until that buffer contains some data. My iterator would return data from the buffer as long as some is available. It would run some iterations of the event loop to get more data. That solution sounds perfectly doable. Take a look at the fopen.c example program for an example of a libcurl wrapper that provides a stdio-like interface. But I wonder if there is an easier way to accomplish this. It sounds like the API you want doesn't quite match the API libcurl provides, in which case some amount of code is going to be necessary to handle the mismatch. It doesn't have to be a lot. Take a look also at the existing C++ binding (or bindings?) to see if that might not better suit you. Dan --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: [PATCH] Addition of trailer headers in HTTP requests generated by libcurl
On Mon, 11 Mar 2013, Chrysovaladis Datsios wrote: changes are in the attached patch. Thanks. I would say that this is close to getting merged. My main issue is remark #4 which kind of made #5 slip in... 1 - invalid email in the Author: field 2 - incomplete commit message 3 - transfer.c:245:11: error: conversion to 'int' from 'size_t' may alter its value [-Werror=conversion] 4 - lack of test cases. It makes it really hard for us to say that this works even in corner cases. 5 - this version still has memory leaks, like if the realloc (on transfer.c:228) fails the previous trailer_headers_buf is not freed. 6 - getting the two new options documented The window for new features is just closed for 7.30.0, but it is looking good for getting merged when we open it again after that release. -- / daniel.haxx.se --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: [PATCH] Addition of trailer headers in HTTP requests generated by libcurl
On Mon, Mar 11, 2013 at 06:09:29PM +0200, Chrysovaladis Datsios wrote:
--- a/lib/transfer.c
+++ b/lib/transfer.c
@@ -193,6 +193,71 @@ CURLcode Curl_fillreadbuffer(struct connectdata *conn,
int bytes, int *nreadp)
/* copy the prefix to the buffer, leaving out the NUL */
memcpy(data-req.upload_fromhere, hexbuffer, hexlen);
+/* If is the last chunk, here put the trailer headers if any */
+if((nread - hexlen) == 0) {
+ if(data-set.trailerheaders_func) {
+struct curl_slist *trailer_headers;
+struct curl_slist *list_head;
+
+/* The callback function that adds trailer header values */
+trailer_headers = (data-set.trailerheaders_func)(data,
+ data-set.trailer_udata);
+list_head = trailer_headers;
+if(trailer_headers) {
+ char *ptr;
While we're here, this could be made const char *ptr (and it could be moved
into the body of the while loop).
+ size_t headers_buf_size;
+ size_t endl_len;
+ size_t data_len;
+ size_t headers_index;
+ char *trailer_headers_buf;
+
+ endl_len = strlen(endofline_native);
+ headers_buf_size = HEADERSIZE;
+ trailer_headers_buf = malloc(headers_buf_size);
+ if(trailer_headers_buf == NULL)
+return CURLE_BAD_FUNCTION_ARGUMENT;
+ memcpy(trailer_headers_buf, hexbuffer, hexlen);
+ headers_index = hexlen;
+
+ while(trailer_headers) {
+/* header name_field : header value_field */
+ptr = strchr(trailer_headers-data, ':');
+if(ptr) {
+ data_len = strlen(trailer_headers-data);
+ if(headers_index + data_len + 2*endl_len headers_buf_size) {
+size_t newsize = headers_index + data_len + 2*endl_len;
+if(newsize CURL_MAX_HTTP_HEADER) {
+ curl_slist_free_all(list_head);
+ list_head = NULL;
+ return CURLE_BAD_FUNCTION_ARGUMENT;
trailer_headers_buf will leak here.
+}
+trailer_headers_buf = realloc(trailer_headers_buf, newsize);
+if(trailer_headers_buf == NULL) {
+ curl_slist_free_all(list_head);
+ list_head = NULL;
I wouldn't bother setting a local variable to NULL immediately before
returning. Same just above.
+ return CURLE_BAD_FUNCTION_ARGUMENT;
+}
+headers_buf_size = newsize;
+ }
+ memcpy(trailer_headers_buf + headers_index,
+ trailer_headers-data,
data_len);
+ headers_index += data_len;
+ memcpy(trailer_headers_buf + headers_index,
+ endofline_native,
endl_len);
+ headers_index += endl_len;
+}
+trailer_headers = trailer_headers-next;
+ }
+ nread = headers_index;
+ data-req.upload_fromhere = trailer_headers_buf;
I still don't see where trailer_headers_buf is ever freed.
+ curl_slist_free_all(list_head);
+ list_head = NULL;
I wouldn't bother with this NULL setting either, since list_head goes
out of scope here anyway.
+}
+ }
+ /* mark this as done once this chunk is transferred */
+ data-req.upload_done = TRUE;
Moving this here changes the behaviour when the CURL_DOES_CONVERSIONS
exits due to an error. This is probably fine, but have you verified that there
aren't any unexpected effects through the error path?
+}
+
/* always append ASCII CRLF to the data */
memcpy(data-req.upload_fromhere + nread,
endofline_network,
@@ -215,10 +280,6 @@ CURLcode Curl_fillreadbuffer(struct connectdata *conn,
int bytes, int *nreadp)
return(res);
#endif /* CURL_DOES_CONVERSIONS */
-if((nread - hexlen) == 0)
- /* mark this as done once this chunk is transferred */
- data-req.upload_done = TRUE;
-
nread+=(int)strlen(endofline_native); /* for the added end of line */
}
#ifdef CURL_DOES_CONVERSIONS
Dan
---
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
Re: VMS fixes needed to allow building
On Wed, Mar 13, 2013 at 4:13 PM, Tor Arntsen kspt@gmail.com wrote: [...] Usually the latter is achieved by having extern int vms_show; in a header file and including it in the necessary .c files (and in exactly one of them you put the 'int vms_show=0;') Removed redundant 'extern' now with https://github.com/bagder/curl/commit/911b2d3f67 Thanks, -- -=[Yang]=- --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
vms_show hack explained
The vms_show hack can definitely be improved and simplified. I just have not had the time to investigate it. What is going on is that on VMS, the parameter to exit() is a packed bitmap that encodes a pointer to a message text and also a severity code. On exit, if that severity code is for warning or more severe level, the message associated with the packed bitmap is displayed by the DCL shell. This message can be suppressed by setting a bit in the bitmap, which is done if the application has already displayed it. Now what was done on VMS, is that the vms_show variable is being set to indicate that an error message has been displayed. Then on vms_special_exit() the Unix exit status is mapped to the VMS packed exit code, and the vms_show is used to suppress the message. But there is one more wrinkle. This VMS status code is useful for the VMS shell, but is the wrong code for programs like bash, that may have launched curl. So when curl is run under Bash, it needs to encode the UNIX exit code so that can be then decoded by Bash. Normally this is done at compile time by mapping exit() to actually be posix_exit() and the UNIX program does not need VMS specific code. But curl was ported to use its own VMS specific exit codes first, and those can not be passed to the posix_exit() routine. So the vms_special_exit() needs to check to see which variant of exit that it should call. Now if we know at the time that vms_special_exit is called that any exit status other than CURLE_OK means that an error message has been written, there is no need to have the vms_show hack at all. We just need to have the VMS specific wrapper to exit() handle everything. If that a VMS specific wrapper is changed to be a macro in a VMS header file, then this all can be hidden from the common Unix code. One feature of the separate message file on VMS is that it can be built as a shared image, and you can have a different shared image for different languages, but one program executable. The VMS port of curl does not take advantage of this, as the messages in the VMS message file are never displayed. Regards, -John wb8...@qsl.net Personal Opinion Only --- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
