Hi Curlers,
for a few days i've been trying to tackle this issue and Bagder asked me on
IRC to put it up on the ML.
On my server, i'm running a cgi-bin (written in plain c, see minimal testcase
attachment ret_400.c ) which is supposed to handle incoming HTTP PUT uploads.
Under certain conditions (e.g. if argument validation fails) it may set an
HTTP status code 400 and exit. when i use the command line curl utility and
PUT a file to this cgi, i receive the appropriate error code (case 2 in
testcases.log)
In my peer application which is supposed to invoke the put operation, i could
never receive the server's error response, instead it kept calling the
READFUNC and upload forever, even though the cgi-bin had already terminated.
i hacked a simple test case (curlputtest.c), which shows that the error status
is very well received if i upload a file with a known size (test-case 4). if
the size is unknown (i generate dummy buffers to simulate our live source use
case), then the problem occurs (test-case 7). when the specified url is
faulty, then the server 404 response is received correctly no matter what
(test-cases 5+8), only when cgi-bin itself generates the status, then i don't
get it.
apparently though, i am doing something wrong in my code, because the cli curl
utility receives the 400 status just fine.
another issue that i'm having:
sp.curl->res = curl_easy_pause(sp.curl->easy, CURLPAUSE_SEND) seems to be
ineffctive too, since the READFUNC is started anyway, neither does it pause if
i return CURL_READFUNC_PAUSE there.
i'd be grateful for any suggestions :)
fraxinas
//////////////////////////////////////////////////////////////////////////////
// 1) uploading a file without unknown size and with correct arguments
$ curl -XPUT -T - "http://fox/cgi-bin/ret_400.cgi?1234567890"; -vvv < ret_400.c
* About to connect() to fox port 80 (#0)
* Trying 127.0.0.1...
* Connected to fox (127.0.0.1) port 80 (#0)
> PUT /cgi-bin/ret_400.cgi?1234567890 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: fox
> Accept: */*
> Transfer-Encoding: chunked
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 201 Created
< Date: Wed, 13 Mar 2013 22:01:14 GMT
< Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.1e DAV/2
< Connection: Close
< Content-Length: 22
< Content-Type: text/html
<
Created the resource
* Closing connection 0
//////////////////////////////////////////////////////////////////////////////
// 2) uploading a file without unknown size and with FALSE arguments
$ curl -XPUT -T - "http://fox/cgi-bin/ret_400.cgi?false_arguments"; -vvv < ret_400.c
* About to connect() to fox port 80 (#0)
* Trying 127.0.0.1...
* Connected to fox (127.0.0.1) port 80 (#0)
> PUT /cgi-bin/ret_400.cgi?false_arguments HTTP/1.1
> User-Agent: curl/7.29.0
> Host: fox
> Accept: */*
> Transfer-Encoding: chunked
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 400 Bad Request
< Date: Wed, 13 Mar 2013 22:01:51 GMT
< Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.1e DAV/2
< Connection: Close
< Content-Length: 40
< Content-Type: text/html
<
Invalid argument data: false_arguments
* Closing connection 0
//////////////////////////////////////////////////////////////////////////////
// 3) uploading a file with determined size and with correct arguments
$ ./curlputtest
GET /home/fraxinas/corelpolyguiconf.txt HTTP/1.0
X-DESTINATION: http://fox/cgi-bin/ret_400.cgi?1234567890
curl put mode on!! source path '/home/fraxinas/corelpolyguiconf.txt' destination url 'http://fox/cgi-bin/ret_400.cgi?1234567890'
HTTP/1.0 200 OK
Connection: Close
Content-Length: 144
Content-Type: video/mpeg
PAUSED OK* About to connect() to fox port 80 (#0)
* Trying 127.0.0.1...
* Connected to fox (127.0.0.1) port 80 (#0)
> PUT /cgi-bin/ret_400.cgi?1234567890 HTTP/1.1
Host: fox
Accept: */*
Content-Length: 144
Expect: 100-continue
< HTTP/1.1 100 Continue
curl read_callback nread=144
* We are completely uploaded and fine
< HTTP/1.1 201 Created
< Date: Mon, 18 Mar 2013 10:49:35 GMT
< Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.1e DAV/2
< Connection: Close
< Transfer-Encoding: chunked
< Content-Type: text/html
<
QOS: peer finished <body>Created the resource
</body>
* Closing connection 0
//////////////////////////////////////////////////////////////////////////////
// 4) uploading a file with determined size and with FALSE arguments
$ ./curlputtest
GET /home/fraxinas/corelpolyguiconf.txt HTTP/1.0
X-DESTINATION: http://fox/cgi-bin/ret_400.cgi
curl put mode on!! source path '/home/fraxinas/corelpolyguiconf.txt' destination url 'http://fox/cgi-bin/ret_400.cgi'
HTTP/1.0 200 OK
Connection: Close
Content-Length: 144
Content-Type: video/mpeg
PAUSED OK* About to connect() to fox port 80 (#0)
* Trying 127.0.0.1...
* Connected to fox (127.0.0.1) port 80 (#0)
> PUT /cgi-bin/ret_400.cgi HTTP/1.1
Host: fox
Accept: */*
Content-Length: 144
Expect: 100-continue
< HTTP/1.1 100 Continue
curl read_callback nread=144
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Date: Mon, 18 Mar 2013 10:49:44 GMT
< Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.1e DAV/2
< Connection: Close
< Content-Length: 25
< Content-Type: text/html
<
QOS: peer error 400, <body>Invalid argument data:
</body>* Closing connection 0
//////////////////////////////////////////////////////////////////////////////
// 5) uploading a file with determined size and with incorrect URL
$ ./curlputtest
GET /home/fraxinas/corelpolyguiconf.txt HTTP/1.0
X-DESTINATION: http://fox/cgi-bin/alkvjasdv
curl put mode on!! source path '/home/fraxinas/corelpolyguiconf.txt' destination url 'http://fox/cgi-bin/alkvjasdv'
HTTP/1.0 200 OK
Connection: Close
Content-Length: 144
Content-Type: video/mpeg
PAUSED OK* About to connect() to fox port 80 (#0)
* Trying 127.0.0.1...
* Connected to fox (127.0.0.1) port 80 (#0)
> PUT /cgi-bin/alkvjasdv HTTP/1.1
Host: fox
Accept: */*
Content-Length: 144
Expect: 100-continue
< HTTP/1.1 404 Not Found
< Date: Mon, 18 Mar 2013 10:49:53 GMT
< Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.1e DAV/2
< Vary: accept-language,accept-charset
< Accept-Ranges: bytes
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< Content-Language: en
<
QOS: peer error 404, <body><?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";>
<html xmlns="http://www.w3.org/1999/xhtml"; lang="en" xml:lang="en">
<head>
<title>Object not found!</title>
<link rev="made" href="mailto:[email protected]" />
<style type="text/css"><!--/*--><![CDATA[/*><!--*/
body { color: #000000; background-color: #FFFFFF; }
a:link { color: #0000CC; }
p, address {margin-left: 3em;}
span {font-size: smaller;}
/*]]>*/--></style>
</head>
<body>
<h1>Object not found!</h1>
<p>
The requested URL was not found on this server.
If you entered the URL manually please check your
spelling and try again.
</p>
<p>
If you think this is a server error, please contact
the <a href="mailto:[email protected]">webmaster</a>.
</p>
<h2>Error 404</h2>
<address>
<a href="/">fox</a><br />
<span>Mon Mar 18 11:49:53 2013<br />
Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.1e DAV/2</span>
</address>
</body>
</html>
</body>* Closing connection 0
//////////////////////////////////////////////////////////////////////////////
// 6) uploading a stream without predetermined size and with correct arguments
$ ./curlputtest
GET /dummy HTTP/1.0
X-DESTINATION: http://fox/cgi-bin/ret_400.cgi?1234567890
curl put mode on!! source path '/dummy' destination url 'http://fox/cgi-bin/ret_400.cgi?1234567890'
PAUSED OK* About to connect() to fox port 80 (#0)
* Trying 127.0.0.1...
* Connected to fox (127.0.0.1) port 80 (#0)
> PUT /cgi-bin/ret_400.cgi?1234567890 HTTP/1.1
Host: fox
Accept: */*
Transfer-Encoding: chunked
Expect: 100-continue
< HTTP/1.1 100 Continue
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
^C
//////////////////////////////////////////////////////////////////////////////
// 7) uploading a stream without predetermined size and with FALSE arguments
// this is the error case! it's supposed to receive the status 400 from the
// cgi-bin, but it doesn't and just keeps uploading
$ ./curlputtest
GET /dummy HTTP/1.0
X-DESTINATION: http://fox/cgi-bin/ret_400.cgi
curl put mode on!! source path '/dummy' destination url 'http://fox/cgi-bin/ret_400.cgi'
PAUSED OK* About to connect() to fox port 80 (#0)
* Trying 127.0.0.1...
* Connected to fox (127.0.0.1) port 80 (#0)
> PUT /cgi-bin/ret_400.cgi HTTP/1.1
Host: fox
Accept: */*
Transfer-Encoding: chunked
Expect: 100-continue
< HTTP/1.1 100 Continue
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
curl read_callback nread=100
^C
//////////////////////////////////////////////////////////////////////////////
// 8) uploading a stream without predetermined size and with incorrect URL
$ ./curlputtest
GET /dummy HTTP/1.0
X-DESTINATION: http://fox/cgi-bin/alkvjasdv
curl put mode on!! source path '/dummy' destination url 'http://fox/cgi-bin/alkvjasdv'
PAUSED OK* About to connect() to fox port 80 (#0)
* Trying 127.0.0.1...
* Connected to fox (127.0.0.1) port 80 (#0)
> PUT /cgi-bin/alkvjasdv HTTP/1.1
Host: fox
Accept: */*
Transfer-Encoding: chunked
Expect: 100-continue
< HTTP/1.1 404 Not Found
< Date: Mon, 18 Mar 2013 10:51:06 GMT
< Server: Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.1e DAV/2
< Vary: accept-language,accept-charset
< Accept-Ranges: bytes
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< Content-Language: en
<
QOS: peer error 404, <body><?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";>
<html xmlns="http://www.w3.org/1999/xhtml"; lang="en" xml:lang="en">
<head>
<title>Object not found!</title>
<link rev="made" href="mailto:[email protected]" />
<style type="text/css"><!--/*--><![CDATA[/*><!--*/
body { color: #000000; background-color: #FFFFFF; }
a:link { color: #0000CC; }
p, address {margin-left: 3em;}
span {font-size: smaller;}
/*]]>*/--></style>
</head>
<body>
<h1>Object not found!</h1>
<p>
The requested URL was not found on this server.
If you entered the URL manually please check your
spelling and try again.
</p>
<p>
If you think this is a server error, please contact
the <a href="mailto:[email protected]">webmaster</a>.
</p>
<h2>Error 404</h2>
<address>
<a href="/">fox</a><br />
<span>Mon Mar 18 11:51:06 2013<br />
Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.1e DAV/2</span>
</address>
</body>
</html>
</body>* Closing connection 0
$
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
char* arguments = getenv("QUERY_STRING");
if (!arguments || strlen(arguments) != 10)
{
fprintf(stdout, "Status: %i\r\nConnection: Close\r\nContent-Type: text/html\r\n\r\nInvalid argument data: %s\r\n", 400, arguments);fflush(stdout);
exit(-1);
}
fprintf(stdout,"Status: 201\r\nConnection: Close\r\nContent-Type: text/html\r\n\r\nCreated the resource\r\n");
exit(0);
}
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <netinet/ip.h>
#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
#include <curl/curl.h>
#define MAX_LINE_LENGTH 512
typedef struct _curl_private {
char put_url[MAX_LINE_LENGTH]; /* upload-url for streaming server */
CURL *easy;
CURLM *multi;
struct curl_slist *headers;
CURLcode res;
CURLMcode mres;
} curl_private;
typedef struct _curlputtest {
curl_private *curl;
int source_fd;
int response_code;
} curlputtest;
void cleanup(curlputtest *sp)
{
if (sp->curl)
{
if (sp->curl->headers)
curl_slist_free_all (sp->curl->headers);
if (sp->curl->multi)
curl_multi_cleanup(sp->curl->multi);
if (sp->curl->easy)
curl_easy_cleanup(sp->curl->easy);
curl_global_cleanup();
free(sp->curl);
}
sp->curl = NULL;
}
static size_t read_callback(void *ptr, size_t size, size_t nmemb, void *userp)
{
curlputtest *sp = (curlputtest *)userp;
size_t nread;
if ( sp->source_fd > 0 )
nread = read(sp->source_fd, ptr, size*nmemb);
else
{
nread = 100;
memset (ptr, 'X', 100);
}
if (nread == -1)
return CURL_READFUNC_ABORT;
fprintf(stdout, "curl read_callback nread=%d\n", (int)nread);fflush(stdout);
return nread;
}
static size_t write_callback(char *d, size_t n, size_t l, void *userp)
{
curlputtest *sp = (curlputtest *)userp;
long respcode;
curl_easy_getinfo(sp->curl->easy,CURLINFO_RESPONSE_CODE, &respcode);
if (respcode == 201) {
fprintf(stdout, "QOS: peer finished <body>%s</body>\n", d);fflush(stdout);
}
else {
fprintf(stdout, "QOS: peer error %lu, <body>%s</body>", respcode, d);fflush(stdout);
}
return n*l;
}
int main(int argc, char **argv)
{
char request[MAX_LINE_LENGTH], upstream_request[256];
char *c;
curlputtest sp;
sp.curl = malloc(sizeof(curl_private));
sp.curl->easy = NULL;
sp.curl->multi = NULL;
sp.curl->headers = NULL;
sp.curl->put_url[0] = '\0';
sp.source_fd = -1;
if (!fgets(request, MAX_LINE_LENGTH - 1, stdin))
goto bad_request;
if (strncmp(request, "GET /", 5))
goto bad_request;
c = strchr(request + 5, ' ');
if (!c || strncmp(c, " HTTP/1.", 7))
goto bad_request;
*c++ = 0;
c = request + 4;
while (1)
{
char option[MAX_LINE_LENGTH];
if (!fgets(option, MAX_LINE_LENGTH - 1, stdin))
break;
if (!strncasecmp(option, "X-DESTINATION: ", 15))
{
strncpy(sp.curl->put_url, option+15, strlen(option)-16);
fprintf(stdout, "curl put mode on!! source path '%s' destination url '%s'\n", c, sp.curl->put_url);fflush(stdout);
}
if (option[1] && option[strlen(option)-2] == '\r')
option[strlen(option)-2] = 0;
else
option[strlen(option)-1] = 0;
if (!*option)
break;
}
if (sp.curl->put_url[0] == '\0')
goto bad_request;
curl_global_init(CURL_GLOBAL_ALL);
sp.curl->easy = curl_easy_init();
off_t filesize = 0L;
{
char *p = c;
while ( *p != '\0') {
if (*p == '+')
*p = ' ';
p++;
}
char *path = curl_easy_unescape(sp.curl->easy, c, strlen(c), 0);
{
struct stat stInfo;
sp.response_code = 404;
if ( stat(path, &stInfo) == 0 ) {
filesize = stInfo.st_size;
sp.source_fd = open(path, O_RDONLY);
if ( sp.source_fd > 0 ) {
snprintf(upstream_request, sizeof(upstream_request), "HTTP/1.0 200 OK\r\nConnection: Close\r\nContent-Length: %ld\r\nContent-Type: video/mpeg\r\n\r\n", filesize);
write(1, upstream_request, strlen(upstream_request));
}
}
}
}
{
sp.curl->multi = curl_multi_init();
if (sp.curl->easy && sp.curl->multi) {
curl_easy_setopt(sp.curl->easy, CURLOPT_READFUNCTION, read_callback);
curl_easy_setopt(sp.curl->easy, CURLOPT_READDATA, &sp);
curl_easy_setopt(sp.curl->easy, CURLOPT_WRITEFUNCTION, write_callback);
curl_easy_setopt(sp.curl->easy, CURLOPT_WRITEDATA, &sp);
curl_easy_setopt(sp.curl->easy, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(sp.curl->easy, CURLOPT_PUT, 1L);
curl_easy_setopt(sp.curl->easy, CURLOPT_URL, sp.curl->put_url);
// sp.curl->headers = curl_slist_append(sp.curl->headers, "Transfer-Encoding: chunked");
// curl_easy_setopt(sp.curl->easy, CURLOPT_HTTPHEADER, sp.curl->headers);
// sp.curl->headers = curl_slist_append(sp.curl->headers, "Expect:");
// curl_easy_setopt(sp.curl->easy, CURLOPT_HTTPHEADER, sp.curl->headers);
if (filesize)
curl_easy_setopt(sp.curl->easy, CURLOPT_INFILESIZE_LARGE, (curl_off_t)filesize);
curl_easy_setopt(sp.curl->easy, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(sp.curl->easy, CURLOPT_STDERR, stdout);
sp.curl->mres = curl_multi_add_handle(sp.curl->multi, sp.curl->easy);
if ( sp.curl->mres != CURLM_OK )
goto curl_error;
sp.curl->res = curl_easy_pause(sp.curl->easy, CURLPAUSE_SEND);
if ( sp.curl->res != CURLE_OK )
goto curl_error;
else
fprintf(stdout, "PAUSED OK");
}
else
goto curl_error;
}
while (1)
{
struct timeval timeout;
timeout.tv_sec = 0;
timeout.tv_usec = 100000;
fd_set r;
FD_ZERO(&r);
FD_SET(0, &r);
fd_set w;
FD_ZERO(&w);
int maxfd = 5, curl_maxfd;
if (sp.source_fd != -1)
{
maxfd = sp.source_fd;
FD_SET(sp.source_fd, &r);
}
{
fd_set fdwrite, fdexcep;
curl_multi_fdset(sp.curl->multi, &r, &fdwrite, &fdexcep, &curl_maxfd);
// fprintf(stdout, "curl_multi_fdset curl_maxfd=%d\n", curl_maxfd);fflush(stdout);
if ( curl_maxfd > sp.source_fd )
maxfd = curl_maxfd;
}
if (select(maxfd+1, &r, &w, 0, &timeout) < 0) {
break;
}
if (FD_ISSET(0, &r)) /* check for client disconnect */
if (read(0, request, sizeof(request)) <= 0)
break;
// if (sp.source_fd > 0 && FD_ISSET(sp.source_fd, &r))
{
{
// curl_easy_pause(sp.curl->easy, CURLPAUSE_CONT );
int curl_still_running;
sp.curl->mres = curl_multi_perform(sp.curl->multi, &curl_still_running);
long respcode;
curl_easy_getinfo(sp.curl->easy,CURLINFO_RESPONSE_CODE, &respcode);
// fprintf(stdout, "#### curl_easy_getinfo CURLINFO_RESPONSE_CODE %ld\n", respcode);fflush(stdout);
if ( sp.curl->mres != CURLM_OK )
goto curl_error;
// fprintf(stdout, "curl_still_running=%d\n", curl_still_running);fflush(stdout);
if ( curl_still_running == 0 )
{
CURLMsg *msg;
int msgs_left;
while ((msg = curl_multi_info_read(sp.curl->multi, &msgs_left))) {
// fprintf(stdout, "curl_multi_info_read msg %d data.result %d\n", msg->msg, msg->data.result);fflush(stdout);
if ( msg->msg == CURLMSG_DONE )
{
if ( msg->data.result != CURLE_OK )
{
goto destination_error;
}
}
}
break;
}
}
}
}
cleanup(&sp);
return 0;
bad_request:
printf("HTTP/1.0 400 Bad Request\r\n\r\n");
cleanup(&sp);
return 1;
curl_error:
printf("HTTP/1.0 500 Internal Server Error\r\n\r\n");
printf("curl perform failed: %s %s\n",
curl_easy_strerror(sp.curl->res), curl_multi_strerror(sp.curl->mres));
cleanup(&sp);
return 1;
destination_error:
printf("HTTP/1.0 462 Destination Unreachable\r\ncurl error code %d\r\n", sp.response_code);
cleanup(&sp);
return 1;
}
-------------------------------------------------------------------
List admin: http://cool.haxx.se/list/listinfo/curl-library
Etiquette: http://curl.haxx.se/mail/etiquette.html
