Re: POST with x-www-form-urlencoded Content-Type
Hi Dan, On Wed, Jul 09, 2014 at 07:13:33PM +, Daniel Dubovik wrote: Hello all, I am attempting to balance traffic to a number of backend instances. I am balancing based off the Host header, and for the most part everything is working. When testing a bit more today, I came across some weird behavior, and am hoping someone can help out. When POSTing to a site, if it is done using the Content-Type application/x-www-form-urlencoded, and has actual data, HAProxy falls back to a roundrobin balancing scheme. POSTing using a Content-Type of multipart/form-data, however, works just fine. Oddly, application/x-www-form-urlencoded with no actual data, also works as expected. (...) There's indeed a bug, the amount of data forwarded is not deduced correctly to rewind the buffer, I'm even wondering if it's expected that we let them pass at this point. I'm investigating, thanks for your report! Willy
Re: POST with x-www-form-urlencoded Content-Type
Hi Dan, On Thu, Jul 10, 2014 at 05:20:18PM +0200, Willy Tarreau wrote: Hi Dan, On Wed, Jul 09, 2014 at 07:13:33PM +, Daniel Dubovik wrote: Hello all, I am attempting to balance traffic to a number of backend instances. I am balancing based off the Host header, and for the most part everything is working. When testing a bit more today, I came across some weird behavior, and am hoping someone can help out. When POSTing to a site, if it is done using the Content-Type application/x-www-form-urlencoded, and has actual data, HAProxy falls back to a roundrobin balancing scheme. POSTing using a Content-Type of multipart/form-data, however, works just fine. Oddly, application/x-www-form-urlencoded with no actual data, also works as expected. (...) There's indeed a bug, the amount of data forwarded is not deduced correctly to rewind the buffer, I'm even wondering if it's expected that we let them pass at this point. I'm investigating, thanks for your report! OK I could fix it. The patch is very small but it required some extra care because that's a sensible area that I already fixed in dev23 but not enough. Other balancing algorithms are affected, and worse, http-send-name-header was bogus as well in this case. I've applied the fix, I'm attaching it here, it applies both to 1.5 and to 1.6. Thanks for your report, that was a nasty one and I'm glad we got rid of it early! Willy From bb2e669f9e73531ac9cc9277b40066b701eec918 Mon Sep 17 00:00:00 2001 From: Willy Tarreau w...@1wt.eu Date: Thu, 10 Jul 2014 19:06:10 +0200 Subject: BUG/MAJOR: http: correctly rewind the request body after start of forwarding Daniel Dubovik reported an interesting bug showing that the request body processing was still not 100% fixed. If a POST request contained short enough data to be forwarded at once before trying to establish the connection to the server, we had no way to correctly rewind the body. The first visible case is that balancing on a header does not always work on such POST requests since the header cannot be found. But there are even nastier implications which are that http-send-name-header would apply to the wrong location and possibly even affect part of the request's body due to an incorrect rewinding. There are two options to fix the problem : - first one is to force the HTTP_MSG_F_WAIT_CONN flag on all hash-based balancing algorithms and http-send-name-header, but there's always a risk that any new algorithm forgets to set it ; - the second option is to account for the amount of skipped data before the connection establishes so that we always know the position of the request's body relative to the buffer's origin. The second option is much more reliable and fits very well in the spirit of the past changes to fix forwarding. Indeed, at the moment we have msg-sov which points to the start of the body before headers are forwarded and which equals zero afterwards (so it still points to the start of the body before forwarding data). A minor change consists in always making it point to the start of the body even after data have been forwarded. It means that it can get a negative value (so we need to change its type to signed).. In order to avoid wrapping, we only do this as long as the other side of the buffer is not connected yet. Doing this definitely fixes the issues above for the requests. Since the response cannot be rewound we don't need to perform any change there. This bug was introduced/remained unfixed in 1.5-dev23 so the fix must be backported to 1.5. --- doc/internals/body-parsing.txt | 20 +--- include/types/proto_http.h | 11 ++- src/proto_http.c | 9 +++-- 3 files changed, 26 insertions(+), 14 deletions(-) diff --git a/doc/internals/body-parsing.txt b/doc/internals/body-parsing.txt index e9c8b4b..5baa549 100644 --- a/doc/internals/body-parsing.txt +++ b/doc/internals/body-parsing.txt @@ -67,12 +67,17 @@ msg.next : points to the next byte to inspect. This offset is automatically automatically adjusted to the number of bytes already inspected. msg.sov : start of value. First character of the header's value in the header - states, start of the body in the data states until headers are - forwarded. This offset is automatically adjusted when inserting or - removing some headers. In data states, it always constains the size - of the whole HTTP headers (including the trailing CRLF) that needs - to be forwarded before the first byte of body. Once the headers are - forwarded, this value drops to zero. + states, start of the body in the data states. Strictly positive + values indicate that headers were not forwarded yet (buf.p is + before the start of the body), and null or positive values are seen + after headers are forwarded (buf.p is at or past the start of the + body).
Re: POST with x-www-form-urlencoded Content-Type
Hi Willy, I built a new package with the patch, and my test cases are passing now. Just wanted to say thanks for the super quick turn around on this issue! Thanks! Dan Dubovik Senior Linux Systems Engineer 480-505-8800 x4257 On 7/10/14 10:34 AM, Willy Tarreau w...@1wt.eu wrote: Hi Dan, On Thu, Jul 10, 2014 at 05:20:18PM +0200, Willy Tarreau wrote: Hi Dan, On Wed, Jul 09, 2014 at 07:13:33PM +, Daniel Dubovik wrote: Hello all, I am attempting to balance traffic to a number of backend instances. I am balancing based off the Host header, and for the most part everything is working. When testing a bit more today, I came across some weird behavior, and am hoping someone can help out. When POSTing to a site, if it is done using the Content-Type application/x-www-form-urlencoded, and has actual data, HAProxy falls back to a roundrobin balancing scheme. POSTing using a Content-Type of multipart/form-data, however, works just fine. Oddly, application/x-www-form-urlencoded with no actual data, also works as expected. (...) There's indeed a bug, the amount of data forwarded is not deduced correctly to rewind the buffer, I'm even wondering if it's expected that we let them pass at this point. I'm investigating, thanks for your report! OK I could fix it. The patch is very small but it required some extra care because that's a sensible area that I already fixed in dev23 but not enough. Other balancing algorithms are affected, and worse, http-send-name-header was bogus as well in this case. I've applied the fix, I'm attaching it here, it applies both to 1.5 and to 1.6. Thanks for your report, that was a nasty one and I'm glad we got rid of it early! Willy
Re: POST with x-www-form-urlencoded Content-Type
Hi Dan, On Fri, Jul 11, 2014 at 01:09:32AM +, Daniel Dubovik wrote: Hi Willy, I built a new package with the patch, and my test cases are passing now. Just wanted to say thanks for the super quick turn around on this issue! Great, thank you for the quick feedback! Willy
POST with x-www-form-urlencoded Content-Type
Hello all,
I am attempting to balance traffic to a number of backend instances. I am
balancing based off the Host header, and for the most part everything is
working. When testing a bit more today, I came across some weird behavior, and
am hoping someone can help out. When POSTing to a site, if it is done using
the Content-Type application/x-www-form-urlencoded, and has actual data,
HAProxy falls back to a roundrobin balancing scheme. POSTing using a
Content-Type of multipart/form-data, however, works just fine. Oddly,
application/x-www-form-urlencoded with no actual data, also works as expected.
Log line I receive when posting with data, using multipart/form-data:
Jul 9 11:40:06 xxx haproxy[28084]: 172.19.46.89:52564
[09/Jul/2014:11:40:06.238] fromvarnish fromvarnish/port_10945 0/0/0/45/45 200
426 - - 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xxx.com|multipart/form-data;
boundary=155819574760c61e} POST /posttome.php
HTTP/1.1
Note, it picked a backend that I would expect.
Curl command used to generate the above log:
curl -i -F submit=submit -F firstname= -H Host:
web.2014-07-09-08-28-39.egyitnews.mobi http://10.224.67.9/posttome.php
Log line I receive when posting with out data, using
application/x-www-form-urlencoded
Jul 9 11:41:11 xxx haproxy[28084]: 172.19.46.89:52572
[09/Jul/2014:11:41:11.457] fromvarnish fromvarnish/port_10945 0/0/0/2/2 200 401
- - 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xx.com|application/x-www-form-urlencoded} POST
/posttome.php HTTP/1.1
Note, same backend is picked. This is the ideal behavior.
Curl command used to generate the above log:
curl -i --data-urlencode '' -H Host: web.2014-07-09-08-28-39.egyitnews.mobi
http://10.224.67.9/posttome.php
Log line I receive when posting data, using application/x-www-form-url-encoded
Jul 9 11:40:39 xxx haproxy[28084]: 172.19.46.89:52569
[09/Jul/2014:11:40:39.635] fromvarnish fromvarnish/port_10004 0/0/0/1/1 200 330
- - 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xx.com|application/x-www-form-urlencoded} POST
/posttome.php HTTP/1.1
Jul 9 11:46:29 xxx haproxy[28084]: 172.19.46.89:52597
[09/Jul/2014:11:46:29.703] fromvarnish fromvarnish/port_10005 0/0/0/1/1 200 330
- - 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xx.com|application/x-www-form-urlencoded} POST
/posttome.php HTTP/1.1
Jul 9 11:46:36 xxx haproxy[28084]: 172.19.46.89:52600
[09/Jul/2014:11:46:36.829] fromvarnish fromvarnish/port_10006 0/0/0/1/1 200 330
- - 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xx.com|application/x-www-form-urlencoded} POST
/posttome.php HTTP/1.1
Note, in this one, it picked a different backend each time, in a roundrobin
manner
Curl command used to generate the above log:
curl -i --data-urlencode a=b -H Host: web.2014-07-09-08-28-39.egyitnews.mobi
http://10.224.67.9/posttome.php
My configuration is as follows:
global
log /dev/loglocal0 debug
log 127.0.0.1 local1 notice
maxconn 4096
uid 99
gid 99
#daemon
debug
#quiet
defaults
log global
modehttp
option httplog
option dontlognull
#retries3
#option redispatch
maxconn 2000
timeout connect 5000
timeout client 5
timeout server 5
listen stats
bind :81
stats enable
stats uri /
listen fromvarnish
capture request header Host len 63
capture request header Content-Type len 100
option http-keep-alive
#frontend fromvarnish
bind *:80
# default_backend toapache
#backend toapache
option forwardfor
balance hdr(Host)
hash-type map-based djb2
server port_1 10.224.67.9:1
server port_10001 10.224.67.9:10001
server port_10002 10.224.67.9:10002
server port_10003 10.224.67.9:10003
server port_10004 10.224.67.9:10004
.
.
.
server port_10996 10.224.67.9:10996
server port_10997 10.224.67.9:10997
server port_10998 10.224.67.9:10998
server port_10999 10.224.67.9:10999
http-response set-header X-Port %s
I disabled retries and redispatch, thinking that there was some issue on the
backend, I tried it using either a listen or a frontend-backend pair, and
receive the same results.
I attempted to look at an strace of the two x-www-form-urlencoded behaviors
(with and without content), the results of that are:
12:01:42.011403 recvfrom(2, POST /posttome.php HTTP/1.1\r\nUser-Agent:
curl/7.35.0\r\nAccept: */*\r\nHost:
web.2014-07-09-08-28-39.egyitnews.mobi\r\nContent-Length: 3\r\nContent-Type:
application/x-www-form-urlencoded\r\n\r\na=b, 8192, 0, NULL, NULL) = 186
12:01:42.011804 sendto(3, POST /posttome.php HTTP/1.1\r\nUser-Agent:
curl/7.35.0\r\nAccept: */*\r\nHost:
web.2014-07-09-08-28-39.egyitnews.mobi\r\nContent-Length: 3\r\nContent-Type:
