Hello all,
I am attempting to balance traffic to a number of backend instances. I am
balancing based off the Host header, and for the most part everything is
working. When testing a bit more today, I came across some weird behavior, and
am hoping someone can help out. When POSTing to a site, if it is done using
the Content-Type application/x-www-form-urlencoded, and has actual data,
HAProxy falls back to a roundrobin balancing scheme. POSTing using a
Content-Type of multipart/form-data, however, works just fine. Oddly,
application/x-www-form-urlencoded with no actual data, also works as expected.
Log line I receive when posting with data, using multipart/form-data:
Jul 9 11:40:06 xxxxxxx haproxy[28084]: 172.19.46.89:52564
[09/Jul/2014:11:40:06.238] fromvarnish fromvarnish/port_10945 0/0/0/45/45 200
426 - - ---- 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xxxxxxx.com|multipart/form-data;
boundary=------------------------155819574760c61e} "POST /posttome.php
HTTP/1.1"
Note, it picked a backend that I would expect.
Curl command used to generate the above log:
curl -i -F submit=submit -F firstname= -H "Host:
web.2014-07-09-08-28-39.egyitnews.mobi" http://10.224.67.9/posttome.php
Log line I receive when posting with out data, using
application/x-www-form-urlencoded
Jul 9 11:41:11 xxxxxxx haproxy[28084]: 172.19.46.89:52572
[09/Jul/2014:11:41:11.457] fromvarnish fromvarnish/port_10945 0/0/0/2/2 200 401
- - ---- 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xxxxxx.com|application/x-www-form-urlencoded} "POST
/posttome.php HTTP/1.1"
Note, same backend is picked. This is the ideal behavior.
Curl command used to generate the above log:
curl -i --data-urlencode '' -H "Host: web.2014-07-09-08-28-39.egyitnews.mobi"
http://10.224.67.9/posttome.php
Log line I receive when posting data, using application/x-www-form-url-encoded
Jul 9 11:40:39 xxxxxxx haproxy[28084]: 172.19.46.89:52569
[09/Jul/2014:11:40:39.635] fromvarnish fromvarnish/port_10004 0/0/0/1/1 200 330
- - ---- 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xxxxxx.com|application/x-www-form-urlencoded} "POST
/posttome.php HTTP/1.1"
Jul 9 11:46:29 xxxxxxx haproxy[28084]: 172.19.46.89:52597
[09/Jul/2014:11:46:29.703] fromvarnish fromvarnish/port_10005 0/0/0/1/1 200 330
- - ---- 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xxxxxx.com|application/x-www-form-urlencoded} "POST
/posttome.php HTTP/1.1"
Jul 9 11:46:36 xxxxxxx haproxy[28084]: 172.19.46.89:52600
[09/Jul/2014:11:46:36.829] fromvarnish fromvarnish/port_10006 0/0/0/1/1 200 330
- - ---- 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.xxxxxx.com|application/x-www-form-urlencoded} "POST
/posttome.php HTTP/1.1"
Note, in this one, it picked a different backend each time, in a roundrobin
manner
Curl command used to generate the above log:
curl -i --data-urlencode a=b -H "Host: web.2014-07-09-08-28-39.egyitnews.mobi"
http://10.224.67.9/posttome.php
My configuration is as follows:
global
log /dev/log local0 debug
log 127.0.0.1 local1 notice
maxconn 4096
uid 99
gid 99
#daemon
debug
#quiet
defaults
log global
mode http
option httplog
option dontlognull
#retries 3
#option redispatch
maxconn 2000
timeout connect 5000
timeout client 50000
timeout server 50000
listen stats
bind :81
stats enable
stats uri /
listen fromvarnish
capture request header Host len 63
capture request header Content-Type len 100
option http-keep-alive
#frontend fromvarnish
bind *:80
# default_backend toapache
#backend toapache
option forwardfor
balance hdr(Host)
hash-type map-based djb2
server port_10000 10.224.67.9:10000
server port_10001 10.224.67.9:10001
server port_10002 10.224.67.9:10002
server port_10003 10.224.67.9:10003
server port_10004 10.224.67.9:10004
.
.
.
server port_10996 10.224.67.9:10996
server port_10997 10.224.67.9:10997
server port_10998 10.224.67.9:10998
server port_10999 10.224.67.9:10999
http-response set-header X-Port %s
I disabled retries and redispatch, thinking that there was some issue on the
backend, I tried it using either a listen or a frontend-backend pair, and
receive the same results.
I attempted to look at an strace of the two x-www-form-urlencoded behaviors
(with and without content), the results of that are:
12:01:42.011403 recvfrom(2, "POST /posttome.php HTTP/1.1\r\nUser-Agent:
curl/7.35.0\r\nAccept: */*\r\nHost:
web.2014-07-09-08-28-39.egyitnews.mobi\r\nContent-Length: 3\r\nContent-Type:
application/x-www-form-urlencoded\r\n\r\na=b", 8192, 0, NULL, NULL) = 186
12:01:42.011804 sendto(3, "POST /posttome.php HTTP/1.1\r\nUser-Agent:
curl/7.35.0\r\nAccept: */*\r\nHost:
web.2014-07-09-08-28-39.egyitnews.mobi\r\nContent-Length: 3\r\nContent-Type:
application/x-www-form-urlencoded\r\nX-Forwarded-For: 172.19.46.89\r\n\r\na=b",
217, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 217
12:01:42.013062 sendto(5, "<134>Jul 9 12:01:42 haproxy[28084]:
172.19.46.89:52726 [09/Jul/2014:12:01:42.010] fromvarnish
fromvarnish/port_10007 0/0/0/1/1 200 330 - - ---- 2/2/0/1/0 0/0
{web.2014-07-09-08-28-39.egyitnews.mobi|application/x-www-form-urlencoded}
\"POST /posttome.php HTTP/1.1\"\n", 264, MSG_DONTWAIT|MSG_NOSIGNAL,
{sa_family=AF_FILE, path="/dev/log"}, 110) = 264
12:01:48.324089 recvfrom(1, "POST /posttome.php HTTP/1.1\r\nUser-Agent:
curl/7.35.0\r\nAccept: */*\r\nHost:
web.2014-07-09-08-28-39.egyitnews.mobi\r\nContent-Length: 0\r\nContent-Type:
application/x-www-form-urlencoded\r\n\r\n", 8192, 0, NULL, NULL) = 183
12:01:48.324405 sendto(2, "POST /posttome.php HTTP/1.1\r\nUser-Agent:
curl/7.35.0\r\nAccept: */*\r\nHost:
web.2014-07-09-08-28-39.egyitnews.mobi\r\nContent-Length: 0\r\nContent-Type:
application/x-www-form-urlencoded\r\nX-Forwarded-For: 172.19.46.89\r\n\r\n",
214, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 214
12:01:48.367549 recvfrom(2, "HTTP/1.1 200 OK\r\nDate: Wed, 09 Jul 2014 19:01:48
GMT\r\nServer: Apache\r\nVary: Accept-Encoding\r\nX-XID:
2052615\r\nX-Web-Server: xxxxxxxxxx.dev.glbt1.gdg\r\nTransfer-Encoding:
chunked\r\nContent-Type: text/html\r\n\r\nb6\r\n<form action = \"\" method =
POST>\nFirst name: <input type=\"text\" name=\"firstname\"><br>\nLast name:
<input type=\"text\" name=\"lastname\"><br>\n<input type=\"submit\"
value=\"Submit\">\n</form>\n\r\n0\r\n\r\n", 8192, 0, NULL, NULL) = 401
12:01:48.367692 sendto(5, "<134>Jul 9 12:01:48 haproxy[28084]:
172.19.46.89:52728 [09/Jul/2014:12:01:48.323] fromvarnish
fromvarnish/port_10945 0/0/0/43/43 200 401 - - ---- 1/1/0/1/0 0/0
{web.2014-07-09-08-28-39.egyitnews.mobi|application/x-www-form-urlencoded}
\"POST /posttome.php HTTP/1.1\"\n", 266, MSG_DONTWAIT|MSG_NOSIGNAL,
{sa_family=AF_FILE, path="/dev/log"}, 110) = 266
12:01:48.367968 sendto(1, "HTTP/1.1 200 OK\r\nDate: Wed, 09 Jul 2014 19:01:48
GMT\r\nServer: Apache\r\nVary: Accept-Encoding\r\nX-XID:
2052615\r\nX-Web-Server: xxxxxxx.dev.glbt1.gdg\r\nTransfer-Encoding:
chunked\r\nContent-Type: text/html\r\nX-Port: port_10945\r\n\r\nb6\r\n<form
action = \"\" method = POST>\nFirst name: <input type=\"text\"
name=\"firstname\"><br>\nLast name: <input type=\"text\"
name=\"lastname\"><br>\n<input type=\"submit\"
value=\"Submit\">\n</form>\n\r\n0\r\n\r\n", 421, MSG_DONTWAIT|MSG_NOSIGNAL,
NULL, 0) = 421
Thanks!
Dan Dubovik
Senior Linux Systems Engineer
480-505-8800 x4257
