Hello.
I have setup haproxy as frontend for php/nginx farm for our company application.
Our app contains two main modules - http frontend and webservice
apllication server.
http uses normal port - 80 (ssl through stunel) and webservice apps
are available on port 8000
Ive decided to use one backend server with leastconn algorithm and
setting "+" prefix for each server to redirect connection to port 81
or 8001 on which runs nginx.
Everything worked perfect until i checked log on client application
wich uses our webservice application.
For some request there were 503 responses "service unavailble"
Digging into logs and tcpdump i found out that haproxy sometimes
manages to send client_port instead of destination port to rewriting
algoritm which ends up in requesting backend server on nonexisting
port.
Previous config
backend app
balance leastconn
server diabel2_php diabel2_php:+1 check port 81 weight 2 maxconn 1
server diabel3_php diabel3_php:+1 check port 81 weight 2 maxconn 2
server diabel1_php diabel1_php:+1 check port 81 weight 2 maxconn 4
server bies1_php bies1_php:+1 check port 81 weight 3 maxconn 250
server bies2_php bies2_php:+1 check port 81 weight 3 maxconn 250
server bies3_php bies3_php:+1 check port 81 weight 3 maxconn 250
server bies4_php bies4_php:+1 check port 81 weight 3 maxconn 250
haproxy.strace
grep connect haproxy.strace |grep "10.6.10"|grep -v 8001|grep -v 81
10:10:52 connect(11, {sa_family=AF_INET, sin_port=htons(60988),
sin_addr=inet_addr("10.6.10.102")}, 16) = -1 EINPROGRESS (Operation
now in progress)
10:10:53 connect(17, {sa_family=AF_INET, sin_port=htons(60988),
sin_addr=inet_addr("10.6.10.102")}, 16) = -1 EINPROGRESS (Operation
now in progress)
10:10:54 connect(1, {sa_family=AF_INET, sin_port=htons(60988),
sin_addr=inet_addr("10.6.10.102")}, 16) = -1 EINPROGRESS (Operation
now in progress)
10:10:55 connect(12, {sa_family=AF_INET, sin_port=htons(60988),
sin_addr=inet_addr("10.6.10.1")}, 16) = -1 EINPROGRESS (Operation now
in progress)
10:10:56 connect(13, {sa_family=AF_INET, sin_port=htons(1973),
sin_addr=inet_addr("10.6.10.1")}, 16) = -1 EINPROGRESS (Operation now
in progress)
10:10:56 connect(13, {sa_family=AF_INET, sin_port=htons(4756),
sin_addr=inet_addr("10.6.10.104")}, 16) = -1 EINPROGRESS (Operation
now in progress)
10:10:56 connect(11, {sa_family=AF_INET, sin_port=htons(4037),
sin_addr=inet_addr("10.6.10.102")}, 16) = -1 EINPROGRESS (Operation
now in progress)
coresponding haproxy.log:
Feb 8 10:10:55 localhost haproxy[25750]: 79.189.159.77:60987
[08/Feb/2012:10:10:52.036] http app/diabel1_php 6/3005/-1/-1/3012 503
212 - - SC-- 4/4/2/0/+3 0/0 "GET
/f82b0a1554d0426a476c391ce49442ba/?request=PopReceipt HTTP/1.1"
Feb 8 10:10:59 localhost haproxy[25750]: 79.187.248.101:1972
[08/Feb/2012:10:10:56.213] http app/bies1_php 1/3004/-1/-1/3006 503
212 - - SC-- 9/9/9/1/+3 0/0 "GET
/56b919f0b3592d6624ef32e3e96f1de9/?request=PopReceipt HTTP/1.1"
Feb 8 10:10:59 localhost haproxy[25750]: 95.49.216.250:4755
[08/Feb/2012:10:10:56.347] http app/bies2_php 8/3006/-1/-1/3016 503
212 - - SC-- 6/6/6/0/+3 0/0 "GET
/ab48266e45a77df9ba5f693c05484691/?request=PopReceipt HTTP/1.1"
10.6.10.* are backend servers (diabel|bies)[1234]_php
For now i created separated backends for webservice and normal web
frontend and i havent spoted errors anymore.
If i can provide anymore info just let me know.
Greetins
--
Pozdrawiam
Mateusz Ratajski // http://exp.pl
