Dear all,
I'm using haproxy 1.3.15.9 on Ubuntu.
I have a setup with two web servers. One is hosting haproxy and
nginx, it's called s1, and the other one only hosts nginx, and is
called s2. I notice several thousands times a day that the connection
timeout is reached when haproxy on s1 tries to reach nginx on s2.
This is completely unexpected because there are no network problems
between the two. I checked with "ab -n 1000 -c 100" there are no
failed requests. I checked with tcpdump, there are no RST flags
involved. The two servers are located at the same hosting provider,
so I don't believe in network problems anyway. There are no issue at
all on s1 between haproxy and nginx on the same host, only between s1
and s2.
I'm also sure the maxconn setting for this backend is not reached, it
is set to a high value 2000, stats report that max sessions ever seen
is about 200, and the timeouts also happen when there is no other
connection on haproxy.
Other settings involved, stated in the defaults section:
contimeout 5000
retries 3
redispatch
Attached is an example of offending log entries. I notice 1 or 2
retries, never more, but it affects response time because the request
either takes 5 seconds (contimeout) or 10 seconds (2 * contimeout) to
complete.
Note that there are actually two backends configured with the same
server host and port, but with different httpchk settings:
backend nginx_app
option httpchk OPTIONS /check HTTP/1.1\r\nHost:\ site.com
server s1 s1.site.com:83 check inter 10s
server s2 s2.site.com:83 check inter 10s
backend nginx
option httpchk HEAD /site.css HTTP/1.1\r\nHost:\ data.nomao.com
server s1 s1.site.com:83 check inter 10s
server s2 s2.site.com:83 check inter 10s
Last but not least, I have an ufw firewall on s1 and s2 that protect
port 83 from the outside. Only s1 is allowed to connect to port 83 on
s2.
Tell me if you need more information, I'll be glad to provide some.
This is a critical issue for me, as users may go away from my site
because of the response time :S
Thanks in advance,
--
Jean-Baptiste Quenot
http://jbq.caraldi.com/
Sep 4 10:49:24 s1 haproxy[19977]: 127.0.0.1:35243 [04/Sep/2009:10:49:19.370]
frontend nginx/s2 0/5000/0/2/5003 200 21668 - - CD-- 25/15/1/1/1 0/0
{data.site.com} "GET /js/map.js?232be2f HTTP/1.1"
Sep 4 10:49:25 s1 haproxy[19977]: 127.0.0.1:35371 [04/Sep/2009:10:49:20.069]
frontend nginx/s2 0/5005/4/0/5009 200 2128 - - ---- 21/12/0/0/1 0/0
{data.site.com} "GET /img/loader/loader_small_1.gif?232be2f HTTP/1.1"
Sep 4 10:49:30 s1 haproxy[19977]: 127.0.0.1:36535 [04/Sep/2009:10:49:25.036]
frontend nginx_app/s2 0/5000/0/56/5056 200 4796 - - ---- 0/0/0/0/1 0/0
{img.site.com} "GET
/map/spots?size=256&box=2.197265625,48.893615361480194,2.2412109375,48.922499263758254&zoom=13&tagstring=faire%20du%20shopping%20supermarche%20franprix&ids=1315616,1315575
HTTP/1.1"
Sep 4 10:49:38 s1 haproxy[19977]: 127.0.0.1:38338 [04/Sep/2009:10:49:33.009]
frontend nginx/eg2 0/5005/0/0/5005 200 638 - - ---- 4/2/0/0/1 0/0
{data.site.com} "GET /img/nav/fbconnect.png?232be2f HTTP/1.1"
