Hi John, On Thu, Oct 15, 2009 at 10:44:58AM -0400, John Lauro wrote: > I am starting to have problems with one of our servers behind haproxy. It's > busy, but it has more resources to handle more connections, but is having a > bit of trouble with the incoming rate and is getting flagged down. I am > having trouble finding what /proc (linux 2.6) to tweak. > > > > (These are from the server processing the requests, not the haproxy server) > > # netstat -s | grep -i list > > 1911871 times the listen queue of a socket overflowed > > 1911871 SYNs to LISTEN sockets ignored > > > > Anything look bad or missing: > > echo 1024 60999 > /proc/sys/net/ipv4/ip_local_port_range > > echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout > > echo 4096 > /proc/sys/net/ipv4/tcp_max_syn_backlog > > echo 262144 > /proc/sys/net/ipv4/tcp_max_tw_buckets > > echo 262144 > /proc/sys/net/ipv4/tcp_max_orphans > > echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time > > echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle > > echo 0 > /proc/sys/net/ipv4/tcp_timestamps > > echo 0 > /proc/sys/net/ipv4/tcp_ecn > > echo 1 > /proc/sys/net/ipv4/tcp_sack > > echo 0 > /proc/sys/net/ipv4/tcp_dsack > > echo 3000 > /proc/sys/net/core/netdev_max_backlog > > echo 3000 > /proc/sys/net/core/somaxconn
These parameters look OK. One thing that can explain the high number of dropped SYNs would be the backlog configured by the application itself. Check where it does its listen(), most likely there is a "listen(fd,5)" there because that example was very common in old books but it sets a limit on the backlog queue. It's also possible that your application accepts a connection, processes one part of a request, then accepts a new one, thus increasing latency between connections acceptance and limiting the connection rate. > The haproxy server shows a max of about 200 connections to this backend at > any one time, and the server does make lots of outgoing connections too, but > not much else incoming. The application should be able to handle 500 > connections. > > Shouldn't be sending more then about 200, maybe 300 connections a second. > Could be bursty though, so I guess some seconds might be higher. What you describe typically matches a high SYN drop rate caused by too short a backlog or too long a delay between two accept() calls. Regards, Willy
