Hello,
On Sun, May 29, 2016 at 12:53:21AM +0430, Behrad Zari wrote:
> We have over 90K concurrent TCP persistent connections on our single haproxy
> instance facing mobile clients on the Internet. Our normal conn rate for 100K
> is around
> 200/sec which I don't have a clue if it is a good one or not for public
> internet mobile
> clients with keepalive 3-5mins.
It's easy to do the computation, 100K/200 = 500 seconds of duration on average
for a connection. It may indicate that your website is well designed (few
redirects etc) as it's not quite common to see that low a connection rate for
a high connection count.
> Our stats normally shows:
> (process #1, nbproc = 1)
> system limits: memmax = unlimited; ulimit-n = 4000096
> maxsock = 800060; maxconn = 200000; maxpipes = 200000
> current conns = 104402; current pipes = 0/0; conn rate = 164/sec
> Running tasks: 24/104441; idle = 10 %
Here there's something abnormal : 10% idle (meaning the process spends 90%
of it time doing something). For only 164 conn/s!
> After a shortage in our backends, when all clients got disconnected, and
> tried to
> reconnect afterwards, We saw haproxy machine was not responsive! Even stats
> page
> was not showing up completely, conn rate was around 6600/sec at once, and
> only some
> kernel TCP too many orphaned sockets in messages log. It took us hours,
> starting with
> a very low maxconn & increasing it timely to handle clients rush of
> reconnects...! :(
Here's the list of possible things I'm thinking about for such a situation :
- improperly tuned sysctls (somaxconn and tcp_max_backlog) limiting the
incoming connection rate ; this results in the inability to accept new
connections and SYN packets being retransmitted by the client. The CPU
should then be saturated.
- too short a "backlog" parameter in the affected frontend (causes the
same effect as above).
- a "rate-limit sessions" setting being present. CPU will not be saturated.
- huge amount of iptables rules on the system causing each new connection to
take a long time to evaluate. In this case most of the CPU will be spent
in softirq.
- a lot of SSL rekeying. As a rule of thumb, a 3 GHz CPU can produce
around 1000 RSA2048 signatures per second. Since you're running at 200/s
when everything's OK, less than 20% of the CPU is spent doing handshakes.
If all connections have to be re-handshaked, it can take a while (100 sec
of pure CPU just for RSA). This can definitely cause large delays. The
recent TLS ticket secret manipulation that was introduced in 1.7-dev can
definitely help regarding this.
- a large number of orphans can eat a lot of socket memory. It may be
possible that you experienced some packet drops because of a large number
of orphans. But at the same time the fact that the kernel complained about
them indicates it got rid of them, which means that you reached the orphan
limit before reaching the memory limit, so it should not be an issue in
fact.
> We have multiple front-end, backends, also using SSL offloading & some sysctl
> tunings.
>
> Is multiple thousand TCP persistent conn rate too much for a single haproxy
> box?
No, but it definitely requires some careful tuning. At least the fact that it
usually works seems to indicate to me that you've done this tuning (or part of
it).
> Which kernel/tcp/haproxy parameters can we use to tune-up for such scenarios?
Well, a lot :-)
Please start by checking the points above to see if something may match your
case.
Regards,
Willy
