Hello, I managed to get better performance after doing a bit more configuration Such as binding the haproxy processes to all cores in the same physical CPU Also bonded all HTTP to these cores and TCP to the other cores on the second physical CPU
I was able to protect the backends (maxconn) now it is up to me to optimize and find the magic number to utilize exactly 65% of each backend cpu. (more than that will cause latency) As far as I can see in the stats page the total haproxy connections increases when the backends hit maxconn limit And I guess this is what I should expect. After tuning the global ulimit -n and maxconn to prevent the haproxy itself from crashing everything looks better. (no haproxy crashes) I will update (with configuration details) if I'll see this issue again, but I think this is the right direction. Thank you for the info. Haim -----Original Message----- From: Lukas Tribus [mailto:[email protected]] Sent: Friday, June 3, 2016 12:45 PM To: Haim Ari <[email protected]>; [email protected] Subject: Re: Haproxy limiting queue / connections Hi Ari, > If the backend servers are responding slowly for some reason, all > haproxy nodes will crash. > > i configured queue to 1 and set up reasonable server and client > timeouts on haproxy > > > however if for some reason connections builds up on haproxy crashes > and the machine becomes unresponsive (all haproxy nodes) > > > I don't see any kernel errors. > > > 1. how can i debug this ? > > 2. what is the best practice to protect haproxy from such situation ? > So there is nothing in the logs, nothing in dmesg? No OOM events, no crashes? Then how do you know haproxy is crashing, if there is not a single log line about it? The node becomes completely unresponsive, like not even ping and ssh works? Then it is likely not a haproxy problem (but do provide "top" output anyway). Sounds more like a stateful firewall in front of haproxy with exhausted resources. Also, how do you know connections are building up on haproxy? Can you see that on the stat interface? If yes, how does that look like exactly? Please share the haproxy configuration and check your stateful firewalls for resource exhaustion. Regards, Lukas
