Hi Jérôme, On Wed, Jan 24, 2018 at 02:38:20PM +0100, Jérôme Magnin wrote: > Hi, > > I've been toying with haproxy and rate limiting lately, and noticed an odd > behavior with rate-limit sessions, or maybe I misunderstood how it is supposed > to be used. > > I'm using the following config: > > global > maxconn 20000 > log 127.0.0.1 local0 > user haproxy > chroot /usr/share/haproxy > pidfile /run/haproxy.pid > daemon > stats socket /var/run/haproxy.sock > > defaults > mode http > > frontend fe_foo > bind *:1234 > bind *:1235 ssl crt /etc/haproxy/www.pem > rate-limit sessions 10 > default_backend be_foo > > backend be_foo > server s1 127.0.0.1:8001 > > I'm using ab to send traffic to the frontend. > > 1/ ab -c 40 -n 100 http://127.0.0.1:1234/ > > the output of show info shows maxconnrate 10 and maxsessrate 10. > This is coherent with the value I set for rate-limit sessions. > > 2/ ab -c 40 -n 100 https://127.0.0.1:1235/ > > the output of show info shows maxconnrate, maxsslrate, maxsessrate and > sslfrontendmaxkeyrate equal 40, 4 times the value for my rate-limit sessions. > > Am I doing something wrong here ?
No but we're on an annoying corner case as I explained to you. I'm putting all the elements here so that everyone has them as well. What happens is that the listener_accept() function which performs the accept() call refrains from accepting a new connection when the frontend's session rate has reached the configured session rate limit. Sessions are instanciated at the end of handshakes, and in the case of SSL, it happens once the SSL handshake with the client is completed. So in practice the behaviour you're observing is that many clients connect at once. The frontend's session rate remains zero since none of them has yet completed a handshake, and once a few handshakes are completed, we stop accepting new connections for a while, but all already accepted connections still have to be handshaked. Thus we see a rush of handshakes during all initial connections, and only then it falls and is smoothed. I'm a bit embarrassed with how to address this. Addressing it with a connection rate limit is the wrong response since it will not allow you to use tcp-request connection rules to ignore certain connections, so that creates very trivial DoS vectors (ie just connect 10 times in a row using netcat and nobody connects for one second ; worse, send 50000 connection requests and you'll spend 1.5 hours slowly draining the pending handshakes). I thought that one really nice thing to do would be to try to improve the server side to support per-server connection rate limiting in addition to the connection concurrency control we already have. Given that most of the time, "rate-limit session" is used in fact to protect backend servers, it would allow the limits to be more accurate. But it doesn't solve the root cause of the problem. Another idea I was considering would be to place a limit on the number of concurrent handshakes per frontend (or listener). That would be very convenient as we'd refrain from accepting new connections if we have already queued enough of them to keep the handshake code busy. But this one is tricky and there's a high risk of losing some accounting in certain error cases. So I'm still thinking about it, not having many solutions to propose at the moment :-/ Thanks, Willy
