Re: Three questions about stick-tables and request rate limiting
Hi Dennis, Answering inline in your email. > Question 1: Is there a better way to reset the gpc0 counter other than > waiting for the stick-table entry to expire? > > In my test if I hit haproxy with the load-testing tool apache bench to > trigger the 10 req/s limit for two seconds and then follow that up with > a pattern of 1 req/s for a minute these requests will never succeed > because gpc0 is greater than zero, will never reset and the stick-table > entry will never expire because the timer will always get reset by the 1 > req/s pattern so the user is effectively locked out forever even though > he is no longer exceeding the request/s limit. > > Wouldn't it be better to reset the gpc0 counter to zero once > http_req_rate has dropped below 10 again to not create this kind of > perma-block? yes, you can, there is a sample called sc0_clr_gpc0 whose purpose is to clear the value of gpc0. an other solution would not to measure gpc0 itself but its growing rate using sc0_gpc0_rate. growing would be very low with 1 request per minute. > Question 2: When I use wrk instead of ab it seems the request limiting > doesn't work at all. What wrk does is it doesn't create new connections > for each request but only creates a bunch of connections initially and > then sends all requests using these permanent connections. These are a > couple of stick-table dumps I did after starting the wrk test: > > 0xe5e854: key=10.99.0.1 use=10 exp=7791 gpc0=15771 conn_cur=10 > http_req_rate(1)=15780 > 0xe5e854: key=10.99.0.1 use=10 exp=7247 gpc0=19767 conn_cur=10 > http_req_rate(1)=19776 > 0xe5e854: key=10.99.0.1 use=10 exp=6727 gpc0=23606 conn_cur=10 > http_req_rate(1)=23615 > 0xe5e854: key=10.99.0.1 use=10 exp=6247 gpc0=26718 conn_cur=10 > http_req_rate(1)=26727 > 0xe5e854: key=10.99.0.1 use=10 exp=5823 gpc0=29760 conn_cur=10 > http_req_rate(1)=29769 > 0xe5e854: key=10.99.0.1 use=10 exp=5424 gpc0=32622 conn_cur=10 > http_req_rate(1)=32631 > 0xe5e854: key=10.99.0.1 use=10 exp=4967 gpc0=35964 conn_cur=10 > http_req_rate(1)=35973 > 0xe5e854: key=10.99.0.1 use=10 exp=4567 gpc0=38779 conn_cur=10 > http_req_rate(1)=38788 > > Notice how the http_req_rate keeps going up as does the gpc0 counter yet > wrk doesn't report any failed requests and a result of several thousand > requests per second. > > The impression I get here is that this configuration doesn't *really* > limit the number of requests but only the number of connections based on > the request rate which is semantically a bit different and still allows > a potential abuser to send as many requests as he wants as long as he > keeps using an existing connection. > Is this impressions correct and is the a way to truly limit the number > of requests/s even when no new connections are made? instead of flagging a request, you can simply deny it. HAProxy will then close the TCP connection and the user won't be allowed to establish a new one. > Question 3: As you can see in the configuration I'm using a https > frontend that proxies the traffic to the http frontend so that I can get > the combined stats in the single-process http frontend while still being > able to put the https frontend on independent processes to distribute > the load among cores. > > What I noticed though is that when I do the above tests on the SSL > frontend I don't get any stick-table entries in the regular http > frontend. Apparently the proxied connection aren't registered by the > stick-table. Is there a way to get these connections to show up as well > or do I have to copy+paste the stick-table and abuse settings and keep > them manually in sync between the two frontends? There should be no difference between SSL and clear traffic. I can reproduce the behavior: there might a bug when passing through a unix socket. As a workaround, you can failover to a loopback IP address. In order to populate a blacklist between clear and SSL frontends, you can use the 'http-response add-acl'. Hope this helps. Baptiste
Three questions about stick-tables and request rate limiting
Hi,
I'm currently getting a feel for how the stick-tables work in connection
with rate limiting requests and three questions have come up.
As a foundation I used this link:
http://blog.haproxy.com/2012/02/27/use-a-load-balancer-as-a-first-row-of-defense-against-ddos/
(Specifically the "Limiting the HTTP request rate" section)
I've attached the configuration that I'm currently using.
Question 1: Is there a better way to reset the gpc0 counter other than
waiting for the stick-table entry to expire?
In my test if I hit haproxy with the load-testing tool apache bench to
trigger the 10 req/s limit for two seconds and then follow that up with
a pattern of 1 req/s for a minute these requests will never succeed
because gpc0 is greater than zero, will never reset and the stick-table
entry will never expire because the timer will always get reset by the 1
req/s pattern so the user is effectively locked out forever even though
he is no longer exceeding the request/s limit.
Wouldn't it be better to reset the gpc0 counter to zero once
http_req_rate has dropped below 10 again to not create this kind of
perma-block?
Question 2: When I use wrk instead of ab it seems the request limiting
doesn't work at all. What wrk does is it doesn't create new connections
for each request but only creates a bunch of connections initially and
then sends all requests using these permanent connections. These are a
couple of stick-table dumps I did after starting the wrk test:
0xe5e854: key=10.99.0.1 use=10 exp=7791 gpc0=15771 conn_cur=10
http_req_rate(1)=15780
0xe5e854: key=10.99.0.1 use=10 exp=7247 gpc0=19767 conn_cur=10
http_req_rate(1)=19776
0xe5e854: key=10.99.0.1 use=10 exp=6727 gpc0=23606 conn_cur=10
http_req_rate(1)=23615
0xe5e854: key=10.99.0.1 use=10 exp=6247 gpc0=26718 conn_cur=10
http_req_rate(1)=26727
0xe5e854: key=10.99.0.1 use=10 exp=5823 gpc0=29760 conn_cur=10
http_req_rate(1)=29769
0xe5e854: key=10.99.0.1 use=10 exp=5424 gpc0=32622 conn_cur=10
http_req_rate(1)=32631
0xe5e854: key=10.99.0.1 use=10 exp=4967 gpc0=35964 conn_cur=10
http_req_rate(1)=35973
0xe5e854: key=10.99.0.1 use=10 exp=4567 gpc0=38779 conn_cur=10
http_req_rate(1)=38788
Notice how the http_req_rate keeps going up as does the gpc0 counter yet
wrk doesn't report any failed requests and a result of several thousand
requests per second.
The impression I get here is that this configuration doesn't *really*
limit the number of requests but only the number of connections based on
the request rate which is semantically a bit different and still allows
a potential abuser to send as many requests as he wants as long as he
keeps using an existing connection.
Is this impressions correct and is the a way to truly limit the number
of requests/s even when no new connections are made?
Question 3: As you can see in the configuration I'm using a https
frontend that proxies the traffic to the http frontend so that I can get
the combined stats in the single-process http frontend while still being
able to put the https frontend on independent processes to distribute
the load among cores.
What I noticed though is that when I do the above tests on the SSL
frontend I don't get any stick-table entries in the regular http
frontend. Apparently the proxied connection aren't registered by the
stick-table. Is there a way to get these connections to show up as well
or do I have to copy+paste the stick-table and abuse settings and keep
them manually in sync between the two frontends?
Regards,
Dennis
listen front-https
bind-process 2
bind 10.99.0.200:443 ssl crt /etc/pki/tls/certs/STAR.example.de.chain.pem
ciphers
ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!eNULL:!NULL:!DH:!EDH:!AESGCM
no-sslv3 npn http/1.1
http-request set-header X-Forwarded-Proto https
server clear abns@ssl-proxy send-proxy
frontend front1
bind-process 1
bind 10.99.0.200:80
bind abns@ssl-proxy accept-proxy
stick-table type ip size 200k expire 10s store
conn_cur,gpc0,http_req_rate(10s)
tcp-request connection track-sc0 src
tcp-request connection reject if { src_get_gpc0 gt 0 }
default_backend back1
backend back1
bind-process 1
mode http
balance roundrobin
option httpchk GET /health.txt HTTP/1.1\r\nHost:\ 10.99.0.1
http-check expect string alive
# If the source IP sent 10 or more http request over the defined period,
# flag the IP as abuser on the frontend
acl abuse src_http_req_rate(front1) ge 10
acl flag_abuser src_inc_gpc0(front1) --
tcp-request content reject if abuse flag_abuser
stick-table type ip size 200k expire 30m
stick on src
server websvr1 10.99.0.1:80 check
server websvr2 10.99.0.200:8080 check
