Hi, On Wed, Oct 26, 2016 at 11:11:23AM +0000, Gernot Pörner wrote: > Hi, > > we are running haproxy 1.5.18 on Ubuntu 14. > > Recently we had some issues with misuse of our api by botnets coming from > many different IPs > and implemented a mitigation based on haproxys stick-table feature which > works pretty good for us. > > The relevant part of our config looks like this: > > acl document_request path_beg -i /api/v1/sessions > acl too_many_tries sc0_gpc0_rate() gt 5 > acl mark_seen sc0_inc_gpc0 gt 0 > stick-table type string size 100k expire 5m store gpc0_rate(5s) > tcp-request content track-sc0 hdr(True-Client-IP) if > METH_POST document_request > use_backend l2b_slow_down if mark_seen too_many_tries > backend l2b_slow_down > mode http > timeout tarpit 20s > http-request tarpit > > Now, when some IP (IPv4 or IPv6 doesn't matter since we use string here) > sends more than > 5 POST requests to our api resource in 5 seconds, it will end up in the table > and all > subsequent requests are delayed and answered with an error 500. > > After 5 minutes the table entry should expire. This usually works but > sometimes it > doesn't. We see entries in the stick table whose expiry is down to 0 but they > don't > get removed since the "use" counter is something greater than zero > > It then looks like this: > > echo "show table l1f_loadbalancer" | sudo socat stdio /var/run/haproxy.stat > > # table: l1f_loadbalancer, type: string, size:102400, used:7 > 0x429af6c: key=37.191.169.199 use=1 exp=0 gpc0_rate(5000)=0 > 0x2f1d54c: key=37.191.204.19 use=1 exp=0 gpc0_rate(5000)=0 > 0x3c105bc: key=50.115.248.18 use=1 exp=0 gpc0_rate(5000)=0 > 0x3293ebc: key=65.214.65.27 use=1 exp=0 gpc0_rate(5000)=0 > 0x30b0acc: key=68.180.24.112 use=2 exp=0 gpc0_rate(5000)=0 > 0x53571bc: key=68.180.57.63 use=1 exp=0 gpc0_rate(5000)=0 > 0x300f93c: key=68.187.196.250 use=1 exp=0 gpc0_rate(5000)=0 > > I also cannot delete them while they are "in use": > > echo "clear table l1f_loadbalancer key 50.115.248.18" | sudo socat stdio > /var/run/haproxy.stat > Entry currently in use, cannot remove > > clear table without a key only removes entries which are at use=0
That's expected. "use>0" means that some sessions still track this entry, so it cannot be removed. The value represents the number of trackers still on it. It is possible that you're having some persistent HTTP connections bound to it and that you may have to wait for the idle timeout to expire. If you're seeing the same ones last much longer than the request timeout, there might be another issue though. Then issuing "show sess all" on the stats socket to get a dump before restarting may possibly help. By the way if you want to use stick-tables for such usages, I strongly recommend trying 1.6 which is richer there. You can for example track in http-request rules, and you can exchange the stick counters between peers. Willy
