> On Thu, Oct 27, 2016 at 12:29:56PM +0000, Gernot Pörner wrote: >> > That's expected. "use>0" means that some sessions still track this entry, >> > so it cannot be removed. The value represents the number of trackers >> > still on it. It is possible that you're having some persistent HTTP >> > connections bound to it and that you may have to wait for the idle >> > timeout to expire. >> >> As far as I can see there are no related sessions anymore. These entries >> are in there since 2 days now, when the attacker stopped hammering on our >> api. > > OK so it seems like something odd happened. > >> Our longest timeout in haproxy anywhere is 180 seconds. Or did you mean >> the tcp timeouts of the kernel? These are currently set to 600s >> (/proc/sys/net/ipv4/tcp_keepalive_time). > > No I was talking about haproxy's timeouts. If all sessions are closed > it's abnormal. > >> > If you're seeing the same ones last much longer >> > than the request timeout, there might be another issue though. Then >> > issuing "show sess all" on the stats socket to get a dump before >> > restarting may possibly help. >> >> There are only session of the last couple of minutes in there. Are there >> any other ways/places I should look? > > No, that was the right place. > >> With lsof I also can't see any related TCP sessions anymore. > > So in the end they disappeared on both sides, yet there's a refcount > issue. I'd call that a bug. I don't remember, do you use peers to > synchronize your stick tables ? Nothing special, I'm just trying to > bisect the problem.
No, it's a single instance, nothing fancy. It's actually the first time we use stick tables this way. Besides the somehow odd behaviour it seems to be fine but since we also want to monitor on the amount of entries in that table it's a bit sub-optimal. I don't think/hope that "normal" users with harmless IPs end up there. >> > By the way if you want to use stick-tables for such usages, I strongly >> > recommend trying 1.6 which is richer there. You can for example track in >> > http-request rules, and you can exchange the stick counters between peers. >> >> I plan on doing that as soon as possible. > > OK. As a rule of thumb, don't change your config and your version at the > same time. That way if you were to hit a regression (always to be expected > once you've met a problem nobody has ever met), it's easier to roll back > to the previous version. I just tested our config with the latest 1.6 from the ppa, it seems to load flawlessly. As soon as our QA gives the go I'll introduce it in production. Then I have to wait for the attacker to try again ;-) Right now we accumulate kind of a sediment of ~130 table entries... Thanks Gernot
