Hi again, we are running haproxy 1.6.9 for a week now and the problem persists, we had some 30000 entries in there, all of them should now be timed out but about half of them are still in there flagged as "in use" and with a exp of 0:
0x7faef417d3ec: key=105.106.165.238 use=120 exp=0 gpc0_rate(10000)=0 0x7faef429db1c: key=105.106.172.152 use=171 exp=0 gpc0_rate(10000)=0 0x7faef42f0c2c: key=105.106.177.245 use=235 exp=0 gpc0_rate(10000)=0 0x7faef3fd001c: key=105.106.184.115 use=99 exp=0 gpc0_rate(10000)=0 0x7faef3b2c1fc: key=105.106.21.28 use=40 exp=0 gpc0_rate(10000)=0 0x7faef2c1d89c: key=105.106.3.207 use=5 exp=0 gpc0_rate(10000)=0 0x7faef2ec2d4c: key=105.106.6.240 use=66 exp=0 gpc0_rate(10000)=0 0x7faef406a9ac: key=105.107.105.77 use=15 exp=0 gpc0_rate(10000)=0 0x7faef36b2cdc: key=105.107.117.186 use=7 exp=0 gpc0_rate(10000)=0 0x7faef412a71c: key=105.107.132.95 use=23 exp=0 gpc0_rate(10000)=0 0x7faef3f97d4c: key=105.107.160.117 use=69 exp=0 gpc0_rate(10000)=0 0x7faef41afaec: key=105.107.32.8 use=142 exp=0 gpc0_rate(10000)=0 0x7faef408676c: key=105.107.41.220 use=21 exp=0 gpc0_rate(10000)=0 0x7faef4051c9c: key=105.107.45.99 use=59 exp=0 gpc0_rate(10000)=0 0x7faef216001c: key=105.107.85.185 use=25 exp=0 gpc0_rate(10000)=0 0x7faef2edc4cc: key=105.107.87.236 use=25 exp=0 gpc0_rate(10000)=0 Some of the newer entries which come in do time out as expected. A restart of haproxy clears the table but nothing else will do it Any ideas? Gernot ----- Original Message ----- > From: "Willy Tarreau" <[email protected]> > To: "Gernot Pörner" <[email protected]> > Cc: "haproxy" <[email protected]> > Sent: Thursday, October 27, 2016 8:08:43 PM > Subject: Re: stick table entries in use > On Thu, Oct 27, 2016 at 12:29:56PM +0000, Gernot Pörner wrote: >> > That's expected. "use>0" means that some sessions still track this entry, >> > so it cannot be removed. The value represents the number of trackers >> > still on it. It is possible that you're having some persistent HTTP >> > connections bound to it and that you may have to wait for the idle >> > timeout to expire. >> >> As far as I can see there are no related sessions anymore. These entries >> are in there since 2 days now, when the attacker stopped hammering on our >> api. > > OK so it seems like something odd happened. > >> Our longest timeout in haproxy anywhere is 180 seconds. Or did you mean >> the tcp timeouts of the kernel? These are currently set to 600s >> (/proc/sys/net/ipv4/tcp_keepalive_time). > > No I was talking about haproxy's timeouts. If all sessions are closed > it's abnormal. > >> > If you're seeing the same ones last much longer >> > than the request timeout, there might be another issue though. Then >> > issuing "show sess all" on the stats socket to get a dump before >> > restarting may possibly help. >> >> There are only session of the last couple of minutes in there. Are there >> any other ways/places I should look? > > No, that was the right place. > >> With lsof I also can't see any related TCP sessions anymore. > > So in the end they disappeared on both sides, yet there's a refcount > issue. I'd call that a bug. I don't remember, do you use peers to > synchronize your stick tables ? Nothing special, I'm just trying to > bisect the problem. > >> > By the way if you want to use stick-tables for such usages, I strongly >> > recommend trying 1.6 which is richer there. You can for example track in >> > http-request rules, and you can exchange the stick counters between peers. >> >> I plan on doing that as soon as possible. > > OK. As a rule of thumb, don't change your config and your version at the > same time. That way if you were to hit a regression (always to be expected > once you've met a problem nobody has ever met), it's easier to roll back > to the previous version. > > Cheers, > Willy
