Hi Sven, I don't know the basis precise for it, but I can vouch for the fact that tcp_tw_recycle is incompatible with NAT on the server side. I would guess it is because the NAT gateway keeps a connection tracking list and is unhappy that the webserver is trying to reuse the same ip:port hash whilst it is registered in TIME_WAIT mode.
There was a discussion of this previously: http://projects.linpro.no/pipermail/varnish-misc/2009-April/002764.html As you say tw_reuse works OK with NAT. Cheers, Nick. Sven Ulland wrote: > I was recently debugging an issue where several clients experienced > sporadic problems connecting to a website cached by varnish. Every now > and then (say, something like every 20-50th TCP connection) would time > out, or sometimes take a few SYNs before being accepted. > > Here's a typical example. It's observed at the spot marked 'X' in this > network structure from the client network's perspective: > > [clients] -> [NAT gateway] -> [bridge firewall]X -> [Internet] > > 0.00 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283647429 > TSER=0 WS=6 > 2.99 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283648179 > TSER=0 WS=6 > 8.99 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283649679 > TSER=0 WS=6 > 20.99 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283652679 TSER=0 > WS=6 > 44.99 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283658679 TSER=0 > WS=6 > 93.00 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283670679 TSER=0 > WS=6 > 93.00 varni-extip natgw-extip TCP http > 4292 [SYN, ACK] TSV=2342207123 > TSER=283670679 > > Note: The NAT gateway didn't do port translation here. Also, the > timestamp values were not touched by the NAT gateway. The varnish node > is behind LVS-TUN, but the LVS was not the culprit. > > After troubleshooting with the website owner, tcpdumping at various > points on both sides, it was clear that the packets were reaching the > varnish node, but except the last SYN, they were all dropped. This > turned out to be because the varnish node had the tcp_tw_recycle sysctl > enabled. Switching it off fixed the problem. > > The performance page on the varnish wiki features recommends Linux > sysctl settings, including enabling tcp_tw_recycle, since april 2008. > The recycle setting was removed from that page recently, but I would > think there are a lot of installations around the world that have it > enabled. > > I tried to figure out exactly how the recycling mechanism works, but the > code is too complex to figure out without time or kernel network > experience. Recycling was introduced by David Miller in 2.3.15, ref > <URL:http://lxr.linux.no/#linux-old+v2.3.15/net/ipv4/tcp_ipv4.c#L324> > and e.g. <URL:http://lxr.linux.no/#linux+v2.6.31/net/ipv4/tcp_ipv4.c#L1255>. > Do anyone have a good grasp on how it works, its connection to the RFC > 1323 PAWS mechanism, and its claimed incompatibility with NAT (ref > <URL:http://lkml.org/lkml/2008/11/15/83>)? > > When observing the same issue previously (dropped SYNs), I ditched > tw_recycle in favour of tcp_tw_reuse, which doesn't seem to cause any > problems (this was on a normal Apache system). It too is severely > underdocumented, so I was hoping to shed some light on them both, and > the exact circumstances where they are suitable for use. > > Sven > _______________________________________________ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc > > ______________________________________________________________________ > This email has been scanned by the MessageLabs Email Security System. > For more information please visit http://www.messagelabs.com/email > ______________________________________________________________________ > _______________________________________________ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc