Re: Dropped connections with tcp_tw_recycle=1
Hi Sven, I don't know the basis precise for it, but I can vouch for the fact that tcp_tw_recycle is incompatible with NAT on the server side. I would guess it is because the NAT gateway keeps a connection tracking list and is unhappy that the webserver is trying to reuse the same ip:port hash whilst it is registered in TIME_WAIT mode. There was a discussion of this previously: http://projects.linpro.no/pipermail/varnish-misc/2009-April/002764.html As you say tw_reuse works OK with NAT. Cheers, Nick. Sven Ulland wrote: I was recently debugging an issue where several clients experienced sporadic problems connecting to a website cached by varnish. Every now and then (say, something like every 20-50th TCP connection) would time out, or sometimes take a few SYNs before being accepted. Here's a typical example. It's observed at the spot marked 'X' in this network structure from the client network's perspective: [clients] - [NAT gateway] - [bridge firewall]X - [Internet] 0.00 natgw-extip varni-extip TCP 4292 http [SYN] TSV=283647429 TSER=0 WS=6 2.99 natgw-extip varni-extip TCP 4292 http [SYN] TSV=283648179 TSER=0 WS=6 8.99 natgw-extip varni-extip TCP 4292 http [SYN] TSV=283649679 TSER=0 WS=6 20.99 natgw-extip varni-extip TCP 4292 http [SYN] TSV=283652679 TSER=0 WS=6 44.99 natgw-extip varni-extip TCP 4292 http [SYN] TSV=283658679 TSER=0 WS=6 93.00 natgw-extip varni-extip TCP 4292 http [SYN] TSV=283670679 TSER=0 WS=6 93.00 varni-extip natgw-extip TCP http 4292 [SYN, ACK] TSV=2342207123 TSER=283670679 Note: The NAT gateway didn't do port translation here. Also, the timestamp values were not touched by the NAT gateway. The varnish node is behind LVS-TUN, but the LVS was not the culprit. After troubleshooting with the website owner, tcpdumping at various points on both sides, it was clear that the packets were reaching the varnish node, but except the last SYN, they were all dropped. This turned out to be because the varnish node had the tcp_tw_recycle sysctl enabled. Switching it off fixed the problem. The performance page on the varnish wiki features recommends Linux sysctl settings, including enabling tcp_tw_recycle, since april 2008. The recycle setting was removed from that page recently, but I would think there are a lot of installations around the world that have it enabled. I tried to figure out exactly how the recycling mechanism works, but the code is too complex to figure out without time or kernel network experience. Recycling was introduced by David Miller in 2.3.15, ref URL:http://lxr.linux.no/#linux-old+v2.3.15/net/ipv4/tcp_ipv4.c#L324 and e.g. URL:http://lxr.linux.no/#linux+v2.6.31/net/ipv4/tcp_ipv4.c#L1255. Do anyone have a good grasp on how it works, its connection to the RFC 1323 PAWS mechanism, and its claimed incompatibility with NAT (ref URL:http://lkml.org/lkml/2008/11/15/83)? When observing the same issue previously (dropped SYNs), I ditched tw_recycle in favour of tcp_tw_reuse, which doesn't seem to cause any problems (this was on a normal Apache system). It too is severely underdocumented, so I was hoping to shed some light on them both, and the exact circumstances where they are suitable for use. Sven ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc __ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email __ ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish User Group Meeting 2009-09
On Fri, Aug 07, 2009 at 12:08:38PM +0200, Tollef Fog Heen wrote: On September 21st and 22nd, the first Varnish User Group meeting will be held, in Canonical Ltd's offices in Millbank Tower, London, UK. Please see http://varnish.projects.linpro.no/wiki/200909UserGroupMeeting A little update, since we seem to have forgotten to mention it: We will begin at 09:00 London-time and keep going through the day. Canonical have been kind enough to lend us the meeting room we'll be using. See you there :) -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 pgpgI9QiwPkTW.pgp Description: PGP signature ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish User Group Meeting 2009-09
In message 20090920153645.gb5...@kjeks, Kristian Lyngstol writes: We will begin at 09:00 London-time and keep going through the day. Canonical have been kind enough to lend us the meeting room we'll be using. I will attempt to be there at 9, but I have still not figured out the details of getting from Cambridge to London out, working on that right now. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish User Group Meeting 2009-09
From Cambridge, take the train to London Kings Cross (approximately 50 minutes, runs every half hour). From Kings Cross take the Victoria Line (Underground) to Pimlico. Millbank tower is then a 1km walk. London journey planner: http://www.tfl.gov.uk/ National rail journey planner: http://www.nationalrail.co.uk/ Laurence 2009/9/20 Poul-Henning Kamp p...@phk.freebsd.dk: In message 20090920153645.gb5...@kjeks, Kristian Lyngstol writes: We will begin at 09:00 London-time and keep going through the day. Canonical have been kind enough to lend us the meeting room we'll be using. I will attempt to be there at 9, but I have still not figured out the details of getting from Cambridge to London out, working on that right now. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish User Group Meeting 2009-09
In message e95443d90909201009k7fddd1etc6e1d9e7900ab...@mail.gmail.com, Lauren ce Rowe writes: From Cambridge, take the train to London Kings Cross (approximately 50 minutes, runs every half hour). From Kings Cross take the Victoria Line (Underground) to Pimlico. Millbank tower is then a 1km walk. Yes, I have reached the same conclusion. I think I'll aim for the 0715 from cambridge, that should have me at Pimlico around 0830. Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 p...@freebsd.org | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Dropped connections with tcp_tw_recycle=1
On Sep 20, 2009, at 6:20 AM, Nils Goroll wrote: tcp_tw_recycle is incompatible with NAT on the server side ... because it will enforce the verification of TCP time stamps. Unless all clients behind a NAT (actually PAD/masquerading) device use identical timestamps (within a certain range), most of them will send invalid TCP timestamps so SYNs will get dropped. Since you seem pretty knowledgeable on the subject, can you please explain the difference between tcp_tw_reuse and tcp_tw_recycle? Thanks, --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
died signal=6 , panic and restart every few sec. to min.
Plz help, anyone have idea howto solve this problem ?
varnishd -a 0.0.0.0:80 -T 127.0.0.1:3500 -p client_http11=on -f vconf2 -s
file,/usr/local/varnish/cache.bin,80G -h classic,59 -p listen_depth=4096
-p obj_workspace=32768 -p sess_workspace=32768 -p send_timeout=327
I got this message from /var/log/messages
Sep 20 21:26:36 x2
varnishd[21933]http://varnish.projects.linpro.no/changeset/21933:
Child (21934) died signal=6 Sep 20 21:26:36 x2
varnishd[21933]http://varnish.projects.linpro.no/changeset/21933:
Child (21934) Panic message: Assert error in VRT_IP_string(), cache_vrt.c
line 693: Condition((p = WS_Alloc(sp-http-ws, len)) != 0) nlient =
211.74.185.119:2909, step = STP_RECV, handling = error, err_code = 503,
err_reason = (null), ws = 0x2abeb5926078 { overflow id = sess, {s,f,r,e} =
cname = { input, Default, }, }, },
Sep 20 21:26:36 x2
varnishd[21933]http://varnish.projects.linpro.no/changeset/21933:
child (21952) Started Sep 20 21:26:36 x2
varnishd[21933]http://varnish.projects.linpro.no/changeset/21933:
Child (21952) said Closed fds: 4 5 8 9 11 12 Sep 20 21:26:36 x2 varnishd
[21933] http://varnish.projects.linpro.no/changeset/21933: Child (21952)
said Child starts Sep 20 21:26:36 x2
varnishd[21933]http://varnish.projects.linpro.no/changeset/21933:
Child (21952) said managed to mmap 85899345920 bytes of 85899345920 Sep 20
21:26:36 x2 varnishd[21933]http://varnish.projects.linpro.no/changeset/21933:
Child (21952) said Ready Sep 20 21:28:10 x2
varnishd[21933]http://varnish.projects.linpro.no/changeset/21933:
Child (21952) died signal=6 Sep 20 21:28:10 x2
varnishd[21933]http://varnish.projects.linpro.no/changeset/21933:
Child (21952) Panic message: Assert error in WS_Release(), cache_ws.c line
170: Condition(bytes = ws-e - ws-f) not true. thread = (10:32759, step =
STP_RECV, handling = error, err_code = 503, err_reason = (null), ws =
0x2abeb5a65078 { id = sess, {s,f,r,e} = {0x2abeb5a65808+32738,+32
Default, }, }, },
Thanks alot
T W
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: died signal=6 , panic and restart every few sec. to min.
On Sun, Sep 20, 2009 at 3:29 PM, M L m...@tinwong.com wrote:
Plz help, anyone have idea howto solve this problem ?
varnishd -a 0.0.0.0:80 -T 127.0.0.1:3500 -p client_http11=on -f vconf2 -s
file,/usr/local/varnish/cache.bin,80G -h classic,59 -p listen_depth=4096
-p obj_workspace=32768 -p sess_workspace=32768 -p send_timeout=327
I got this message from /var/log/messages
Sep 20 21:26:36 x2 varnishd[21933]: Child (21934) died signal=6 Sep 20
21:26:36 x2 varnishd[21933]: Child (21934) Panic message: Assert error in
VRT_IP_string(), cache_vrt.c line 693: Condition((p = WS_Alloc(sp-http-ws,
len)) != 0) nlient = 211.74.185.119:2909, step = STP_RECV, handling = error,
err_code = 503, err_reason = (null), ws = 0x2abeb5926078 { overflow id =
sess, {s,f,r,e} = cname = { input, Default, }, }, },
Sep 20 21:26:36 x2 varnishd[21933]: child (21952) Started Sep 20 21:26:36 x2
varnishd[21933]: Child (21952) said Closed fds: 4 5 8 9 11 12 Sep 20
21:26:36 x2 varnishd[21933]: Child (21952) said Child starts Sep 20 21:26:36
x2 varnishd[21933]: Child (21952) said managed to mmap 85899345920 bytes of
85899345920 Sep 20 21:26:36 x2 varnishd[21933]: Child (21952) said Ready Sep
20 21:28:10 x2 varnishd[21933]: Child (21952) died signal=6 Sep 20 21:28:10
x2 varnishd[21933]: Child (21952) Panic message: Assert error in
WS_Release(), cache_ws.c line 170: Condition(bytes = ws-e - ws-f) not
true. thread = (10:32759, step = STP_RECV, handling = error, err_code = 503,
err_reason = (null), ws = 0x2abeb5a65078 { id = sess, {s,f,r,e} =
{0x2abeb5a65808+32738,+32 Default, }, }, },
what about your vcl file?
are you modifying the object in vcl_hit at all?
Thanks alot
T W
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: died signal=6 , panic and restart every few sec. to min.
Hi David
Thanks for reply i never modifying vcl_hit
my vcl
backend default {
.host = 10.0.0.5;
.port = 80;
.connect_timeout = 1s;
.first_byte_timeout = 5s;
.between_bytes_timeout = 2s;
}
backend srv1 {
.host = 10.0.0.5;
.port = 80;
.connect_timeout = 1s;
.first_byte_timeout = 5s;
.between_bytes_timeout = 2s;
}
backend srv2 {
.host = 10.0.0.5;
.port = 80;
.connect_timeout = 1s;
.first_byte_timeout = 5s;
.between_bytes_timeout = 2s;
}
acl purge {
localhost; 127.0.0.1;
}
#recv
sub vcl_recv {
if (req.http.host ~ www.foobar.com) {
set req.http.host = www.foobar.com;
if (req.restarts == 0) {
set req.backend = srv1;
} else if (req.restarts == 1) {
set req.backend = allhabit2;
}
}elseif ( req.http.host ~ www.zoobar.com) {
set req.http.host = www.zoobar.com;
if (req.restarts == 0) {
set req.backend = srv1;
} else if (req.restarts == 1) {
set req.backend = srv2;
}
}elseif ( req.http.host ~ www.yoobar.com) {
set req.http.host = www.yoobar.com;
if (req.restarts == 0) {
set req.backend = srv1;
} else if (req.restarts == 1) {
set req.backend = srv2;
}
}elseif ( req.http.host ~ 218.242.39.202) {
set req.http.host = 118.142.39.202;
if (req.restarts == 0) {
set req.backend = srv1;
} else if (req.restarts == 1) {
set req.backend = srv2;
}
}elseif ( req.http.host ~ 218.242.39.203) {
set req.http.host = 118.142.39.203;
if (req.restarts == 0) {
set req.backend = srv1;
} else if (req.restarts == 1) {
set req.backend = srv2;
}
}elseif ( req.http.host ~ 204.186.59.41) {
set req.http.host = 204.186.59.41;
if (req.restarts == 0) {
set req.backend = srv1;
} else if (req.restarts == 1) {
set req.backend = srv2;
}
}elseif ( req.http.host ~ 204.126.59.45) {
set req.http.host = 204.126.59.45;
if (req.restarts == 0) {
set req.backend = srv1;
} else if (req.restarts == 1) {
set req.backend = srv2;
}
}else{
error 401 Bad Domain;
}
#set req.grace = 30s;
# Add a unique header containing the client address
remove req.http.X-Forwarded-For;
setreq.http.X-Forwarded-For = client.ip;
# [...]
if (req.request == PURGE) {
if(!client.ip ~ purge) {
error 405 Not Allowed;
} lookup;}
#if (req.request != GET req.request != HEAD) {
#pipe;
#}
#if (req.request == POST) {
# pass;
# }
if (req.http.Expect) {
pipe;
}
if (req.request != GET
req.request != HEAD
req.request != PUT
req.request != POST
req.request != TRACE
req.request != OPTIONS
req.request != DELETE) {
/* Non-RFC2616 or CONNECT which is weird. */
pipe;
}
if (req.request != GET req.request != HEAD) {
/* We only deal with GET and HEAD by default */
pass;
}
if (req.http.Cache-Control ~ no-cache) {
pass;
}
if (req.http.Authenticate) {
pass;
}
#if (req.http.Cookie) {
#pass;
# }
if (req.url ~
\.(zip|ico|dat|torrent|png|gif|jpg|swf|css|js|bmp|bz2|tbz|mp3|ogg)$) {
unset req.http.cookie;
lookup;
#unset req.http.authenticate;
}
if (req.http.Accept-Encoding) {
if (req.url ~
\.(zip|ico|dat|torrent|png|gif|jpg|swf|css|js|bmp|bz2|tbz|mp3|ogg)$) {
# No point in compressing these
remove req.http.Accept-Encoding;
} elsif (req.http.Accept-Encoding ~ gzip) {
set req.http.Accept-Encoding = gzip;
} elsif (req.http.Accept-Encoding ~ deflate) {
set req.http.Accept-Encoding = deflate;
} else {
# unkown algorithm
remove req.http.Accept-Encoding;
}
}
} #end recv
sub vcl_hash {
set req.hash += req.url;
set req.hash += req.http.host;
#set req.hash += req.http.cookie;
#set req.hash += server.ip;
hash;
} #end hash
# sub vcl_hash {
# set req.hash += req.url;
# if (req.http.host) {
# set req.hash += req.http.host;
# } else {
# set req.hash += server.ip;
# }
# hash;
# }
#if (req.http.Accept-Encoding ~ gzip) {
#set req.hash += gzip;
#}
#else if (req.http.Accept-Encoding ~ deflate) {
#set req.hash += deflate;
#}
#hash;
#} #end hash
#sub vcl_hash {
#set req.hash += req.url;
#set req.hash += req.http.host;
#if (req.http.Accept-Encoding ~ gzip) {
#set req.hash += gzip;
#}
#else if (req.http.Accept-Encoding ~ deflate) {
#set req.hash += deflate;
#}
#}
# strip the cookie before the image is inserted into cache.
sub vcl_fetch {
#if (obj.status != 200 obj.status != 302) {
#restart;
#}
if(obj.http.Set-Cookie){
pass;
httpd asking for AUTH _twice_ when behind Varnish proxy ? works as expected without Varnish ...
hi,
i've just done a 1st migration from
apache2+mod_ssl
to
pound + varnish + apache2
using,
pound -V
Version 2.4.5
varnishd -V
varnishd (varnish-2.0.4)
httpd2 -V
Server version: Apache/2.2.13 (Linux/SUSE)
in my original apache/ssl config, i've httpd DIGEST Auth set up (atm)
on the web root. it works as expected.
now that i've switched to the pound/varnish/apache2 setup, Auth still
works -- but makes the request twice!
if i visit
https://www.mysite.com
i get an initial request for AUTH at my defined realm :443, then after
entering credentials there, the page paints -- and i get a second http
AUTH dialog for the _same_ realm, but at :8081. switch back to a
direct connect, and just the one AUTH dialog ...
my relevant configs are below ...
any ideas as to what's causing the double-AUTH request, and how to fix
it would be much appreciated!
thanks!
/etc/pound.cfg
ListenHTTP
Address xx.xx.xx.xx
Port80
Service
Redirect https://www.mysite.com;
End
End
ListenHTTPS
Address xx.xx.xx.xx
Port443
Cert/crypt/ssl/ssl.crt/combined.pem
Ciphers AES256-SHA:AES128-SHA
NoHTTPS11 2
Service
BackEnd
Address 127.0.0.1
Port8080
End
End
End
/etc/sysconfig/varnish
VARNISHD_PARAMS=-f /etc/varnish/vcl.conf -a 127.0.0.1:8080 -T
127.0.0.1:6082 -s file,/var/cache/varnish/varnish.bin,100M -n test
/etc/varnish/vcl.conf
# cp of /etc/varnish/default.vcl, except:
backend default {
.host = xx.xx.xx.xx;
.port = 8081;
}
/etc/apache2/vhosts.d/www.mysite.com
...
VirtualHost xx.xx.xx.xx:8081
...
DocumentRoot /svr/www/mysite
...
Directory /svr/www/mysite
Options +ExecCGI +FollowSymLinks +Indexes
DirectoryIndex index.html index.php
AuthType Digest
AuthName AUTH mysite
AuthDigestProvider file
AuthUserFile /crypt/wwwauth/.passwords.md5
AuthDigestDomain /
require valid-user
AddHandler fcgid-script .php
FCGIWrapper /usr/bin/php-cgi5 -d apc.shm_size=25 -c
/etc/php5/fastcgi/ .php
...
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
