Re: Dropped connections with tcp_tw_recycle=1

2009-09-20 Thread Nick Loman
Hi Sven,

I don't know the basis precise for it, but I can vouch for the fact that 
tcp_tw_recycle is incompatible with NAT on the server side. I would 
guess it is because the NAT gateway keeps a connection tracking list and 
is unhappy that the webserver is trying to reuse the same ip:port hash 
whilst it is registered in TIME_WAIT mode.

There was a discussion of this previously:
http://projects.linpro.no/pipermail/varnish-misc/2009-April/002764.html

As you say tw_reuse works OK with NAT.

Cheers,

Nick.


Sven Ulland wrote:
 I was recently debugging an issue where several clients experienced
 sporadic problems connecting to a website cached by varnish. Every now
 and then (say, something like every 20-50th TCP connection) would time
 out, or sometimes take a few SYNs before being accepted.

 Here's a typical example. It's observed at the spot marked 'X' in this
 network structure from the client network's perspective:

[clients] - [NAT gateway] - [bridge firewall]X - [Internet]

   0.00 natgw-extip varni-extip TCP 4292  http [SYN] TSV=283647429 
 TSER=0 WS=6
   2.99 natgw-extip varni-extip TCP 4292  http [SYN] TSV=283648179 
 TSER=0 WS=6
   8.99 natgw-extip varni-extip TCP 4292  http [SYN] TSV=283649679 
 TSER=0 WS=6
 20.99 natgw-extip varni-extip TCP 4292  http [SYN] TSV=283652679 TSER=0 
 WS=6
 44.99 natgw-extip varni-extip TCP 4292  http [SYN] TSV=283658679 TSER=0 
 WS=6
 93.00 natgw-extip varni-extip TCP 4292  http [SYN] TSV=283670679 TSER=0 
 WS=6
 93.00 varni-extip natgw-extip TCP http  4292 [SYN, ACK] TSV=2342207123 
 TSER=283670679

 Note: The NAT gateway didn't do port translation here. Also, the
 timestamp values were not touched by the NAT gateway. The varnish node
 is behind LVS-TUN, but the LVS was not the culprit.

 After troubleshooting with the website owner, tcpdumping at various
 points on both sides, it was clear that the packets were reaching the
 varnish node, but except the last SYN, they were all dropped. This
 turned out to be because the varnish node had the tcp_tw_recycle sysctl
 enabled. Switching it off fixed the problem.

 The performance page on the varnish wiki features recommends Linux
 sysctl settings, including enabling tcp_tw_recycle, since april 2008.
 The recycle setting was removed from that page recently, but I would
 think there are a lot of installations around the world that have it
 enabled.

 I tried to figure out exactly how the recycling mechanism works, but the
 code is too complex to figure out without time or kernel network
 experience. Recycling was introduced by David Miller in 2.3.15, ref
 URL:http://lxr.linux.no/#linux-old+v2.3.15/net/ipv4/tcp_ipv4.c#L324
 and e.g. URL:http://lxr.linux.no/#linux+v2.6.31/net/ipv4/tcp_ipv4.c#L1255.
 Do anyone have a good grasp on how it works, its connection to the RFC
 1323 PAWS mechanism, and its claimed incompatibility with NAT (ref
 URL:http://lkml.org/lkml/2008/11/15/83)?

 When observing the same issue previously (dropped SYNs), I ditched
 tw_recycle in favour of tcp_tw_reuse, which doesn't seem to cause any
 problems (this was on a normal Apache system). It too is severely
 underdocumented, so I was hoping to shed some light on them both, and
 the exact circumstances where they are suitable for use.

 Sven
 ___
 varnish-misc mailing list
 varnish-misc@projects.linpro.no
 http://projects.linpro.no/mailman/listinfo/varnish-misc

 __
 This email has been scanned by the MessageLabs Email Security System.
 For more information please visit http://www.messagelabs.com/email 
 __
   

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Apache DoS - is Varnish affected?

2009-06-19 Thread Nick Loman
I would guess that Varnish isn't affected by this, but does anyone know 
for sure? Does Varnish protect against this attack in all cases if you 
have Apache as your backend?

http://isc.sans.org/diary.html?storyid=6601

Many thanks,

Nick.

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Theoretical connections/second limit using Varnish

2009-04-30 Thread Nick Loman
Michael S. Fischer wrote:

 I've done that for a specific reason relating to backend PHP processes.
 
 I don't dispute your reasoning; my employer does this as well.  
 KeepAlive with Apache/PHP can be a recipe for resource starvation on 
 your origin servers.

Hi Michael,

Precisely, we only have perhaps 50 PHP children serving requests, so if 
these are kept open to serve idle keep-alive connections, that severely 
limits the numbers of dynamic page requests we can serve.

 I typically have thousands of connections in TIME_WAIT mode as a 
 result, which is expected, but I wonder what the solution could be if 
 I ever hit more connections than local ports available.
 
 I think SO_REUSEADDR is the answer - I'm somewhat surprised that Varnish 
 doesn't set it by default for the backend connections.

I've had a browse through the Varnish 2.0.4 source now and have 
convinced myself that SO_REUSEADDR isn't set. Can you think of any 
specific gotchas I might encounter if I added setsockopt() before the 
connect() when getting a backend connection?

Cheers,

Nick.
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Theoretical connections/second limit using Varnish

2009-04-29 Thread Nick Loman
Hi there,

Has anyone come to a satisfactory solution to the issue of running out 
of local port numbers when Varnish makes a connection to the backend server?

Under Linux, my understanding is the number of available port numbers 
can be increased to a maximum of 64511 by setting 
/proc/sys/net/ipv4/ip_local_port_range to 1024 - 65535.

Assuming sockets are left in TIME_WAIT for 60 seconds that would limit 
the number of backend connections Varnish can make to 64511/minute or 
1075/second.

It seems to be acceptable to reduce TIME_WAIT to perhaps 30 seconds, 
doubling that to 2150/second.

A solution often proposed is to use time wait recycling, or tw_reuse, 
but my understanding is that under Linux these settings are global and 
therefore can break NAT for user connections (all connections are 
conntracked and DNATted on our setup).

2150 requests/second is not an impossible number to achieve, especially 
with backend KeepAlive off.

Has Varnish got a solution to this problem which does not involve 
time-wait recycling? One thing I've thought of is perhaps SO_REUSEADDR 
is used or could be used when Varnish makes connections to the backend?

Regards,

Nick.
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Theoretical connections/second limit using Varnish

2009-04-29 Thread Nick Loman
Michael S. Fischer wrote:
 On Apr 29, 2009, at 9:22 AM, Poul-Henning Kamp wrote:
 
 In message 49f87de4.3040...@loman.net, Nick Loman writes:

 Has Varnish got a solution to this problem which does not involve
 time-wait recycling? One thing I've thought of is perhaps SO_REUSEADDR
 is used or could be used when Varnish makes connections to the backend?

 Varnish tries as hard as reasonable to reuse backend connections,
 so you should be able to get multiple requests per backend connection.

 If this is not the case for you, you should find out why backend 
 connections
 are not reused.

Hi Poul-Henning, Michael,

I've configured Apache with KeepAlive off, so I expect the TCP 
connection to be closed after each request and Varnish won't be able to 
use it.

I've done that for a specific reason relating to backend PHP processes.

Is that what you mean?

I typically have thousands of connections in TIME_WAIT mode as a result, 
which is expected, but I wonder what the solution could be if I ever hit 
more connections than local ports available.

If I turned Keep Alive on with Apache, could Varnish multiplex different 
  requests from different TCP sockets to the same backend connection?

 The OP said he turned backend Keep-Alive off.  That's his problem.

Right, exactly, but I want it this way assuming that leaving it on means 
that each user connection translates to one backend connection to Apache 
- not desirable - but perhaps this is not how Varnish operates?

Cheers,

Nick.


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Weird log entries

2009-02-09 Thread Nick Loman
Alecs Henry wrote:

 Those are coupled with:
 127.0.0.1 - - [09/Feb/2009:19:39:46 +] (null) (null) (null) 200 
 39678 - -
 I can see an object in the page that has that size (image) -- through 
 firebug, but the object didn't load into the browser until I hit reload.

I've seen log entries like this for HTTP 1.0 requests (using Varnish 
1.2), as that version of Varnish did not have support for parsing HTTP 
1.0 headers.

Cheers,

Nick.



___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish keep-alive problem

2009-01-06 Thread Nick Loman
Tollef Fog Heen wrote:

 | Varnish is a very good web accelerator, and i find it support KeepAlive.
 | The default keep-alive is on. I want to turn off the keep-alive, but i 
 don't know how to do it.
 | Please tell me how to turn off the keep-alive.
 
 I don't believe that's possible out of the box.  Why do you want to turn
 off keepalive?

You can turn it off on the backend web-server.

Alternatively, you could try adding: Connection: close to the returned 
headers in the VCL which should encourage the client to close the 
connection.

Cheers,

Nick.



___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Ticket #250 (POST error when using Opera)

2008-10-11 Thread Nick Loman
Poul-Henning Kamp wrote:
 In message [EMAIL PROTECTED], Nick Loman writes:
 Hi there

 Just to update you on my experience with Varnish 1.1.2, [...]
 
 At this point we _really_ urge everybody to upgrade to Varnish 2.0.
 
 The release candidate 1 was released this week, and we are ready to
 cut the release now.

Hi Poul-Henning,

I started on Varnish 2.0-rc1, which was 99% great, but we experienced a 
problem which was unfortunately a show-stopper. See my post on the 9th 
October, Mac connection problem with Varnish 2.0-rc1. If it wasn't for 
that, I'd be using Varnish 2.0 on our production sites right now.

Regards,

Nick.


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Ticket #250 (POST error when using Opera)

2008-10-11 Thread Nick Loman
Poul-Henning Kamp wrote:
 In message [EMAIL PROTECTED], Nick Loman writes:
 
 I started on Varnish 2.0-rc1, which was 99% great, but we experienced a 
 problem which was unfortunately a show-stopper. See my post on the 9th 
 October, Mac connection problem with Varnish 2.0-rc1. If it wasn't for 
 that, I'd be using Varnish 2.0 on our production sites right now.
 
 I looked at that and didn't see anything in the varnishlog output
 that indicated a problem.

Nor did I! ;)

 I have no idea what the safari error you quoted in the message mean...

No, nor do I.  But it meant that a user couldn't use the site properly, 
and when I downgraded to Varnish 1.1.2 it worked absolutely fine. This 
was Varnish 2.0-rc1 with totally default settings, so I couldn't blame a 
dodgy VCL.

Where should I go from here? I could try the latest trunk, I guess?

Regards,

Nick.
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Mac connection problem with Varnish 2.0-rc1

2008-10-09 Thread Nick Loman
Hi there,

A client using Mac version of Safari complains of intermittent errors 
kCFErrorDomainCFNetwork error 302, perhaps every other page, on 
dynamically generated pages.

This is a stock install of varnish-2.0-rc1, with default.vcl and default 
configuration values. Everything works perfectly on Safari PC, Firefox 
on Mac/PC, etc. Downgrading to varnish 1.1.2 solves the problem with 
same hardware/software/config.

I've attached the varnish log for the problematic part.

Any help appreciated,

Cheers,

Nick.


 9 SessionOpen  c HIDDEN_IP_ADDRESS 30775 0.0.0.0:8080
 0 CLI  - Rd ping
 0 CLI  - Wr 0 200 PONG 1223567243 1.0
 9 ReqStart c HIDDEN_IP_ADDRESS 30775 363533487
 9 RxRequestc GET
 9 RxURLc /..url deleted
 9 RxProtocol   c HTTP/1.1
 9 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac 
OS X 10_5_3; en-us) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.1 
Safari/525.20
 9 RxHeader c Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
 9 RxHeader c Accept-Language: en-us
 9 RxHeader c Accept-Encoding: gzip, deflate
 9 RxHeader c Cookie: egtv_seen[5462]=yes; egtv_seen[5383]=yes; 
egtv_seen[5466]=yes; egtv_seen[5494]=yes; egtv_seen[5577]=yes; 
egtv_seen[5774]=yes; egtv_seen[5837]=yes; egtv_seen[5847]=yes; 
egtv_seen[5844]=yes; egtv_seen[5898]=yes; egtv_seen[5928]=yes; 
egtv_seen[5957]=
 9 RxHeader c Connection: keep-alive
 9 RxHeader c Host: www.myhost.com
 9 VCL_call c recv
 9 VCL_return   c lookup
 9 VCL_call c hash
 9 VCL_return   c hash
 9 VCL_call c miss
 9 VCL_return   c fetch
10 BackendClose b default
10 BackendOpen  b default 127.0.0.1 55130 127.0.0.1 80
 9 Backend  c 10 default default
10 TxRequestb GET
10 TxURLb 
/forum_threads.php?category_id=group-threadsview=groups
10 TxProtocol   b HTTP/1.1
10 TxHeader b User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac 
OS X 10_5_3; en-us) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.1 
Safari/525.20
10 TxHeader b Accept: 
text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
10 TxHeader b Accept-Language: en-us
10 TxHeader b Accept-Encoding: gzip, deflate
  .. Cookie line deleted
10 TxHeader b Host: www.myhost.com
10 TxHeader b X-Varnish: 363533487
10 TxHeader b X-Forwarded-For: HIDDEN_IP_ADDRESS
10 RxProtocol   b HTTP/1.1
10 RxStatus b 200
10 RxResponse   b OK
10 RxHeader b Date: Thu, 09 Oct 2008 15:47:23 GMT
10 RxHeader b Server: Apache/2.2.9 (Unix)
10 RxHeader b X-Powered-By: PHP/5.2.6-pl7-gentoo
10 RxHeader b Expires: Mon, 26 Jul 1997 05:00:00 GMT
10 RxHeader b Cache-Control: max-age=0
10 RxHeader b Pragma: no-cache
  .. 12 Set-Cookie lines deleted
10 RxHeader b Last-Modified: Thu, 09 Oct 2008 15:47:24 GMT
10 RxHeader b Vary: Accept-Encoding
10 RxHeader b Content-Encoding: gzip
10 RxHeader b Content-Length: 8956
10 RxHeader b Content-Type: text/html; charset=ISO-8859-1
 9 ObjProtocol  c HTTP/1.1
 9 ObjStatusc 200
 9 ObjResponse  c OK
 9 ObjHeaderc Date: Thu, 09 Oct 2008 15:47:23 GMT
 9 ObjHeaderc Server: Apache/2.2.9 (Unix)
 9 ObjHeaderc X-Powered-By: PHP/5.2.6-pl7-gentoo
 9 ObjHeaderc Expires: Mon, 26 Jul 1997 05:00:00 GMT
 9 ObjHeaderc Cache-Control: max-age=0
 9 ObjHeaderc Pragma: no-cache
  .. 12 Set-Cookie lines deleted
 9 ObjHeaderc Last-Modified: Thu, 09 Oct 2008 15:47:24 GMT
 9 ObjHeaderc Vary: Accept-Encoding
 9 ObjHeaderc Content-Encoding: gzip
 9 ObjHeaderc Content-Type: text/html; charset=ISO-8859-1
10 BackendReuse b default
 9 TTL  c 363533487 RFC 0 1223567244 0 0 0 0
 9 VCL_call c fetch
 9 VCL_return   c pass
 9 Length   c 8956
 9 VCL_call c deliver
 9 VCL_return   c deliver
 9 TxProtocol   c HTTP/1.1
 9 TxStatus c 200
 9 TxResponse   c OK
 9 TxHeader c Server: Apache/2.2.9 (Unix)
 9 TxHeader c X-Powered-By: PHP/5.2.6-pl7-gentoo
 9 TxHeader c Expires: Mon, 26 Jul 1997 05:00:00 GMT
 9 TxHeader c Cache-Control: max-age=0
 9 TxHeader c Pragma: no-cache
 9 TxHeader c Last-Modified: Thu, 09 Oct 2008 15:47:24 GMT
 9 TxHeader c Vary: Accept-Encoding
 9 TxHeader c Content-Encoding: gzip
 9 TxHeader c Content-Type: text/html; charset=ISO-8859-1
 9 TxHeader c Content-Length: 8956
 9 TxHeader c Date: Thu, 09 Oct 2008 15:47:24 GMT
 9 TxHeader c X-Varnish: 363533487
 9 TxHeader c Age: 0
 9 TxHeader c Via: 1.1 varnish
 9 TxHeader c Connection: keep-alive
 9 ReqEnd   c 363533487 1223567243.957491398 

Keep-Alive acceleration, is this possible?

2008-09-25 Thread Nick Loman
Hi there,

On our platform we have had to disable Keep-Alive support on our 
Apache/FastCGI/PHP setup because it holds open too many backend 
processes under load even with KeepAliveTimeout set low.

I am looking at Varnish as a web accelerator (it looks great!), and I 
wonder if it is possible to get the best of both worlds, i.e. the user 
still gets KeepAlive support, but the backends server one script per 
connection.

I was thinking it might be possible to strip out the Connection: Close 
header returned by Apache, hopefully resulting in the client keeping its 
connection to Varnish. My question is: would Varnish realise that the 
backend connection has been closed but still keep the client connection 
open so that that future requests could be made on that same connection?

Would this have any adverse side-effects?

Grateful for your help

Nick

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Keep-Alive acceleration, is this possible?

2008-09-25 Thread Nick Loman
Poul-Henning Kamp wrote:

 I was thinking it might be possible to strip out the Connection: Close 
 header returned by Apache, [...]
 
 You don't need to do anything.
 
 Connection: is a hop-by-hop header, so Varnish already deletes it before
 sending the reply to the client

Perfect!

Many thanks,

Nick
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc