Tuning varnish for high load
We are running varnish (default config) in production and are tuning it to handle high load. We have 4 load balanced varnish servers (64 bit RedHat 5.1, 4G of RAM, everything on a gigabit network). We find that under high load varnish is unable to serve pages or the keepalive pages to our load balancer (so it takes the servers offline). Running the following load test on a 65K, uncompressed static page from a single load testing server stops the 4 varnish servers from serving static web pages: httperf --verbose --hog --timeout=25 --server=www.ourdomain.com --port=80 --uri /pagetotest.html --rate=1000 --send-buffer=2000 --recv-buffer=2000 --num-conns=50 --num-calls=1 What tuning recommendations do you have for varnish to help it handle high load? Mark ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish-misc Digest, Vol 23, Issue 25
I guess its the same problem that ive had, and there is a patch regarding it; http://varnish.projects.linpro.no/ticket/197 -- Message: 1 Date: Fri, 29 Feb 2008 10:23:21 +0100 From: Fredrik Nygren [EMAIL PROTECTED] Subject: Blank pages with HTTP/1.0 To: varnish-misc@projects.linpro.no Cc: Stina Lange [EMAIL PROTECTED], Fredrik Sj?dell [EMAIL PROTECTED],Petri Luomala [EMAIL PROTECTED], Peter Granstr?m [EMAIL PROTECTED] Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Hi, I have a couple of servers with Varnish 1.1.2-5 installed. Some visitors has reported a blank page when they are visiting us. What I can se from our logs the problem seems to be related to the visitors HTTP protocol version. Requests with HTTP/1.0 gets a blank page but HTTP/1.1 not. Our Apache logs shows this: [28/Feb/2008:10:32:14 +0100] GET /index.jsp HTTP/1.1 200 145883 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) [28/Feb/2008:10:39:53 +0100] GET /index.jsp HTTP/1.1 200 145863 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) [28/Feb/2008:10:44:18 +0100] GET /index.jsp HTTP/1.1 200 145886 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:32 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:43 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:44 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:45 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:47 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:45:04 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:11:21:09 +0100] GET /index.jsp HTTP/1.1 200 146633 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:12:31:46 +0100] GET /index.jsp HTTP/1.1 200 146205 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) Responses with 17491 bytes in return are blank page views. From our varnishlog I read the following: 134 SessionOpen c 134.25.0.133 14373 134 ReqStart c 134.25.0.133 14373 2174456149 134 RxRequestc GET 134 RxURLc / 134 RxProtocol c HTTP/1.0 134 RxHeader c Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */* 134 RxHeader c Accept-Language: sv 134 RxHeader c UA-CPU: x86 134 RxHeader c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) 134 RxHeader c Host: www.smp.se 134 RxHeader c Cookie: __utma=147067189.81457932.1204189951.1204189951.1204189951.1; __utmb=147067189; __utmz=147067189.1204189951.1.1.utmccn=(direct)| utmcsr=(direct)|utmcmd=(none); InsightXE022004=216172787026292393 134 RxHeader c X-NovINet: v1.2 134 RxHeader c Via: 1.0 portos.sr.se:8080 (squid/2.6.STABLE18) 134 RxHeader c Cache-Control: max-age=259200 134 VCL_call c recv lookup 134 VCL_call c hash hash 134 HitPass c 2174455371 134 VCL_call c pass pass 134 Backend c 123 smp 134 ObjProtocol c HTTP/1.1 134 ObjStatusc 200 134 ObjResponse c OK 134 ObjHeaderc Date: Thu, 28 Feb 2008 09:22:29 GMT 134 ObjHeaderc Server: Apache-Coyote/1.1 134 ObjHeaderc Content-Type: text/html;charset=ISO-8859-1 134 ObjHeaderc Content-Language: sv-SE 134 ObjHeaderc Set-Cookie: JSESSIONID=4CF19E800052C32EC682743E8D6B8920.sehan9078gota8009; Path=/ 134 TTL c 2174456149 RFC 120 1204190549 1204190549 0 0 0 134 VCL_call c fetch pass 134 Length c 0 134 VCL_call c deliver deliver 134 TxProtocol c HTTP/1.1 134 TxStatus c 200 134 TxResponse c OK 134 TxHeader c Server: Apache-Coyote/1.1 134 TxHeader c Content-Type: text/html;charset=ISO-8859-1 134 TxHeader c Content-Language: sv-SE 134 TxHeader c Set-Cookie: JSESSIONID=4CF19E800052C32EC682743E8D6B8920.sehan9078gota8009; Path=/ 134 TxHeader c Date: Thu, 28 Feb 2008 09:22:29 GMT 134 TxHeader c X-Varnish: 2174456149 134 TxHeader c Age: 0 134 TxHeader c Via: 1.1 varnish 134 TxHeader c Connection: close 134 ReqEnd c 2174456149 1204190549.767153978 1204190549.773138046 0.30041 0.005959034 0.25034 I have searched the mailinglist and found this thread which seems to look like our problem but I'm not sure it's the same problem: http://projects.linpro.no/pipermail/varnish-misc/2008-February/001349.html Is there a known problem
Re: VCL purge
Anyone? What I want is when a client sends e.g Pragma: no-cache, I want warnish to fetch from backend and insert, not only do a pass. -- André Øien Langvand [EMAIL PROTECTED] - PGP: 0x7B1E3468 Systemadministrator - Idium AS - http://www.idium.no André Øien Langvand wrote: Hi, A few days ago, DES wrote the following in reply to another mail: If the point of this is to allow the client to Shift-Reload, you should purge the URL to force a reload from the backend, instead of piping the request. And the question is, how can I do this? I'm having problems figuring out what the purge_hash() needs to look like. I guess being allowed to use purge_hash(req.hash) would be the easiest, but that doesn't work. I've tried to use several combinations of req.url and req.http.host, but can't find a working one. purge_hash() is needed because of virtual hosts. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish-misc Digest, Vol 23, Issue 25
Alright. Wonder then if somebody has any rpm's to share for Redhat EL4 i386 with the patch included? Regards fredrik / On 29 feb 2008, at 12.25, Anders Vännman wrote: I guess its the same problem that ive had, and there is a patch regarding it; http://varnish.projects.linpro.no/ticket/197 -- Message: 1 Date: Fri, 29 Feb 2008 10:23:21 +0100 From: Fredrik Nygren [EMAIL PROTECTED] Subject: Blank pages with HTTP/1.0 To: varnish-misc@projects.linpro.no Cc: Stina Lange [EMAIL PROTECTED], Fredrik Sj?dell [EMAIL PROTECTED],Petri Luomala [EMAIL PROTECTED], Peter Granstr?m [EMAIL PROTECTED] Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Hi, I have a couple of servers with Varnish 1.1.2-5 installed. Some visitors has reported a blank page when they are visiting us. What I can se from our logs the problem seems to be related to the visitors HTTP protocol version. Requests with HTTP/1.0 gets a blank page but HTTP/1.1 not. Our Apache logs shows this: [28/Feb/2008:10:32:14 +0100] GET /index.jsp HTTP/1.1 200 145883 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) [28/Feb/2008:10:39:53 +0100] GET /index.jsp HTTP/1.1 200 145863 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) [28/Feb/2008:10:44:18 +0100] GET /index.jsp HTTP/1.1 200 145886 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:32 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:43 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:44 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:45 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:44:47 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:10:45:04 +0100] GET /index.jsp HTTP/1.0 200 17491 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:11:21:09 +0100] GET /index.jsp HTTP/1.1 200 146633 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) [28/Feb/2008:12:31:46 +0100] GET /index.jsp HTTP/1.1 200 146205 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) Responses with 17491 bytes in return are blank page views. From our varnishlog I read the following: 134 SessionOpen c 134.25.0.133 14373 134 ReqStart c 134.25.0.133 14373 2174456149 134 RxRequestc GET 134 RxURLc / 134 RxProtocol c HTTP/1.0 134 RxHeader c Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */* 134 RxHeader c Accept-Language: sv 134 RxHeader c UA-CPU: x86 134 RxHeader c User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) 134 RxHeader c Host: www.smp.se 134 RxHeader c Cookie: __utma=147067189.81457932.1204189951.1204189951.1204189951.1; __utmb=147067189; __utmz=147067189.1204189951.1.1.utmccn=(direct)| utmcsr=(direct)|utmcmd=(none); InsightXE022004=216172787026292393 134 RxHeader c X-NovINet: v1.2 134 RxHeader c Via: 1.0 portos.sr.se:8080 (squid/2.6.STABLE18) 134 RxHeader c Cache-Control: max-age=259200 134 VCL_call c recv lookup 134 VCL_call c hash hash 134 HitPass c 2174455371 134 VCL_call c pass pass 134 Backend c 123 smp 134 ObjProtocol c HTTP/1.1 134 ObjStatusc 200 134 ObjResponse c OK 134 ObjHeaderc Date: Thu, 28 Feb 2008 09:22:29 GMT 134 ObjHeaderc Server: Apache-Coyote/1.1 134 ObjHeaderc Content-Type: text/html;charset=ISO-8859-1 134 ObjHeaderc Content-Language: sv-SE 134 ObjHeaderc Set-Cookie: JSESSIONID=4CF19E800052C32EC682743E8D6B8920.sehan9078gota8009; Path=/ 134 TTL c 2174456149 RFC 120 1204190549 1204190549 0 0 0 134 VCL_call c fetch pass 134 Length c 0 134 VCL_call c deliver deliver 134 TxProtocol c HTTP/1.1 134 TxStatus c 200 134 TxResponse c OK 134 TxHeader c Server: Apache-Coyote/1.1 134 TxHeader c Content-Type: text/html;charset=ISO-8859-1 134 TxHeader c Content-Language: sv-SE 134 TxHeader c Set-Cookie: JSESSIONID=4CF19E800052C32EC682743E8D6B8920.sehan9078gota8009; Path=/ 134 TxHeader c Date: Thu, 28 Feb 2008 09:22:29 GMT 134 TxHeader c X-Varnish: 2174456149 134 TxHeader c Age: 0 134 TxHeader c Via: 1.1 varnish 134 TxHeader c Connection: close 134 ReqEnd c 2174456149 1204190549.767153978 1204190549.773138046 0.30041 0.005959034 0.25034 I have searched the mailinglist and found this thread which seems to look like our problem
Re: varnish-misc Digest, Vol 23, Issue 25
* Fredrik Nygren Alright. Wonder then if somebody has any rpm's to share for Redhat EL4 i386 with the patch included? Post a request with a short recap of the discussion to the varnish-dist list, and I'll consider it. Ingvar -- Buddha wears an iPod ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Child dying with Too many open files
I'm still getting the Too many open files error on the child. $ sudo sysctl -a | grep file fs.file-max = 131072 NFILES is also set to 131072. Any ideas? -Andy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael S. Fischer Sent: Thursday, February 28, 2008 3:51 PM To: Andrew Knapp Cc: varnish-misc@projects.linpro.no Subject: Re: Child dying with Too many open files I can't help but wonder if you'd set it too high. What happens when you set NFILES and fs.file-max both to 131072? I've tested that as a known good value. --Michael On Thu, Feb 28, 2008 at 2:58 PM, Andrew Knapp [EMAIL PROTECTED] wrote: Yup, it is. Here's some output: $ ps auxwww | grep varnish root 12036 0.0 0.0 27704 648 ?Ss 14:54 0:00 /usr/sbin/varnishd -a :80 -f /etc/varnish/photo.vcl -T internalip:6082 -t 120 -w 10,700,30 -s file,/c01/varnish/varnish_storage.bin,12G -u varnish -g varnish -P /var/run/varnish.pid varnish 12037 1.2 0.4 13119108 39936 ? Sl 14:54 0:00 /usr/sbin/varnishd -a :80 -f /etc/varnish/photo.vcl -T internalip:6082 -t 120 -w 10,700,30 -s file,/c01/varnish/varnish_storage.bin,12G -u varnish -g varnish -P /var/run/varnish.pid -Andy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael S. Fischer Sent: Thursday, February 28, 2008 1:57 PM To: Andrew Knapp Cc: varnish-misc@projects.linpro.no Subject: Re: Child dying with Too many open files Is varnishd being started as root? (even if it drops privileges later) Only root can have 1024 file descriptors open, to my knowledge. --Michael On Thu, Feb 28, 2008 at 11:48 AM, Andrew Knapp [EMAIL PROTECTED] wrote: Didn't really get a answer to this, so I'm trying again. I've done some testing with the NFILES variable, and I keep getting the same error as before (Too many open files). I've also verified that the limit is actually being applied by putting a ulimit -a in the /etc/init.d/varnish script. Anyone have any ideas? I'm running the 1.1.2-5 rpms from sf.net on Centos 5.1. Thanks, Andy -Original Message- From: [EMAIL PROTECTED] [mailto:varnish- misc- [EMAIL PROTECTED] On Behalf Of Andrew Knapp Sent: Wednesday, February 20, 2008 5:52 PM To: Michael S. Fischer Cc: varnish-misc@projects.linpro.no Subject: RE: Child dying with Too many open files Here's the output: $ sysctl fs.file-max fs.file-max = 767606 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael S. Fischer Sent: Wednesday, February 20, 2008 5:48 PM To: Andrew Knapp Cc: varnish-misc@projects.linpro.no Subject: Re: Child dying with Too many open files Does 'sysctl fs.file-max' say? It should be = the ulimit. --Michael On Wed, Feb 20, 2008 at 4:04 PM, Andrew Knapp [EMAIL PROTECTED] wrote: Hello, I'm getting this error when running varnishd: Child said (2, 15369): Assert error in wrk_thread(), cache_pool.c line 217: Condition((pipe(w-pipe)) == 0) not true. errno = 24 (Too many open files) Cache child died pid=15369 status=0x6 uname -a: Linux hostname 2.6.18-53.1.4.el5 #1 SMP Fri Nov 30 00:45:55 EST 2007 x86_64 x86_64 x86_64 GNU/Linux command used to start varnish: /usr/sbin/varnishd -d -d -a :80 -f /etc/varnish/photo.vcl -T internalIP:6082 -t 120 -w 10,700,30 -s file,/c01/varnish/varnish_storage.bin,12G -u varnish -g varnish -P /var/run/varnish.pid I have NFILES=27 set in /etc/sysconfig/varnish. Do I just need to up that value? Thanks, Andy ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no
