Re: Varnish 2.0.3 consuming excessive memory
hi darryl i had a simlar problem (varnish child process consuming lots of memory) a week ago (see the thread "make varnish don't start a subprocess") the solution for my problem seems to be to use a high-enough cache-size: starting varnish with -s file,"/tmp/storage",300M will make the child process use *about* 800M RAM (the RES column in top) after some hours. after that i needed to restart varnish to free some ram for my webserver. starting with a bigger cache -s file,"/tmp/storage",600M makes it using about 500M RAM (the RES column in top after 5 days of uptime) and happily coexist with zope ;-) maybe this helps you too. regards, fRiSi Am Mittwoch, 8. April 2009 05:05:36 schrieb Darryl Dixon - Winterhouse Consulting: > > Not that I have an answer, but I'd be curious to see the differences > > in 'pmap -x ' output for the different children. > > > > --Michael > > That's a good thought - I'd almost forgotten about pmap. Anyhow, the > 'normal' process is set up with a 2GB storage cache and pmap shows pretty > much what you would expect - the largest anonymous segment is 41MB, with > the vast majority of allocated memory booked against storage.cache (2GB). > The pmap for the 'runaway' process shows 256MB booked against > storage.cache, and then one of the an anonymous allocations is 2.3GB. For > example: > > > Normal process: > -- > 3896: /u01/app/varnish/sbin/varnishd -a web-cms1.prd.internal:8080 > web-cms.prd.internal:8080 -T web-cms1.prd.internal:18080 -t 108000 -f > /u01/data/prd/varnish/vcl.conf -s > file,/u01/data/prd/varnish/storage.cache,2048M -n > /u01/data/prd/varnish/state -P /u01/data/prd/varnish/varnishd.pid > Address Kbytes RSSAnon Locked Mode Mapping > 0040 312 - - - r-x-- varnishd > 0064d000 8 - - - rw--- varnishd > 0064f000 68 - - - rw---[ anon ] > 4000 4 - - - -[ anon ] > 40001000 10240 - - - rw---[ anon ] > [...snip...] > 2aaab090 2097152 - - - rw-s- storage.cache > 2aab3090 28 - - - r-x-- vcl.1P9zoqAU.so > 2aab309070002044 - - - - vcl.1P9zoqAU.so > 2aab30b06000 8 - - - rw--- vcl.1P9zoqAU.so > 2aab30c02056 - - - rw---[ anon ] > 2aab30f02056 - - - rw---[ anon ] > 2aab3120 41984 - - - rw---[ anon ] > 7fffae8f2000 88 - - - rw---[ stack ] > ff608192 - - - -[ anon ] > -- -- -- -- > total kB 2468512 - - - > -- > > > > Runaway process: > -- > 3940: /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080 > web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600 > -f /u01/data/prd-edit/varnish/vcl.conf -s > file,/u01/data/prd-edit/varnish/storage.cache,256M -n > /u01/data/prd-edit/varnish/state -P > /u01/data/prd-edit/varnish/varnishd.pid > Address Kbytes RSSAnon Locked Mode Mapping > 0040 312 - - - r-x-- varnishd > 0064d000 8 - - - rw--- varnishd > 0064f000 68 - - - rw---[ anon ] > 4000 4 - - - -[ anon ] > 40001000 10240 - - - rw---[ anon ] > [...snip...] > 2aaab090 262144 - - - rw-s- storage.cache > 2aaac090 28 - - - r-x-- vcl.1P9zoqAU.so > 2aaac09070002048 - - - - vcl.1P9zoqAU.so > 2aaac0b07000 8 - - - rw--- vcl.1P9zoqAU.so > 2aaac0c02056 - - - rw---[ anon ] > 2aaac0f02052 - - - rw---[ anon ] > 2aaac120 233476 - - - rw---[ anon ] > 2aaacf70 2337792 - - - rw---[ anon ] > 7fff4b619000 84 - - - rw---[ stack ] > ff608192 - - - -[ anon ] > -- -- -- -- > total kB 3121808 - - - > -- > > > Both processes have been running for the same amount of time - 1day > 17hours. > > regards, > Darryl Dixon > Winterhouse Consulting Ltd > http://www.winterhouseconsulting.com > > > On Apr 7, 2009, at 6:27 PM, Darryl Dixon - Winterhouse Consulting wrote: > >>> Hi All, > >>> > >>> I have an odd problem that I have only noticed happening s
Re: make varnish don't start a subprocess
hi des thanks for your reply. in order to get an idea how to optimize my setup and get a better understanding i'd like to recap to make sure i understood correctly: Am Dienstag, 24. März 2009 17:10:56 schrieb Dag-Erling Smørgrav: > If you're using a storage file, varnishd will use a relatively small > amount of memory + address space roughly equal to the size of the file. -s file,"/home/zope/parts/varnish/storage",500M will make the varnishd use around 750M RAM (see my comment on the next block), a storage file with 700M will result in around 950M RAM used. if that address space needs roughly the same amout of RAM as the storage file's size - and the cache is not persisted (lost when varnishd is restarted) - why do we need a storage file at all? > > whilst the file itself is exactly 500 megabyte in size, varnish > > uses 1115M memory - 44% of the total available 2,5GB RAM (1,5GB RAM, 1GB > > disk-swap) after 5 days of uptime. see `figure 1` below. > > It's not that simple. The file is mapped into varnishd's address space, > so the VIRT number you get from top includes the size of the file (as > well as the size of the program itself and all libraries it uses). RES > is how much of that is actually in RAM. The OS may swap unused parts of > the cache to the storage file, but never to the swap partition. using the sizes given in the RES column varnish uses 673+81=754M RAM:: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 1344 nobody15 0 1349m 673m 133m S0 44.5 0:08.38 varnishd 1244 zope 15 0 637m 461m 3428 S0 30.5 40:41.93 python2.4 1245 root 15 0 98196 81m 81m S0 5.4 0:00.02 varnishd so the "relatively small amount of ram" varnish uses in addition to the address space is about 200M. are these usual numbers? > You should probably look at /proc/$pid/map instead of using top, BTW. i tried out /proc/$pid/maps but did not really succeed in calculating memory usage out of the address blocks (i think this is what needs to be done here?) this is an example from the manpage http://linux.die.net/man/5/proc address perms offset dev inode pathname 08048000-08056000 r-xp 03:0c 64593 /usr/sbin/gpm 08056000-08058000 rw-p d000 03:0c 64593 /usr/sbin/gpm probably the RES column in top is enough to get an idea how much RAM is consumed by a process. regards, harald -- Webmeisterei GmbH - Büro für Netzfragen Tel: +43 5572 908877, Fax: +43 5572 908877-66 Steinebach 18, A-6850 Dornbirn http://www.webmeisterei.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: make varnish don't start a subprocess
hi des Am Freitag, 20. März 2009 19:14:30 schrieb Dag-Erling Smørgrav: > > the varnishd started by supervisor just needs 5% memory and gets > > restarted if it grows above a certain limit. the subprocess eats up my > > RAM and can't be conrolled by supervisor. > > The parent will restart it if you kill it (but doing so will flush the > cache). However, varnishd doesn't "eat memory", unless there is a bug. > You should read varnishd(1) and figure out how to select the storage > backend and how to limit the size of the cache. > > DES i'd be happy to make varnish stick to a certain memory limit. maybe you've got some ideas how to make that happen? is the file backend i'm using too small? what other factors could make the varnish process use such a lot of RAM i try to provide as much information as possible, please let me know if you need further details: atm we're using filestorage with a limit of 500M. whilst the file itself is exactly 500 megabyte in size, varnish uses 1115M memory - 44% of the total available 2,5GB RAM (1,5GB RAM, 1GB disk-swap) after 5 days of uptime. see `figure 1` below. these are the arguments we're starting varnish with: #!/bin/sh exec /home/zope/parts/varnish-build/sbin/varnishd \ -f "/home/zope/etc/varnish.vcl" \ -P "/home/zope/parts/varnish/varnish.pid" \ -a localhost:6181 \ -T localhost:6182 \ -s file,"/home/zope/parts/varnish/storage",500M \ -F \ -t 0 "$@" varnish version: varnish-2.0.2.tar.gz figure 1: memory consumption of varnish and it's subprocess:: Mem: 1548288k total, 1540604k used, 7684k free, 680k buffers Swap: 1048568k total, 1048568k used,0k free, 156252k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 4096 nobody15 0 1921m 666m 81m S0 44.1 0:42.44 varnishd 13263 zope 15 0 908m 646m 1764 S0 42.8 252:01.48 python2.4 4090 root 15 0 98196 80m 80m S0 5.3 0:00.12 varnishd thanks for your help, fRiSi -- Webmeisterei GmbH - Büro für Netzfragen Tel: +43 5572 908877, Fax: +43 5572 908877-66 Steinebach 18, A-6850 Dornbirn http://www.webmeisterei.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Purge and cache content mist.
hi Am Montag, 23. März 2009 15:10:01 schrieb Sébastien FOUTREL: > Thank you. > > About the url.purge, should I understand that url.purge doesn't manage > hosts so url.purge /that-part/.* will purge > http://onedomain.com/that-part/* and http://otherdomain.com/that-part/* ? > Can I use onedomaine\.com/that-part/.* to purge only '/that-part/' in that > host :) ? on my setup i only have one host. i'm sure you can restrict that to one single host, but i don't know by heart. if url.purge can't do that you can setup url purging see the link below the faq question http://varnish.projects.linpro.no/wiki/FAQ#HowcanIforcearefreshonaobjectcachedbyvarnish http://varnish.projects.linpro.no/wiki/VCLExamplePurging regards, harald > > > -Message d'origine- > De : Harald Friessnegger [mailto:har...@webmeisterei.com] > Envoyé : lundi 23 mars 2009 14:14 > À : varnish-misc@projects.linpro.no > Cc : Sébastien FOUTREL > Objet : Re: Purge and cache content mist. > > hi sebastien > > Am Montag, 23. März 2009 12:32:22 schrieb Sébastien FOUTREL: > > Hello, > > > > I read a lot of things in the list and varnish sites but some things stay > > fuzzy to me. > > > > Is there a way to find if something is in the cache, have a map of cached > > content or some script that give one ? > > i don't know of a list output. what i do is to use varnishlog or turn on > debug headers: > > > open varnishlog, request a resource and see at the X-Varnish-Action Header > > > requesting a resource the first time that gets cached will give you > > TxHeader c X-Varnish-Action: FETCH (insert) > > requesting it again will log > > TxHeader c X-Varnish-Action: HIT (deliver - from cache) > > > > personally i'm using some combination with grep that gives me the headers > i'm interested in: > > > For debugging all requests to a certain url (eg /test/test/image_mini) and > see important headers:: > > varnishlog -c -o RxURL '^/test/test/image_mini$' | grep -E > "RxRequest|RxURL| Host:|TxStatus|TxResponse|X-Varnish-Action" > > > Is it possible to do recursive purge in a way or another ? > > > > I'd like to purge "www.example.org/that-part/" is to possible to simply > > purge all objects under that url ? > > run varnish with the -T option (eg "-T localhost:6182") > > connet via telnet: > > telnet localhost 6182 > > and call something like > > url.purge /that-part/.* > > > to see whether to use www.example.org/that-part or just /that-part or any > other fancy url (you might have a webserver rewrite urls for doing > virtualhosting) you can use varnishlog again: > > varnishlog -c -i RxURL > > > > hope that helps > fRiSi > > > Thank you. > > > > -- > > Sébastien FOUTREL > > > > > > > > ___ > > varnish-misc mailing list > > varnish-misc@projects.linpro.no > > http://projects.linpro.no/mailman/listinfo/varnish-misc -- Webmeisterei GmbH - Büro für Netzfragen Tel: +43 5572 908877, Fax: +43 5572 908877-66 Steinebach 18, A-6850 Dornbirn http://www.webmeisterei.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Purge and cache content mist.
hi sebastien Am Montag, 23. März 2009 12:32:22 schrieb Sébastien FOUTREL: > Hello, > > I read a lot of things in the list and varnish sites but some things stay > fuzzy to me. > > Is there a way to find if something is in the cache, have a map of cached > content or some script that give one ? i don't know of a list output. what i do is to use varnishlog or turn on debug headers: open varnishlog, request a resource and see at the X-Varnish-Action Header requesting a resource the first time that gets cached will give you TxHeader c X-Varnish-Action: FETCH (insert) requesting it again will log TxHeader c X-Varnish-Action: HIT (deliver - from cache) personally i'm using some combination with grep that gives me the headers i'm interested in: For debugging all requests to a certain url (eg /test/test/image_mini) and see important headers:: varnishlog -c -o RxURL '^/test/test/image_mini$' | grep -E "RxRequest|RxURL| Host:|TxStatus|TxResponse|X-Varnish-Action" > > Is it possible to do recursive purge in a way or another ? > > I'd like to purge "www.example.org/that-part/" is to possible to simply > purge all objects under that url ? run varnish with the -T option (eg "-T localhost:6182") connet via telnet: telnet localhost 6182 and call something like url.purge /that-part/.* to see whether to use www.example.org/that-part or just /that-part or any other fancy url (you might have a webserver rewrite urls for doing virtualhosting) you can use varnishlog again: varnishlog -c -i RxURL hope that helps fRiSi > > Thank you. > > -- > Sébastien FOUTREL > > > > ___ > varnish-misc mailing list > varnish-misc@projects.linpro.no > http://projects.linpro.no/mailman/listinfo/varnish-misc -- Webmeisterei GmbH - Büro für Netzfragen Tel: +43 5572 908877, Fax: +43 5572 908877-66 Steinebach 18, A-6850 Dornbirn http://www.webmeisterei.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
make varnish don't start a subprocess
hi there when i start varnishd i'll find two varnishd processes running afterwards: # top ... Mem: 1548288k total, 1540624k used, 7664k free, 2172k buffers Swap: 1048568k total, 767628k used, 280940k free, 171980k cached 13271 nobody 15 0 1864m 771m 90m S 0 51.0 0:14.43 varnishd 13263 zope 15 0 635m 454m 3500 S 0 30.1 47:59.16 python2.4 13265 root 15 0 98192 81m 80m S 0 5.4 0:00.04 varnishd 13262 root 16 0 109m 12m 2772 S 0 0.8 0:50.06 python2.4 ... is there a way to run varnish as a single process? background: i'd like to control the memory-usage of processes and restart them if they grow above a certain limit using http://supervisord.org/. the varnishd started by supervisor just needs 5% memory and gets restarted if it grows above a certain limit. the subprocess eats up my RAM and can't be conrolled by supervisor. full problem description can be seen in the supervisor-users mailinglist archive: http://lists.supervisord.org/pipermail/supervisor-users/2009-March/000313.html people at supervisor mailing list pointed me to varnish-misc. thanks for your help! fRiSi -- Webmeisterei GmbH - Büro für Netzfragen Tel: +43 5572 908877, Fax: +43 5572 908877-66 Steinebach 18, A-6850 Dornbirn http://www.webmeisterei.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
