Hi, Varnish will only have the most requested objects in RAM, so why is it a problem to have a large cache?
I've been trying to run with large data sets for a long time, and have had a lot of problems with it. But it has gotten better, due to Poul-Hennings and also Dag-Erlings help. Did you check the Performance page on the wiki (http://varnish.projects.linpro.no/wiki/Performance), particularly the section about running with many objects? By lowering obj_workspace, you can reduce how much space Varnish uses to fit HTTP headers. You also may want to switch to run trunk instead of 1.1.2. Cheers, Anders. On Tue, May 06, 2008 at 02:25:18PM -0700, Demitrious Kelly wrote: > We have a varnish (varnish-1.1.2) server handling about 2,500 requests > per second. The nature of these requests are extraordinarily random, > and the disks cannot keep up with the demand, we therefor must keep the > cache in RAM. Under highest traffic conditions this generates about 3GB > per hour of cached data. The expires on the data being generated are > set to a random number of seconds between 1800 and 3600. So > theoretically the cache size should never reach more than 4gb or so, > well within the servers RAM capacity of 8Gb. What we are seeing however > is that the cache will continue to grow to (and beyond if we allow it,) > the 8gb mark. At which point we have to invalidate the cache by > restarting the processes (naturally this causes the undesired effect of > pegging the back-end servers with very high loads until enough data is > back in the cache). We have tried limiting the cache size to 7gb, > however once the 7gb mark is reached the varnish cache restarts... with > the undesired effect of pegging the back-end servers with very high > loads until enough data is back in the cache. > > One definite oddity in our particular usage patterns is that 30% or more > of our cached data is comprised of headers, and not content. Currently, > for example, varnishstat tells me: > 3424095028 743539.36 851553.10 Total header bytes > 7830249685 1718569.76 1947338.89 Total body bytes > > Does anybody have any insight into this behavior? We would like to not > have to resort to multiple varnish servers with mickey mouse hacks like > staggering restarts to reduce the effect on the back end servers. > > System details below: > --------------------------------------- > > [EMAIL PROTECTED]:~# uname -a > Linux foo.com 2.6.18-6-amd64 #1 SMP Sun Feb 10 17:50:19 UTC 2008 > x86_64 GNU/Linux > > [EMAIL PROTECTED]:~# free -m > total used free shared buffers > cached > Mem: 7991 4708 3282 0 91 > 2317 > -/+ buffers/cache: 2300 5691 > Swap: 0 0 0 > > [EMAIL PROTECTED]:~# cat /proc/cpuinfo | grep -E 'mhz|model name' -i > model name : Dual-Core AMD Opteron(tm) Processor 2218 > cpu MHz : 2593.784 > model name : Dual-Core AMD Opteron(tm) Processor 2218 > cpu MHz : 2593.784 > model name : Dual-Core AMD Opteron(tm) Processor 2218 > cpu MHz : 2593.784 > model name : Dual-Core AMD Opteron(tm) Processor 2218 > cpu MHz : 2593.784 > > Start Command: > /usr/local/sbin/varnishd -a 0.0.0.0:80 -f /etc/varnish/varnish.vcl > -P /var/run/varnishd.pid -T 192.x.x.x:6969 -t 600 -w > > /etc/varnish/varnish.vcl > backend default { > set backend.host = "foo.lan"; > set backend.port = "http"; > } > > sub vcl_recv { > if ( req.url ~ "^/(part1|part2)" && req.http.cookie ) { > lookup; > } > } > > sub vcl_fetch { > if (obj.ttl < 600s) { > set obj.ttl = 600s; > } > if (obj.status == 404) { > set obj.ttl = 10s; > } > if (obj.status == 500 || obj.status == 503) { > pass; > } > insert; > } > _______________________________________________ > varnish-misc mailing list > [email protected] > http://projects.linpro.no/mailman/listinfo/varnish-misc -- Anders. _______________________________________________ varnish-misc mailing list [email protected] http://projects.linpro.no/mailman/listinfo/varnish-misc
