Re: Memory usage
Thanks Darryl! One follow-up about the VIRT size (mine is 55.2GB). I thought the VIRT size includes the entire amount of VARNISH_STORAGE_SIZE (50GB in my case), regardless of how much of the virtual memory is actually being used to store cached objects. This seems to be the case based on a few minutes of experimenting with that setting. That is correct. I was assuming from your initial post (probably wrongly) that your cache was 'full' ie, that there were in your case 50GB of objects cached already. You can see how much of the allocated cache is used with varnishstat; the bytes allocated and bytes free row will tell you this. Unfortunately, what you want is the intersection of bytes allocated and bytes not currently in physical memory and if the entire cache is not full, then it gets pretty tough to know, because Varnish *by design* neither knows nor cares about the OSes swapping arrangements. o I'm not sure I understand how to determine the amount of virtual memory I'm actually using -- in other words, the amount of RAM I need to add for optimal performance -- from the VIRT and RES numbers alone. Any chance I could ask you to please fill in what I'm missing? Probably the two entries from the varnishstat output I mentioned above will give you a better idea. Obviously, if you really have 50GB of items to cache, then you will need a lot of RAM. Perhaps if you monitor the bytes allocated row over time you will see it stabilise - this is the amount of RAM that you would need to keep all your cacheable items in physical memory. Of course, just because something ends up in the cache once, doesn't mean there is necessarily any value in keeping it in physical memory; if no-one ever requests it again, then it makes sense for it to end up in swap... regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
RE: Varnish virtual memory usage
Can you paste your VCL please? Do you use the purge_url function in it? regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com I don't remember how long it took exactly, but it filled up rather quickly. Probably a couple of hours. I also noticed the LRU-moved stat was pretty constant all along, no large spikes towards the end of the swap. Anyways, it was WAY over 1GB, we managed to fill more around 6-7GB. Just for kicks, I tried changing the storage to malloc, and limit it to 1GB, to see the difference. We have live traffic now, so I don't wanna run the load testing scripts, but we should hit the 1GB limit within a day or two. -Original Message- From: Kristian Lyngstol [mailto:krist...@redpill-linpro.com] Sent: Wednesday, October 21, 2009 3:28 PM To: Roi Avinoam Cc: varnish-misc@projects.linpro.no Subject: Re: Varnish virtual memory usage On Wed, Oct 21, 2009 at 03:07:00PM +0200, Roi Avinoam wrote: Thanks for your reply. 1. What I did was create 100 simultaneous processes, and each process requested the same page (with 'curl'): a. Once with the exact same URL - which resulted in a 99.9% hit-ratio and VERY high performance on varnish. b. And once with a random key that changes the URL (something like 'index.php?rand=193837364'), thus forcing Varnish to hit the backend, and store the multiple objects in memory. c. A combination of the two - in an attempt to maintain a 60-70% hit-ratio. What we saw is that the kernel simply filled all of the RAM *and* the swap until it crashed. When did it fill up and how fast? 2. I'm sorry, but I'm still confused about the mmaped file. If it's limited to 1G, Varnish shouldn't use more than 1G of virtual memory, correct? Or in our setup - 1G of RAM? The -s parameter only applies to storage, which is to say that it's an approximiation and there will be some additional overhead which is not stored there, ie datastructures for threads, sessions, backends etc. So with -s ..., 1G, you should expect a little bit over 1GB, but not nearly as much as you describe. -- Kristian Lyngstøl Redpill Linpro AS Tlf: +47 21544179 Mob: +47 99014497 ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish, long lived cache and purge on change
Then we ran out of ram and varnish started causing a ton of swap activity and it increased the response times of everything on the site to unusable. After poking about i seem to have found the culprit. When you use url.purge it seems to keep a record of that and check every object as it is fetched to see if it was purged or not. To test this i set a script to purge a lot of stuff and got the same problem to happen. Hi Karl, This is about the fourth thread now on varnish-misc about the perils of using url.purge :) My suggestion in your case, if managing obj.ttl is too tough, is to look at breaking out the page with ESI and having shorter cache times on the portions of the page that will potentially end up getting updated. This combined with a reasonable grace period for those parts of the page should alleviate most of the load you might otherwise experience, and keep response times low and page freshness high. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory spreading, then stop responding
Darryl Dixon - Winterhouse Consulting wrote: snip Can anyone suggest why varnish is using more memory than it's allocated, and why varnishlog would stop returning any output? Varnishlog was writing to disk, so I can probably extract the end of that, if it's of use. Hi Rob, There have been a few threads about this now on this mailing list. Probably it relates to the use of purge_url in your VCL. Are you using this function at all? regards, Darryl Darryl, Thanks for your reply. Yes we are using purge_url, but I was under the impression that since http://varnish.projects.linpro.no/changeset/3329, there wasn't a problem. I've not succeeded in finding the threads you mentioned in your email. Can you either point me at them, or let me know their conclusion? Hi Rob, See the thread concluding here (the solution to swap purge_url for obj.ttl=0 is the correct one): http://projects.linpro.no/pipermail/varnish-misc/2009-April/002743.html And also the thread concluding here: http://projects.linpro.no/pipermail/varnish-misc/2009-June/002840.html regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: hit for pass problems (was: Problems getting Varnish to cache certain things)
Hi,
On Wed, Jun 17, 2009 at 08:32:07PM +, Poul-Henning Kamp wrote:
Only pass in vcl_fetch will create a hit for pass object.
Hrm. When did this change? We tested and confirmed in 2.0.3(?) that
pass'ing out of vcl_hit caused the hit-for-pass behaviour.
The object will not be marked as hit-for-pass if you do that, but
the resulting behaviour is the same, so unless you inspect stats,
you cannot tell the difference.
I am trying to set ttl before doing pass in vcl_fetch, at all places in
vcl_fetch where I do pass. Like for example:
if (!beresp.cacheable) {
set beresp.ttl = 1200s;
pass;
}
But still, it happens that Varnish will not cache objects, unless I
restart it (I waited more than the above TTL value).
The problem is the call to 'pass'. You must not do this in vcl_fetch if
you want Varnish to be able to cache the item. Instead, restart the
request with 'restart'.
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Problems getting Varnish to cache certain things
In message 41184.118.93.77.92.1244834146.squir...@services.directender.co.nz, Darryl Dixon - Winterhouse Consulting writes: Hi Anders, If you perform certain actions in vcl_hit, Varnish will mark the object as a 'hit for pass' which means that from then on it will always simply pass every matching request to the backend. A restart clears this. If I recall correctly, one of the triggers is using 'pass' inside vcl_hit or vcl_fetch. Only pass in vcl_fetch will create a hit for pass object. Hrm. When did this change? We tested and confirmed in 2.0.3(?) that pass'ing out of vcl_hit caused the hit-for-pass behaviour. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Problems getting Varnish to cache certain things
Hi Anders, If you perform certain actions in vcl_hit, Varnish will mark the object as a 'hit for pass' which means that from then on it will always simply pass every matching request to the backend. A restart clears this. If I recall correctly, one of the triggers is using 'pass' inside vcl_hit or vcl_fetch. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com Hi, I had a situation where some cache servers would cache a URL, and others would not. The strange thing is, if I restarted Varnish on the servers that did not cache it, then it would cache the URL. How is that? When Varnish delivers a miss, it should fetch (and if possible) cache the URL on the next attempt? Why does a restart have any significance? I tried some purging, without any success. After all, how can you purge something that is not cached.. I wrote a simple script to check: Checking server cache2.xx.no for URI banner.xx.no/rest/foo_rest/foo/prepackage/result?BAR=200907sort=1 = HTTP/1.1 200 OK X-Varnish-IP: 192.168.39.142 X-Varnish-Server: cache2 X-Varnish: 942085260 941884380 X-Cache: HIT Sjekker server cache3.xx.no for URI banner.xx.no/rest/foo_rest/foo/prepackage/result?BAR=200907sort=1 = HTTP/1.1 200 OK X-Varnish-IP: 192.168.39.143 X-Varnish-Server: cache3 X-Varnish: 3642432563 X-Cache: MISS Miss.. -- Anders. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish storage tuning
Hi Mike, Quite possibly the purge_url usage is causing you a problem. I assume this is something that is being invoked from your VCL, rather than telnet-ing to the administrative interface or by varnishadm? My testing showed that with purge_url in the VCL, a 'purge record' was created every time the rule was struck, and that record never seemed to be removed, which meant that memory grew without bound nearly continuously (new memory allocated for each new purge record). See the thread I started here: http://www.mail-archive.com/varnish-misc@projects.linpro.no/msg02520.html Instead, in vcl_hit, if the object should be purged, set obj.ttl to 0 and then restart the request. This solved the problem for me. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com We're using Varnish and finding that Linux runs the OOM killer on the large varnish child process every few days. I'm not sure what's causing the memory to grow but now I want to tune it so that I know configuration is not an issue. The default config we were using was 10MB. We're using a small 32-bit EC2 instance (Linux 2.6.21.7-2.fc8xen) with 1.75G of RAM and 10GB of disk so I changed the storage specification to file,/var/lib/varnish/varnish_storage.bin,1500M. I'd like to be able give varnish 8GB of disk but it complains about sizes larger than 2GB. 32-bit limitation? Side note: I couldn't find any good doc on the various command line parameters for varnishd. The 2.0.4 src only contains a man page for vcl. It would be nice to see a man page for varnishd and its options. We are using purge_url heavily as we update documents - this shouldn't cause unchecked grow though, right? We aren't using regexps to purge. Attached is the /var/log/messages output from the oom-killer and here's a few lines for the lazy. I can't grok the output. Jun 11 15:35:02 (none) kernel: varnishd invoked oom-killer: gfp_mask=0x201d2, order=0, oomkilladj=0 [...snip...] Jun 11 15:35:02 (none) kernel: Mem-info: Jun 11 15:35:02 (none) kernel: DMA per-cpu: Jun 11 15:35:02 (none) kernel: CPU0: Hot: hi: 186, btch: 31 usd: 94 Cold: hi: 62, btch: 15 usd: 60 Jun 11 15:35:02 (none) kernel: HighMem per-cpu: Jun 11 15:35:02 (none) kernel: CPU0: Hot: hi: 186, btch: 31 usd: 26 Cold: hi: 62, btch: 15 usd: 14 Jun 11 15:35:02 (none) kernel: Active:213349 inactive:210447 dirty:0 writeback:0 unstable:0 Jun 11 15:35:02 (none) kernel: free:1957 slab:1078 mapped:23 pagetables:1493 bounce:13 Jun 11 15:35:02 (none) kernel: DMA free:7324kB min:3440kB low:4300kB high:5160kB active:355572kB inactive:346580kB present:739644kB pages_scanned:1108980 all_unreclaimable? yes Jun 11 15:35:02 (none) kernel: lowmem_reserve[]: 0 0 972 Jun 11 15:35:02 (none) kernel: HighMem free:504kB min:512kB low:1668kB high:2824kB active:497824kB inactive:495208kB present:995688kB pages_scanned:1537436 all_unreclaimable? yes Jun 11 15:35:02 (none) kernel: lowmem_reserve[]: 0 0 0 Jun 11 15:35:02 (none) kernel: DMA: 11*4kB 10*8kB 42*16kB 12*32kB 2*64kB 23*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 7324kB Jun 11 15:35:02 (none) kernel: HighMem: 1*4kB 6*8kB 4*16kB 0*32kB 0*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 500kB Jun 11 15:35:02 (none) kernel: Swap cache: add 1563900, delete 1563890, find 572160/581746, race 3+9 Jun 11 15:35:02 (none) kernel: Free swap = 0kB Jun 11 15:35:02 (none) kernel: Total swap = 917496kB ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: varnish storage tuning
Hi Mike,
Yes. If all you want to do is just get rid of the object, then all you
need to do is set obj.ttl = 0 (as per your example below). In my case I
wanted to honour the Cache-Control: no-cache header, which involved
re-fetching the object from the backend. If you simply want to remove the
item, then definitely you don't need to restart.
regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
Does restart cause the backend to be queried for the page again? I'd
prefer
if a purge did not re-fetch the URL from the backend as this can caused
unwanted content to stay in the cache and my cache is relatively limited
in
size. Can I do something like:
sub vcl_hit {
if (req.request == PURGE) {
set obj.ttl = 0;
error 200 Purged;
}
}
sub vcl_miss {
if (req.request == PURGE) {
error 200 Purged (not in cache);
}
}
On Thu, Jun 11, 2009 at 4:46 PM, Darryl Dixon - Winterhouse Consulting
darryl.di...@winterhouseconsulting.com wrote:
Instead, in vcl_hit, if the object should be purged, set obj.ttl to 0
and
then restart the request. This solved the problem for me.
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish 2.0.3 consuming excessive memory
| My understanding of using 'pass' in vcl_hit is that it causes the object | to be marked as 'hit for pass' and therefore is uncacheable from that | point onwards. In fact, pass'ing out of vcl_hit was what I used to do a | while ago, but it caused that exact behaviour (items would no longer be | cached by varnish) so I swapped it for purge_url... You can try setting obj.ttl = 0s and then restart in vcl_hit. I believe that should work. Thanks Tollef - I will try this today and report back. This seems to correct the problem - thanks Tollef! Further to root cause of the original issue: using purge_url seems to result inside EXP_Rearm with the objexp (object expire?) being added unconditionally to the exp_heap binheap, and it's not clear to me where (if ever) this heap is pruned. For example, I see in varnishlog multiple calls to the same URL from the same client with the same headers resulting in multiple unique entries being added to exp_heap, eg: ExpBan c 981217322 was banned [...] ExpBan c 981217323 was banned [...etc...] Where, if anywhere, is exp_heap pruned for entries like this? I couldn't quite make sense of exp_timer() to be certain of under what conditions it actually removes things from this heap - it looks like that function is mainly concerned with items expiring from the Expires http header or similar... regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish 2.0.3 consuming excessive memory
| My understanding of using 'pass' in vcl_hit is that it causes the object | to be marked as 'hit for pass' and therefore is uncacheable from that | point onwards. In fact, pass'ing out of vcl_hit was what I used to do a | while ago, but it caused that exact behaviour (items would no longer be | cached by varnish) so I swapped it for purge_url... You can try setting obj.ttl = 0s and then restart in vcl_hit. I believe that should work. Thanks Tollef - I will try this today and report back. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish 2.0.3 consuming excessive memory
Hi DES,
Darryl Dixon - Winterhouse Consulting
darryl.di...@winterhouseconsulting.com writes:
+if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
.*no-cache.*) {
+purge_url(regsub(req.url, [?].*$, .*$));
+}
+
It would be interesting to see how often this condition is true...
Shouldn't be too often in this environment - I have watched with
varnishlog and the rate is very low...
Is there possibly something in the regsub engine being triggered that is
very expensive and would cause it to consume and hold on to large
amounts
of memory?
I don't think so. However, your code will add an entry to the ban list
every time a request that matches the above condition is received, and
this entry will remain on the list until every object that was in the
cache before the entry was added has been requested at least once.
That sounds like a pretty good candidate then, as I can envisage that
there are possibly many items that may fit that criteria in my scenario.
However it definitely seems out of proportion to the actual cache
utilisation (only 25MB of objects in the cache, but memory growing to
multiple GB)
If you really want no-cache to purge the requested object, you should do
it in vcl_hit (set obj.ttl to 0 and pass). Otherwise, you should simply
pass directly from vcl_recv.
My understanding of using 'pass' in vcl_hit is that it causes the object
to be marked as 'hit for pass' and therefore is uncacheable from that
point onwards. In fact, pass'ing out of vcl_hit was what I used to do a
while ago, but it caused that exact behaviour (items would no longer be
cached by varnish) so I swapped it for purge_url...
regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish 2.0.3 consuming excessive memory
hi darryl i had a simlar problem (varnish child process consuming lots of memory) a week ago (see the thread make varnish don't start a subprocess) the solution for my problem seems to be to use a high-enough cache-size: starting varnish with -s file,/tmp/storage,300M will make the child process use *about* 800M RAM (the RES column in top) after some hours. after that i needed to restart varnish to free some ram for my webserver. starting with a bigger cache -s file,/tmp/storage,600M makes it using about 500M RAM (the RES column in top after 5 days of uptime) and happily coexist with zope ;-) maybe this helps you too. Hi Harald, That is a very interesting observation - I will give it a try. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Varnish 2.0.3 consuming excessive memory
Hi All, I have an odd problem that I have only noticed happening since moving from 1.1.2 to 2.0.3 - excessive memory consumption of the varnish child process. For example, I have a varnish instance with a 256MB cache allocated, that is currently consuming 4.9GB of resident memory (6.5GB virtual). The instance has only been running for 4 days and has only got 25MB of objects in the cache. This is clearly excessive and is causing us some serious problems in terms of memory pressure on the machine. Our VCL is largely unchanged from our 1.1.2 setup to the 2.0.3 except for the obvious vcl.syntax changes, and the introduction of request restarts under certain limited scenarios. Can anyone shed some light? regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish 2.0.3 consuming excessive memory
Further to this, I just ran the pmap on the runaway process again (roughly 15 minutes since last time), and the large ~2GB allocation has jumped by about 60MB in that time, eg: -- 3940: /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080 web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600 -f /u01/data/prd-edit/varnish/vcl.conf -s file,/u01/data/prd-edit/varnish/storage.cache,256M -n /u01/data/prd-edit/varnish/state -P /u01/data/prd-edit/varnish/varnishd.pid Address Kbytes RSSAnon Locked Mode Mapping 0040 312 - - - r-x-- varnishd 0064d000 8 - - - rw--- varnishd 0064f000 68 - - - rw---[ anon ] 4000 4 - - - -[ anon ] 40001000 10240 - - - rw---[ anon ] [...snip...] 2aaab090 262144 - - - rw-s- storage.cache 2aaac090 28 - - - r-x-- vcl.1P9zoqAU.so 2aaac09070002048 - - - - vcl.1P9zoqAU.so 2aaac0b07000 8 - - - rw--- vcl.1P9zoqAU.so 2aaac0c02056 - - - rw---[ anon ] 2aaac0f02052 - - - rw---[ anon ] 2aaac120 233476 - - - rw---[ anon ] 2aaacf70 2403328 - - - rw---[ anon ] 7fff4b619000 84 - - - rw---[ stack ] ff608192 - - - -[ anon ] -- -- -- -- total kB 3187344 - - - -- ...And in the time it's taken me to compose this mail it's grown by another 20MB: -- 2aaacf70 2418688 - - - rw---[ anon ] -- ...so it is a pretty consistent, continuous upwards trend in terms of memory usage... regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com Not that I have an answer, but I'd be curious to see the differences in 'pmap -x pid' output for the different children. --Michael That's a good thought - I'd almost forgotten about pmap. Anyhow, the 'normal' process is set up with a 2GB storage cache and pmap shows pretty much what you would expect - the largest anonymous segment is 41MB, with the vast majority of allocated memory booked against storage.cache (2GB). The pmap for the 'runaway' process shows 256MB booked against storage.cache, and then one of the an anonymous allocations is 2.3GB. For example: Normal process: -- 3896: /u01/app/varnish/sbin/varnishd -a web-cms1.prd.internal:8080 web-cms.prd.internal:8080 -T web-cms1.prd.internal:18080 -t 108000 -f /u01/data/prd/varnish/vcl.conf -s file,/u01/data/prd/varnish/storage.cache,2048M -n /u01/data/prd/varnish/state -P /u01/data/prd/varnish/varnishd.pid Address Kbytes RSSAnon Locked Mode Mapping 0040 312 - - - r-x-- varnishd 0064d000 8 - - - rw--- varnishd 0064f000 68 - - - rw---[ anon ] 4000 4 - - - -[ anon ] 40001000 10240 - - - rw---[ anon ] [...snip...] 2aaab090 2097152 - - - rw-s- storage.cache 2aab3090 28 - - - r-x-- vcl.1P9zoqAU.so 2aab309070002044 - - - - vcl.1P9zoqAU.so 2aab30b06000 8 - - - rw--- vcl.1P9zoqAU.so 2aab30c02056 - - - rw---[ anon ] 2aab30f02056 - - - rw---[ anon ] 2aab3120 41984 - - - rw---[ anon ] 7fffae8f2000 88 - - - rw---[ stack ] ff608192 - - - -[ anon ] -- -- -- -- total kB 2468512 - - - -- Runaway process: -- 3940: /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080 web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600 -f /u01/data/prd-edit/varnish/vcl.conf -s file,/u01/data/prd-edit/varnish/storage.cache,256M -n /u01/data/prd-edit/varnish/state -P /u01/data/prd-edit/varnish/varnishd.pid Address Kbytes RSSAnon Locked Mode Mapping 0040 312 - - - r-x-- varnishd 0064d000 8 - - - rw--- varnishd
Re: Articles regarding Varnish, VCL and ESI
Hi Jason, [...snip...] Making Catalyst Sites Shine with Varnish - includes an in-depth exploration of VCL http://www.catalystframework.org/calendar/2008/14 This is a great introduction to VCL - I'm quite certain I'll be directing newbies to it in the future for a gentle introduction. Site Building Hotness with Edge Side Includes - which covers the benefits of using ESI and includes my JS based ESI parser that is useful for developing ESI sites when you aren't behind Varnish. http://www.catalystframework.org/calendar/2008/17 I am especially excited about the JS ESI parser - the concept is an excellent one and it directly addresses an otherwise difficult problem - kudos to you, I'll be giving it a spin shortly. This whole second article in particular is excellent and very timely for me, as I have been thinking about using Varnish's ESI support for a new project I am working on. Currently my strategy for low-overhead dynamic pages revolves around just-in-time ajax. I am definitely inspired now to give it a crack in ESI instead. Thank you very much for this indeed. regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Logged-in users
We do both depending on scenario: we use ajax to update parts of a page
after-delivery (poor mans ESI ;) as suggested by Tim, and we also have a
custom vcl_hash that caches different copies of pages depending on various
cookies and other conditions (much as Miles suggests). Both fit depending
on the use-case.
Miles: FWIW, the CacheFu product for Plone may assist you to maximise the
caching potential of your site without too many custom tweaks to your
Varnish rules.
regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
Another approach is to simply use a small bit of Javascript. It's
easy to test for the existence of the cookie in Javascript and
set that text conditionally.
Then you have only one copy of the page to be cached.
The problem with the approach you've outlined here is
that other downstream caches won't understand the difference
(although most will simply refuse to cache any responses
if the request had a cookie header). Whereas the Javascript
approach also allows downstream caches to cache everything
efficiently.
Tim
On Nov 26, 2008, at 12:31 PM, Miles wrote:
Miles wrote:
Hi,
I have a site where users can log in. This sets a cookie with their
encrypted login details, so they can be authenticated. There are a
small number of pages which are user-specific (change your details
forms, etc), and these are set not to cache.
When a user is logged in, a message is shown at the top of the page
You
are now logged in. However, nothing on the page depends on the
individual user.
My question is, how can I organise the cache to have the most cache
hits, given that there are effectively two versions of each page -
one
for logged in users, and one for anonymous users. I want to
specifically avoid each user having their own version of the page
stored
in the cache.
Thanks in advance for any wisdom anyone can share!
Miles
Thanks to everyone who suggested using ESI - I may have to use this,
but
would quite like to avoid it, as it's useful to be able to run the app
without varnish in front for development/testing.
I wondered whether it was possible to use vcl_hash for my purposes, as
follows:
sub vcl_hash {
//hash the object with url+host
set req.hash += req.url;
set req.hash += req.http.host;
# see if the user has a cookie to indicate they are logged in
if req.http.cookie ~ '__ac=':
set req.hash += 'authenticated';
else:
set req.hash += 'anonymous'
hash;
}
Would this give me the two representations that I require for each
page
- or am I going down a route that will turn out bad?! I couldn't find
much information about vcl_hash, so I'm not sure if I'm barking up the
wrong tree or not...
Regards,
Miles
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Strategy for large cache sets
Our cache files reside on a SAN where we can get around 500MB/s over iSCSI. We went that route because we could not get Linux to swap efficiently enough to make malloc feasible. I know this is off-topic, but, I'm interested in the 500MB/s figure... Are you on 10gbit ethernet? Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish and hot linking
I've set up varnish to handle hot linking, but I have an issue with
varnish caching the hotlink image for the image request.
Here's the relevant part of default.vcl:
sub vcl_recv {
# Add a unique header containing the client address
remove req.http.X-Forwarded-For;
setreq.http.X-Forwarded-For = client.ip;
if (req.request == GET) {
if ( !(req.http.referer ~ [a-z0-9]+.domain.com|
yahoo.|rssbandit.org|newsgator.com|google.|search?q=cache )
(req.http.host ~ ^(raw)?images.domain.) (req.http.referer) ) {
set req.url=http://images.domain.com/hotlink.gif;;
pass;
}
lookup;
}
if (req.request == PURGE) {
purge_url(req.url);
error 200 OK!!;
}
}
The problem is this, a user hotlinks http://images.domain.com/puppy.jpeg
and we return hotlink.gif.
Now if a user legitimately loads puppy.jpeg he receives the cached
copy of hotlink.gif in its place.
I though by setting the req.url that the cached file is now
hotlink.gif and we haven't even attempted to store puppy.jpeg in the
cache.
I need to legitimately serve puppy.jpeg even if it's being hotlinked
from somewhere.
Any help would be appreciated.
Hi Chris,
You need to add some conditions in vcl_fetch to prevent the 'hotlink'
image from being cached on its way back from being fetched from the
backend. In other words, calling 'pass' in vcl_recv ensures that the
current request will get a fresh copy from the backend, but during the
processing of that request, the fresh copy will be 'insert'-ed into the
cache during vcl_fetch, and so subsequent requests which hit the 'lookup'
in vcl_recv will see the previously cached copy. You need to perform a
check in vcl_fetch very similar to the one in vcl_recv to prevent the
'pass'-ed response from being cached.
Hope this helps,
regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Reloading default.vcl
In message [EMAIL PROTECTED], Skye Poier Nott writes : Is it possible to reload /usr/local/etc/varnish/default.vcl without restarting? From the docs it looks like one way to do this would be through the telnet interface: vcl.load some-unique-name /usr/local/etc/varnish/default.vcl vcl.use some-unique-name each time the default.vcl is changed, unless there's reload option I haven't found. Correct. Afterwards you can vcl.discard old-unique-name if you want to get rid of it. It is pretty easy to automate this all with expect(1) as well, if an 'automated' reload solution is required :) regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: cache files pass php
Is there an easy vcl trick to get varnish to always pass php requests to
the
webserver and to cache images, css javascript, etc?
I would like
apache-varnish-client
If the client requests a document, the php is run on the webserver,
varnish
takes it and downloads it to the client so that the apache thread can be
used for other things. Varnish can cache the images and text files as they
don't change.
Thoughts?
Hi Jack,
This seems straightforward, in vcl_recv you would need something like:
sub vcl_recv {
if (req.url ~
\.(ico|gif|jpeg|jpg|png|swf|css|htc|js|bz2|gz|zip|xls|ppt|pdf|doc|rar))
{
lookup;
}
pass;
}
The idea is that items that you 'know' are cacheable get looked up in the
cache and fetched from the backend if necessary, and that everything else
otherwise gets passed straight to the backend for servicing.
(note that this is an extremely abbreviated example that probably omits
several important considerations like cookies, etc. Including, depending
on what headers, etc the backend emits, you may want some more code in
vcl_fetch to decide whether something is insert-ed into the cache or
pass-ed onwards)
Hope this helps,
regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: honoring browser reload request
is the
following VCL the correct solution to support browser shift-reloads to
get fresh content and actually update the cache?
sub vcl_hit {
if (req.http.Cache-Control ~ no-cache) {
set obj.ttl = 0s;
pass;
}
}
Only kind-of; IE and Mozilla send different things with a SHIFT (or CTRL)
+ Reload, we do virtually what you have outlined above, but with a slight
addition:
in vcl_recv, we shortcut and send such requests straight to lookup:
/* Honour Cache-Control: and Pragma: ... */
if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
.*no-cache.*) {
lookup;
}
...and then in vcl_recv we do similar to you:
/* Honour Cache-Control: and Pragma: ... */
if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
.*no-cache.*) {
set obj.ttl = 0s;
pass;
}
As far as I understood vcl(7), this means:
1. fetch the object from cache (implicit in the fact that we're inside
vcl_hit)
2. set ttl to 0s, expiring it (so next requests will fetch new content)
3. pass to backend (so *this* request see new content)
Does this do what I expect it to do?
Yes, this configuration does indeed do what you expect (purge the item, go
get a fresh one from the backend)
Is there a way to avoid hitting the backend twice?
You're only hitting the backend once if you're running the code from vcl_hit
Is there a way to do that in vcl_recv?
Not really, but see our shortcut above
(I guess not, since there's no obj object available yet, there)
I guess I could use purge_url(...), except I have virtual hosts and it
wants a regexp and req.url could contain some special char (could it?).
We tried with purge_url but it was just too hard and didn't want to do
what we expected. We've had good results with our configuration above.
Hope this helps,
regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: honoring browser reload request
in vcl_recv, we shortcut and send such requests straight to lookup:
/* Honour Cache-Control: and Pragma: ... */
if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
.*no-cache.*) {
lookup;
}
...and then in vcl_recv we do similar to you:
Bother, of course I meant: ..and then in *vcl_hit* we do similar to you... :)
/* Honour Cache-Control: and Pragma: ... */
if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
.*no-cache.*) {
set obj.ttl = 0s;
pass;
}
regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish Questions
Hello list, Hi Jack, 1. connection reset issues Can't comment specifically on these, but your varnish version is very old. 1.1.2 is good and stable for us, and -trunk is apparently pretty stable, too... 2. caching pages I have updated content on my main site, and would like to clear the varnish cache, I telnet into the management interface and run purge * (I also tried purge *. , purge .* , purge *.*) I then clear my browser cache, and refresh the page, I get the old page. If I visit the server directly, I get the new page? it's 'url.purge' in 1.1.2, not sure about your version. Below is my vcl.conf file I have comment out the cookie part because I [...snip...] It's important to know/remember that the vcl you write does not *replace* the default vcl, it just runs first, unless you specifically return a different action (eg, lookup, pass, etc). The cookie stuff you hashed out will still run in the default vcl afterward with your current setup. See: http://varnish.projects.linpro.no/wiki/VCLExampleDefault ...for more information on this (be aware the default presented there is probably not correct for your current version). Thanks for any help someone can give. You're welcome ;) regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
