Re: Memory usage

2010-01-26 Thread Darryl Dixon - Winterhouse Consulting
 Thanks Darryl!

 One follow-up about the VIRT size (mine is 55.2GB). I thought the VIRT
 size
 includes the entire amount of VARNISH_STORAGE_SIZE (50GB in my case),
 regardless of how much of the virtual memory is actually being used to
 store cached objects. This seems to be the case based on a few minutes of
 experimenting with that setting.

That is correct. I was assuming from your initial post (probably wrongly)
that your cache was 'full' ie, that there were in your case 50GB of
objects cached already. You can see how much of the allocated cache is
used with varnishstat; the bytes allocated and bytes free row will
tell you this. Unfortunately, what you want is the intersection of bytes
allocated and bytes not currently in physical memory and if the entire
cache is not full, then it gets pretty tough to know, because Varnish *by
design* neither knows nor cares about the OSes swapping arrangements.

 o I'm not sure I understand how to
 determine the amount of virtual memory I'm actually using -- in other
 words, the amount of RAM I need to add for optimal performance -- from
 the VIRT and RES numbers alone. Any chance I could ask you to please fill
 in what I'm missing?

Probably the two entries from the varnishstat output I mentioned above
will give you a better idea. Obviously, if you really have 50GB of items
to cache, then you will need a lot of RAM. Perhaps if you monitor the
bytes allocated row over time you will see it stabilise - this is the
amount of RAM that you would need to keep all your cacheable items in
physical memory.

Of course, just because something ends up in the cache once, doesn't mean
there is necessarily any value in keeping it in physical memory; if no-one
ever requests it again, then it makes sense for it to end up in swap...

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


RE: Varnish virtual memory usage

2009-10-21 Thread Darryl Dixon - Winterhouse Consulting
Can you paste your VCL please? Do you use the purge_url function in it?

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

 I don't remember how long it took exactly, but it filled up rather
 quickly. Probably a couple of hours. I also noticed the LRU-moved stat was
 pretty constant all along, no large spikes towards the end of the swap.

 Anyways, it was WAY over 1GB, we managed to fill more around 6-7GB.

 Just for kicks, I tried changing the storage to malloc, and limit it to
 1GB, to see the difference. We have live traffic now, so I don't wanna run
 the load testing scripts, but we should hit the 1GB limit within a day or
 two.

 -Original Message-
 From: Kristian Lyngstol [mailto:krist...@redpill-linpro.com]
 Sent: Wednesday, October 21, 2009 3:28 PM
 To: Roi Avinoam
 Cc: varnish-misc@projects.linpro.no
 Subject: Re: Varnish virtual memory usage

 On Wed, Oct 21, 2009 at 03:07:00PM +0200, Roi Avinoam wrote:
 Thanks for your reply.

 1. What I did was create 100 simultaneous processes, and each process
 requested the same page (with 'curl'):
  a. Once with the exact same URL - which resulted in a 99.9% hit-ratio
 and VERY high performance on varnish.
  b. And once with a random key that changes the URL (something like
 'index.php?rand=193837364'), thus forcing Varnish to hit the backend,
 and store the multiple objects in memory.
  c. A combination of the two - in an attempt to maintain a 60-70%
 hit-ratio.

 What we saw is that the kernel simply filled all of the RAM *and* the
 swap until it crashed.

 When did it fill up and how fast?

 2. I'm sorry, but I'm still confused about the mmaped file. If it's
 limited to 1G, Varnish shouldn't use more than 1G of virtual memory,
 correct? Or in our setup - 1G of RAM?

 The -s parameter only applies to storage, which is to say that it's an
 approximiation and there will be some additional overhead which is not
 stored there, ie datastructures for threads, sessions, backends etc.

 So with -s ..., 1G, you should expect a little bit over 1GB, but not
 nearly as much as you describe.

 --
 Kristian LyngstĂžl
 Redpill Linpro AS
 Tlf: +47 21544179
 Mob: +47 99014497
 ___
 varnish-misc mailing list
 varnish-misc@projects.linpro.no
 http://projects.linpro.no/mailman/listinfo/varnish-misc


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish, long lived cache and purge on change

2009-08-18 Thread Darryl Dixon - Winterhouse Consulting
 Then we ran
 out of ram and varnish started causing a ton of swap activity and it
 increased the response times of everything on the site to unusable.

 After poking about i seem to have found the culprit.  When you use
 url.purge
 it seems to keep a record of that and check every object as it is fetched
 to
 see if it was purged or not.  To test this i set a script to purge a lot
 of
 stuff and got the same problem to happen.

Hi Karl,

This is about the fourth thread now on varnish-misc about the perils of
using url.purge :) My suggestion in your case, if managing obj.ttl is too
tough, is to look at breaking out the page with ESI and having shorter
cache times on the portions of the page that will potentially end up
getting updated. This combined with a reasonable grace period for those
parts of the page should alleviate most of the load you might otherwise
experience, and keep response times low and page freshness high.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Memory spreading, then stop responding

2009-07-28 Thread Darryl Dixon - Winterhouse Consulting
 Darryl Dixon - Winterhouse Consulting wrote:
 snip
 Can anyone suggest why varnish is using more memory than it's
 allocated,
 and why varnishlog would stop returning any output?  Varnishlog was
 writing to disk, so I can probably extract the end of that, if it's of
 use.



 Hi Rob,

 There have been a few threads about this now on this mailing list.
 Probably it relates to the use of purge_url in your VCL. Are you using
 this function at all?

 regards,
 Darryl
 Darryl,

 Thanks for your reply.  Yes we are using purge_url, but I was under the
 impression that since http://varnish.projects.linpro.no/changeset/3329,
 there wasn't a problem.  I've not succeeded in finding the threads you
 mentioned in your email.  Can you either point me at them, or let me
 know their conclusion?


Hi Rob,

See the thread concluding here (the solution to swap purge_url for
obj.ttl=0 is the correct one):
http://projects.linpro.no/pipermail/varnish-misc/2009-April/002743.html

And also the thread concluding here:
http://projects.linpro.no/pipermail/varnish-misc/2009-June/002840.html

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: hit for pass problems (was: Problems getting Varnish to cache certain things)

2009-06-24 Thread Darryl Dixon - Winterhouse Consulting
 Hi,

 On Wed, Jun 17, 2009 at 08:32:07PM +, Poul-Henning Kamp wrote:
 Only pass in vcl_fetch will create a hit for pass object.
Hrm. When did this change? We tested and confirmed in 2.0.3(?) that
pass'ing out of vcl_hit caused the hit-for-pass behaviour.
 The object will not be marked as hit-for-pass if you do that, but
 the resulting behaviour is the same, so unless you inspect stats,
 you cannot tell the difference.

 I am trying to set ttl before doing pass in vcl_fetch, at all places in
 vcl_fetch where I do pass. Like for example:

 if (!beresp.cacheable) {
 set beresp.ttl = 1200s;
 pass;
 }

 But still, it happens that Varnish will not cache objects, unless I
 restart it (I waited more than the above TTL value).

The problem is the call to 'pass'. You must not do this in vcl_fetch if
you want Varnish to be able to cache the item. Instead, restart the
request with 'restart'.

Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Problems getting Varnish to cache certain things

2009-06-16 Thread Darryl Dixon - Winterhouse Consulting
 In message
 41184.118.93.77.92.1244834146.squir...@services.directender.co.nz,
  Darryl Dixon - Winterhouse Consulting writes:
Hi Anders,

If you perform certain actions in vcl_hit, Varnish will mark the object
 as
a 'hit for pass' which means that from then on it will always simply pass
every matching request to the backend. A restart clears this. If I recall
correctly, one of the triggers is using 'pass' inside vcl_hit or
vcl_fetch.

 Only pass in vcl_fetch will create a hit for pass object.

Hrm. When did this change? We tested and confirmed in 2.0.3(?) that
pass'ing out of vcl_hit caused the hit-for-pass behaviour.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Problems getting Varnish to cache certain things

2009-06-12 Thread Darryl Dixon - Winterhouse Consulting
Hi Anders,

If you perform certain actions in vcl_hit, Varnish will mark the object as
a 'hit for pass' which means that from then on it will always simply pass
every matching request to the backend. A restart clears this. If I recall
correctly, one of the triggers is using 'pass' inside vcl_hit or
vcl_fetch.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

 Hi,

 I had a situation where some cache servers would cache a URL, and others
 would not. The strange thing is, if I restarted Varnish on the servers
 that did not cache it, then it would cache the URL. How is that? When
 Varnish delivers a miss, it should fetch (and if possible) cache the URL
 on the next attempt? Why does a restart have any significance? I tried
 some purging, without any success. After all, how can you purge
 something that is not cached..

 I wrote a simple script to check:

 Checking server cache2.xx.no for URI
 banner.xx.no/rest/foo_rest/foo/prepackage/result?BAR=200907sort=1
 =
 HTTP/1.1 200 OK
 X-Varnish-IP: 192.168.39.142
 X-Varnish-Server: cache2
 X-Varnish: 942085260 941884380
 X-Cache: HIT

 Sjekker server cache3.xx.no for URI
 banner.xx.no/rest/foo_rest/foo/prepackage/result?BAR=200907sort=1
 =
 HTTP/1.1 200 OK
 X-Varnish-IP: 192.168.39.143
 X-Varnish-Server: cache3
 X-Varnish: 3642432563
 X-Cache: MISS

 Miss..

 --
 Anders.
 ___
 varnish-misc mailing list
 varnish-misc@projects.linpro.no
 http://projects.linpro.no/mailman/listinfo/varnish-misc


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: varnish storage tuning

2009-06-11 Thread Darryl Dixon - Winterhouse Consulting
Hi Mike,

Quite possibly the purge_url usage is causing you a problem. I assume this
is something that is being invoked from your VCL, rather than telnet-ing
to the administrative interface or by varnishadm?

My testing showed that with purge_url in the VCL, a 'purge record' was
created every time the rule was struck, and that record never seemed to be
removed, which meant that memory grew without bound nearly continuously
(new memory allocated for each new purge record). See the thread I started
here:
http://www.mail-archive.com/varnish-misc@projects.linpro.no/msg02520.html

Instead, in vcl_hit, if the object should be purged, set obj.ttl to 0 and
then restart the request. This solved the problem for me.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com




 We're using Varnish and finding that Linux runs the OOM killer on the
 large
 varnish child process every few days.  I'm not sure what's causing the
 memory to grow but now I want to tune it so that I know configuration is
 not
 an issue.
 The default config we were using was 10MB.  We're using a small 32-bit EC2
 instance (Linux 2.6.21.7-2.fc8xen) with 1.75G of RAM and 10GB of disk so I
 changed the storage specification to
 file,/var/lib/varnish/varnish_storage.bin,1500M.  I'd like to be able
 give
 varnish 8GB of disk but it complains about sizes larger than 2GB.  32-bit
 limitation?

 Side note: I couldn't find any good doc on the various command line
 parameters for varnishd.  The 2.0.4 src only contains a man page for vcl.
  It would be nice to see a man page for varnishd and its options.

 We are using purge_url heavily as we update documents - this shouldn't
 cause
 unchecked grow though, right?  We aren't using regexps to purge.



 Attached is the /var/log/messages output from the oom-killer and here's a
 few lines for the lazy.  I can't grok the output.

 Jun 11 15:35:02 (none) kernel: varnishd invoked oom-killer:
 gfp_mask=0x201d2, order=0, oomkilladj=0
 [...snip...]
 Jun 11 15:35:02 (none) kernel: Mem-info:
 Jun 11 15:35:02 (none) kernel: DMA per-cpu:
 Jun 11 15:35:02 (none) kernel: CPU0: Hot: hi:  186, btch:  31 usd:  94
 Cold: hi:   62, btch:  15 usd:  60
 Jun 11 15:35:02 (none) kernel: HighMem per-cpu:
 Jun 11 15:35:02 (none) kernel: CPU0: Hot: hi:  186, btch:  31 usd:  26
 Cold: hi:   62, btch:  15 usd:  14
 Jun 11 15:35:02 (none) kernel: Active:213349 inactive:210447 dirty:0
 writeback:0 unstable:0
 Jun 11 15:35:02 (none) kernel:  free:1957 slab:1078 mapped:23
 pagetables:1493 bounce:13
 Jun 11 15:35:02 (none) kernel: DMA free:7324kB min:3440kB low:4300kB
 high:5160kB active:355572kB inactive:346580kB present:739644kB
 pages_scanned:1108980 all_unreclaimable? yes

 Jun 11 15:35:02 (none) kernel: lowmem_reserve[]: 0 0 972

 Jun 11 15:35:02 (none) kernel: HighMem free:504kB min:512kB low:1668kB
 high:2824kB active:497824kB inactive:495208kB present:995688kB
 pages_scanned:1537436 all_unreclaimable? yes
 Jun 11 15:35:02 (none) kernel: lowmem_reserve[]: 0 0 0

 Jun 11 15:35:02 (none) kernel: DMA: 11*4kB 10*8kB 42*16kB 12*32kB 2*64kB
 23*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 7324kB
 Jun 11 15:35:02 (none) kernel: HighMem: 1*4kB 6*8kB 4*16kB 0*32kB 0*64kB
 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 500kB
 Jun 11 15:35:02 (none) kernel: Swap cache: add 1563900, delete 1563890,
 find
 572160/581746, race 3+9
 Jun 11 15:35:02 (none) kernel: Free swap  = 0kB

 Jun 11 15:35:02 (none) kernel: Total swap = 917496kB
 ___
 varnish-misc mailing list
 varnish-misc@projects.linpro.no
 http://projects.linpro.no/mailman/listinfo/varnish-misc


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: varnish storage tuning

2009-06-11 Thread Darryl Dixon - Winterhouse Consulting
Hi Mike,

Yes. If all you want to do is just get rid of the object, then all you
need to do is set obj.ttl = 0 (as per your example below). In my case I
wanted to honour the Cache-Control: no-cache header, which involved
re-fetching the object from the backend. If you simply want to remove the
item, then definitely you don't need to restart.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

 Does restart cause the backend to be queried for the page again?  I'd
 prefer
 if a purge did not re-fetch the URL from the backend as this can caused
 unwanted content to stay in the cache and my cache is relatively limited
 in
 size.  Can I do something like:
 sub vcl_hit {
 if (req.request == PURGE) {
 set obj.ttl = 0;
 error 200 Purged;
 }
 }

 sub vcl_miss {
 if (req.request == PURGE) {
 error 200 Purged (not in cache);
 }
 }


 On Thu, Jun 11, 2009 at 4:46 PM, Darryl Dixon - Winterhouse Consulting 
 darryl.di...@winterhouseconsulting.com wrote:


 Instead, in vcl_hit, if the object should be purged, set obj.ttl to 0
 and
 then restart the request. This solved the problem for me.




___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish 2.0.3 consuming excessive memory

2009-04-15 Thread Darryl Dixon - Winterhouse Consulting
 | My understanding of using 'pass' in vcl_hit is that it causes the
 object
 | to be marked as 'hit for pass' and therefore is uncacheable from that
 | point onwards. In fact, pass'ing out of vcl_hit was what I used to do
 a
 | while ago, but it caused that exact behaviour (items would no longer
 be
 | cached by varnish) so I swapped it for purge_url...

 You can try setting obj.ttl = 0s and then restart in vcl_hit.  I believe
 that should work.


 Thanks Tollef - I will try this today and report back.

This seems to correct the problem - thanks Tollef!

Further to root cause of the original issue: using purge_url seems to
result inside EXP_Rearm with the objexp (object expire?) being added
unconditionally to the exp_heap binheap, and it's not clear to me where
(if ever) this heap is pruned. For example, I see in varnishlog multiple
calls to the same URL from the same client with the same headers resulting
in multiple unique entries being added to exp_heap, eg:
ExpBan   c 981217322 was banned
[...]
ExpBan   c 981217323 was banned
[...etc...]

Where, if anywhere, is exp_heap pruned for entries like this? I couldn't
quite make sense of exp_timer() to be certain of under what conditions it
actually removes things from this heap - it looks like that function is
mainly concerned with items expiring from the Expires http header or
similar...

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com



___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish 2.0.3 consuming excessive memory

2009-04-14 Thread Darryl Dixon - Winterhouse Consulting
 | My understanding of using 'pass' in vcl_hit is that it causes the object
 | to be marked as 'hit for pass' and therefore is uncacheable from that
 | point onwards. In fact, pass'ing out of vcl_hit was what I used to do a
 | while ago, but it caused that exact behaviour (items would no longer be
 | cached by varnish) so I swapped it for purge_url...

 You can try setting obj.ttl = 0s and then restart in vcl_hit.  I believe
 that should work.


Thanks Tollef - I will try this today and report back.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish 2.0.3 consuming excessive memory

2009-04-08 Thread Darryl Dixon - Winterhouse Consulting
Hi DES,

 Darryl Dixon - Winterhouse Consulting
 darryl.di...@winterhouseconsulting.com writes:
 +if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
 .*no-cache.*) {
 +purge_url(regsub(req.url, [?].*$, .*$));
 +}
 +

 It would be interesting to see how often this condition is true...


Shouldn't be too often in this environment - I have watched with
varnishlog and the rate is very low...

 Is there possibly something in the regsub engine being triggered that is
 very expensive and would cause it to consume and hold on to large
 amounts
 of memory?

 I don't think so.  However, your code will add an entry to the ban list
 every time a request that matches the above condition is received, and
 this entry will remain on the list until every object that was in the
 cache before the entry was added has been requested at least once.


That sounds like a pretty good candidate then, as I can envisage that
there are possibly many items that may fit that criteria in my scenario.
However it definitely seems out of proportion to the actual cache
utilisation (only 25MB of objects in the cache, but memory growing to
multiple GB)

 If you really want no-cache to purge the requested object, you should do
 it in vcl_hit (set obj.ttl to 0 and pass).  Otherwise, you should simply
 pass directly from vcl_recv.


My understanding of using 'pass' in vcl_hit is that it causes the object
to be marked as 'hit for pass' and therefore is uncacheable from that
point onwards. In fact, pass'ing out of vcl_hit was what I used to do a
while ago, but it caused that exact behaviour (items would no longer be
cached by varnish) so I swapped it for purge_url...

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish 2.0.3 consuming excessive memory

2009-04-08 Thread Darryl Dixon - Winterhouse Consulting
 hi darryl

 i had a simlar problem (varnish child process consuming lots of memory) a
 week
 ago (see the thread make varnish don't start a subprocess)

 the solution for my problem seems to be to use a high-enough cache-size:

 starting varnish with

   -s file,/tmp/storage,300M

 will make the child process use *about* 800M RAM (the RES column in top)
 after
 some hours. after that i needed to restart varnish to free some ram for my
 webserver.


 starting with a bigger cache

   -s file,/tmp/storage,600M

 makes it using about 500M RAM (the RES column in top after 5 days of
 uptime)
 and happily coexist with zope ;-)



 maybe this helps you too.


Hi Harald,

That is a very interesting observation - I will give it a try.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Varnish 2.0.3 consuming excessive memory

2009-04-07 Thread Darryl Dixon - Winterhouse Consulting
Hi All,

I have an odd problem that I have only noticed happening since moving from
1.1.2 to 2.0.3 - excessive memory consumption of the varnish child
process. For example, I have a varnish instance with a 256MB cache
allocated, that is currently consuming 4.9GB of resident memory (6.5GB
virtual). The instance has only been running for 4 days and has only got
25MB of objects in the cache.

This is clearly excessive and is causing us some serious problems in terms
of memory pressure on the machine. Our VCL is largely unchanged from our
1.1.2 setup to the 2.0.3 except for the obvious vcl.syntax changes, and
the introduction of request restarts under certain limited scenarios. Can
anyone shed some light?

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish 2.0.3 consuming excessive memory

2009-04-07 Thread Darryl Dixon - Winterhouse Consulting
Further to this, I just ran the pmap on the runaway process again (roughly
15 minutes since last time), and the large ~2GB allocation has jumped by
about 60MB in that time, eg:

--
3940:   /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080
web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600
-f /u01/data/prd-edit/varnish/vcl.conf -s
file,/u01/data/prd-edit/varnish/storage.cache,256M -n
/u01/data/prd-edit/varnish/state -P
/u01/data/prd-edit/varnish/varnishd.pid
Address   Kbytes RSSAnon  Locked Mode   Mapping
0040 312   -   -   - r-x--  varnishd
0064d000   8   -   -   - rw---  varnishd
0064f000  68   -   -   - rw---[ anon ]
4000   4   -   -   - -[ anon ]
40001000   10240   -   -   - rw---[ anon ]
[...snip...]
2aaab090  262144   -   -   - rw-s-  storage.cache
2aaac090  28   -   -   - r-x--  vcl.1P9zoqAU.so
2aaac09070002048   -   -   - -  vcl.1P9zoqAU.so
2aaac0b07000   8   -   -   - rw---  vcl.1P9zoqAU.so
2aaac0c02056   -   -   - rw---[ anon ]
2aaac0f02052   -   -   - rw---[ anon ]
2aaac120  233476   -   -   - rw---[ anon ]
2aaacf70 2403328   -   -   - rw---[ anon ]
7fff4b619000  84   -   -   - rw---[ stack ]
ff608192   -   -   - -[ anon ]
  --  --  --  --
total kB 3187344   -   -   -
--


...And in the time it's taken me to compose this mail it's grown by
another 20MB:
--
2aaacf70 2418688   -   -   - rw---[ anon ]
--
...so it is a pretty consistent, continuous upwards trend in terms of
memory usage...

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com



 Not that I have an answer, but I'd be curious to see the differences
 in 'pmap -x pid' output for the different children.

 --Michael


 That's a good thought - I'd almost forgotten about pmap. Anyhow, the
 'normal' process is set up with a 2GB storage cache and pmap shows pretty
 much what you would expect - the largest anonymous segment is 41MB, with
 the vast majority of allocated memory booked against storage.cache (2GB).
 The pmap for the 'runaway' process shows 256MB booked against
 storage.cache, and then one of the an anonymous allocations is 2.3GB. For
 example:


 Normal process:
 --
 3896:   /u01/app/varnish/sbin/varnishd -a web-cms1.prd.internal:8080
 web-cms.prd.internal:8080 -T web-cms1.prd.internal:18080 -t 108000 -f
 /u01/data/prd/varnish/vcl.conf -s
 file,/u01/data/prd/varnish/storage.cache,2048M -n
 /u01/data/prd/varnish/state -P /u01/data/prd/varnish/varnishd.pid
 Address   Kbytes RSSAnon  Locked Mode   Mapping
 0040 312   -   -   - r-x--  varnishd
 0064d000   8   -   -   - rw---  varnishd
 0064f000  68   -   -   - rw---[ anon ]
 4000   4   -   -   - -[ anon ]
 40001000   10240   -   -   - rw---[ anon ]
 [...snip...]
 2aaab090 2097152   -   -   - rw-s-  storage.cache
 2aab3090  28   -   -   - r-x--  vcl.1P9zoqAU.so
 2aab309070002044   -   -   - -  vcl.1P9zoqAU.so
 2aab30b06000   8   -   -   - rw---  vcl.1P9zoqAU.so
 2aab30c02056   -   -   - rw---[ anon ]
 2aab30f02056   -   -   - rw---[ anon ]
 2aab3120   41984   -   -   - rw---[ anon ]
 7fffae8f2000  88   -   -   - rw---[ stack ]
 ff608192   -   -   - -[ anon ]
   --  --  --  --
 total kB 2468512   -   -   -
 --



 Runaway process:
 --
 3940:   /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080
 web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600
 -f /u01/data/prd-edit/varnish/vcl.conf -s
 file,/u01/data/prd-edit/varnish/storage.cache,256M -n
 /u01/data/prd-edit/varnish/state -P
 /u01/data/prd-edit/varnish/varnishd.pid
 Address   Kbytes RSSAnon  Locked Mode   Mapping
 0040 312   -   -   - r-x--  varnishd
 0064d000   8   -   -   - rw---  varnishd

Re: Articles regarding Varnish, VCL and ESI

2008-12-17 Thread Darryl Dixon - Winterhouse Consulting
Hi Jason,

[...snip...]
 Making Catalyst Sites Shine with Varnish  - includes an in-depth
 exploration of VCL

   http://www.catalystframework.org/calendar/2008/14

This is a great introduction to VCL - I'm quite certain I'll be directing
newbies to it in the future for a gentle introduction.

 Site Building Hotness with Edge Side Includes  - which covers the
 benefits of using ESI and includes my JS based ESI parser that is
 useful for developing ESI sites when you aren't behind Varnish.

  http://www.catalystframework.org/calendar/2008/17

I am especially excited about the JS ESI parser - the concept is an
excellent one and it directly addresses an otherwise difficult problem -
kudos to you, I'll be giving it a spin shortly.

This whole second article in particular is excellent and very timely for
me, as I have been thinking about using Varnish's ESI support for a new
project I am working on. Currently my strategy for low-overhead dynamic
pages revolves around just-in-time ajax. I am definitely inspired now to
give it a crack in ESI instead.

Thank you very much for this indeed.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Logged-in users

2008-11-26 Thread Darryl Dixon - Winterhouse Consulting
We do both depending on scenario: we use ajax to update parts of a page
after-delivery (poor mans ESI ;) as suggested by Tim, and we also have a
custom vcl_hash that caches different copies of pages depending on various
cookies and other conditions (much as Miles suggests). Both fit depending
on the use-case.

Miles: FWIW, the CacheFu product for Plone may assist you to maximise the
caching potential of your site without too many custom tweaks to your
Varnish rules.

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com


 Another approach is to simply use a small bit of Javascript.  It's
 easy to test for the existence of the cookie in Javascript and
 set that text conditionally.

 Then you have only one copy of the page to be cached.

 The problem with the approach you've outlined here is
 that other downstream caches won't understand the difference
 (although most will simply refuse to cache any responses
 if the request had a cookie header).  Whereas the Javascript
 approach also allows downstream caches to cache everything
 efficiently.

 Tim



 On Nov 26, 2008, at 12:31 PM, Miles wrote:

 Miles wrote:
 Hi,

 I have a site where users can log in.  This sets a cookie with their
 encrypted login details, so they can be authenticated.  There are a
 small number of pages which are user-specific (change your details
 forms, etc), and these are set not to cache.

 When a user is logged in, a message is shown at the top of the page
 You
 are now logged in.  However, nothing on the page depends on the
 individual user.

 My question is, how can I organise the cache to have the most cache
 hits, given that there are effectively two versions of each page -
 one
 for logged in users, and one for anonymous users.  I want to
 specifically avoid each user having their own version of the page
 stored
 in the cache.

 Thanks in advance for any wisdom anyone can share!

 Miles

 Thanks to everyone who suggested using ESI - I may have to use this,
 but
 would quite like to avoid it, as it's useful to be able to run the app
 without varnish in front for development/testing.

 I wondered whether it was possible to use vcl_hash for my purposes, as
 follows:

 sub vcl_hash {

//hash the object with url+host
set req.hash += req.url;
set req.hash += req.http.host;

# see if the user has a cookie to indicate they are logged in
if req.http.cookie ~ '__ac=':
   set req.hash += 'authenticated';
else:
   set req.hash += 'anonymous'
hash;

 }

 Would this give me the two representations that I require for each
 page
 - or am I going down a route that will turn out bad?!  I couldn't find
 much information about vcl_hash, so I'm not sure if I'm barking up the
 wrong tree or not...

 Regards,

 Miles

 ___
 varnish-misc mailing list
 varnish-misc@projects.linpro.no
 http://projects.linpro.no/mailman/listinfo/varnish-misc

 ___
 varnish-misc mailing list
 varnish-misc@projects.linpro.no
 http://projects.linpro.no/mailman/listinfo/varnish-misc


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Strategy for large cache sets

2008-07-06 Thread Darryl Dixon - Winterhouse Consulting
 Our cache files reside on a SAN where
 we can get around 500MB/s over iSCSI.  We went that route because we
 could not get Linux to swap efficiently enough to make malloc
 feasible.

I know this is off-topic, but, I'm interested in the 500MB/s figure... Are
you on 10gbit ethernet?

Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish and hot linking

2008-07-02 Thread Darryl Dixon - Winterhouse Consulting
 I've set up varnish to handle hot linking, but I have an issue with
 varnish caching the hotlink image for the image request.

 Here's the relevant part of default.vcl:

 sub vcl_recv {

 # Add a unique header containing the client address
 remove req.http.X-Forwarded-For;
 setreq.http.X-Forwarded-For = client.ip;
  if (req.request == GET) {
  if ( !(req.http.referer ~ [a-z0-9]+.domain.com|
 yahoo.|rssbandit.org|newsgator.com|google.|search?q=cache ) 
 (req.http.host ~ ^(raw)?images.domain.)  (req.http.referer) ) {
  set req.url=http://images.domain.com/hotlink.gif;;
  pass;
  }
  lookup;
  }

  if (req.request == PURGE) {
  purge_url(req.url);
  error 200 OK!!;
  }
 }

 The problem is this,  a user hotlinks http://images.domain.com/puppy.jpeg
   and we return hotlink.gif.
 Now if a user legitimately loads puppy.jpeg he receives the cached
 copy of hotlink.gif in its place.
 I though by setting the req.url that the cached file is now
 hotlink.gif and we haven't even attempted to store puppy.jpeg in the
 cache.
 I need to legitimately serve puppy.jpeg even if it's being hotlinked
 from somewhere.
 Any help would be appreciated.

Hi Chris,

You need to add some conditions in vcl_fetch to prevent the 'hotlink'
image from being cached on its way back from being fetched from the
backend. In other words, calling 'pass' in vcl_recv ensures that the
current request will get a fresh copy from the backend, but during the
processing of that request, the fresh copy will be 'insert'-ed into the
cache during vcl_fetch, and so subsequent requests which hit the 'lookup'
in vcl_recv will see the previously cached copy. You need to perform a
check in vcl_fetch very similar to the one in vcl_recv to prevent the
'pass'-ed response from being cached.

Hope this helps,

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Reloading default.vcl

2008-06-30 Thread Darryl Dixon - Winterhouse Consulting
 In message [EMAIL PROTECTED], Skye Poier Nott
 writes
 :
Is it possible to reload /usr/local/etc/varnish/default.vcl without
restarting?

 From the docs it looks like one way to do this would be through the
telnet interface:

vcl.load some-unique-name /usr/local/etc/varnish/default.vcl
vcl.use some-unique-name

each time the default.vcl is changed, unless there's reload option I
haven't found.

 Correct.

 Afterwards you can
   vcl.discard old-unique-name

 if you want to get rid of it.


It is pretty easy to automate this all with expect(1) as well, if an
'automated' reload solution is required :)

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: cache files pass php

2008-06-27 Thread Darryl Dixon - Winterhouse Consulting
 Is there an easy vcl trick to get varnish to always pass php requests to
 the
 webserver and to cache images, css javascript, etc?

 I would like

 apache-varnish-client

 If the client requests a document, the php is run on the webserver,
 varnish
 takes it and downloads it to the client so that the apache thread can be
 used for other things. Varnish can cache the images and text files as they
 don't change.

 Thoughts?

Hi Jack,

This seems straightforward, in vcl_recv you would need something like:
sub vcl_recv {
if (req.url ~
\.(ico|gif|jpeg|jpg|png|swf|css|htc|js|bz2|gz|zip|xls|ppt|pdf|doc|rar))
{
lookup;
}
pass;
}

The idea is that items that you 'know' are cacheable get looked up in the
cache and fetched from the backend if necessary, and that everything else
otherwise gets passed straight to the backend for servicing.

(note that this is an extremely abbreviated example that probably omits
several important considerations like cookies, etc. Including, depending
on what headers, etc the backend emits, you may want some more code in
vcl_fetch to decide whether something is insert-ed into the cache or
pass-ed onwards)

Hope this helps,

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com


___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: honoring browser reload request

2008-06-24 Thread Darryl Dixon - Winterhouse Consulting
 is the
following VCL the correct solution to support browser shift-reloads to
get fresh content and actually update the cache?

sub vcl_hit {
 if (req.http.Cache-Control ~ no-cache) {
 set obj.ttl = 0s;
 pass;
 }
}


Only kind-of; IE and Mozilla send different things with a SHIFT (or CTRL)
+ Reload, we do virtually what you have outlined above, but with a slight
addition:

in vcl_recv, we shortcut and send such requests straight to lookup:

/* Honour Cache-Control: and Pragma: ... */
if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
.*no-cache.*) {
lookup;
}

...and then in vcl_recv we do similar to you:
/* Honour Cache-Control: and Pragma: ... */
if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
.*no-cache.*) {
set obj.ttl = 0s;
pass;
}


As far as I understood vcl(7), this means:
1. fetch the object from cache (implicit in the fact that we're inside
vcl_hit)
2. set ttl to 0s, expiring it (so next requests will fetch new content)
3. pass to backend (so *this* request see new content)
Does this do what I expect it to do?

Yes, this configuration does indeed do what you expect (purge the item, go
get a fresh one from the backend)

Is there a way to avoid hitting the backend twice?

You're only hitting the backend once if you're running the code from vcl_hit

Is there a way to do that in vcl_recv?

Not really, but see our shortcut above

(I guess not, since there's no obj object available yet, there)
I guess I could use purge_url(...), except I have virtual hosts and it
wants a regexp and req.url could contain some special char (could it?).

We tried with purge_url but it was just too hard and didn't want to do
what we expected. We've had good results with our configuration above.

Hope this helps,

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: honoring browser reload request

2008-06-24 Thread Darryl Dixon - Winterhouse Consulting
 in vcl_recv, we shortcut and send such requests straight to lookup:

 /* Honour Cache-Control: and Pragma: ... */
 if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
 .*no-cache.*) {
 lookup;
 }

 ...and then in vcl_recv we do similar to you:

Bother, of course I meant: ..and then in *vcl_hit* we do similar to you... :)

 /* Honour Cache-Control: and Pragma: ... */
 if (req.http.Pragma ~ .*no-cache.* || req.http.Cache-Control ~
 .*no-cache.*) {
 set obj.ttl = 0s;
 pass;
 }


regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc


Re: Varnish Questions

2008-06-24 Thread Darryl Dixon - Winterhouse Consulting
 Hello list,


Hi Jack,


 1. connection reset issues

Can't comment specifically on these, but your varnish version is very old.
1.1.2 is good and stable for us, and -trunk is apparently pretty stable,
too...

 2. caching pages
 I have updated content on my main site, and would like to clear the
 varnish
 cache,  I telnet into the management interface and run purge * (I also
 tried
 purge *. , purge .* , purge *.*) I then clear my browser cache, and
 refresh
 the page, I get the old page.  If I visit the server directly, I get the
 new
 page?


it's 'url.purge' in 1.1.2, not sure about your version.


 Below is my vcl.conf file I have comment out the cookie part because I
[...snip...]

It's important to know/remember that the vcl you write does not *replace*
the default vcl, it just runs first, unless you specifically return a
different action (eg, lookup, pass, etc). The cookie stuff you hashed out
will still run in the default vcl afterward with your current setup. See:
http://varnish.projects.linpro.no/wiki/VCLExampleDefault
...for more information on this (be aware the default presented there is
probably not correct for your current version).

 Thanks for any help someone can give.

You're welcome ;)

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc