Re: Memory spreading, then stop responding
Thanks Darryl. However, I don't think this solution will work in our usage. We're running a blog. Administrators get un-cached access, straight through varnish. Then, when they publish, we issue a purge across the entire site. We need to do this as there's various bits of navigation that'd need to be updated. I can't see that we can do this if we set obj.ttl. Has anyone any recommendations as to how best to deal with purges like this? >>> >>> If you're issuing a PURGE across the entire site, why not simply >>> restart Varnish with an empty cache? >>> >>> --Michael >>> >> Because Varnish is also working for other hosts which don't need >> purging at the same time... > > My company gets around this madness by versioning its URLs. It works > pretty well. > > --Michael Thanks. Are there any varnish developers who can comment on this memory-usage-growth-when-purging? I can't see any open tickets for this, and I'm sure there are several mailing list members who might like to contribute a bounty for development of a fix to this. Rob ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory spreading, then stop responding
On Jul 28, 2009, at 3:09 PM, Rob S wrote: > Michael S. Fischer wrote: >> On Jul 28, 2009, at 2:35 PM, Rob S wrote: >>> Thanks Darryl. However, I don't think this solution will work in >>> our >>> usage. We're running a blog. Administrators get un-cached access, >>> straight through varnish. Then, when they publish, we issue a purge >>> across the entire site. We need to do this as there's various >>> bits of >>> navigation that'd need to be updated. I can't see that we can do >>> this >>> if we set obj.ttl. >>> >>> Has anyone any recommendations as to how best to deal with purges >>> like this? >> >> If you're issuing a PURGE across the entire site, why not simply >> restart Varnish with an empty cache? >> >> --Michael >> > Because Varnish is also working for other hosts which don't need > purging at the same time... My company gets around this madness by versioning its URLs. It works pretty well. --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory spreading, then stop responding
Michael S. Fischer wrote: > On Jul 28, 2009, at 2:35 PM, Rob S wrote: >> Thanks Darryl. However, I don't think this solution will work in our >> usage. We're running a blog. Administrators get un-cached access, >> straight through varnish. Then, when they publish, we issue a purge >> across the entire site. We need to do this as there's various bits of >> navigation that'd need to be updated. I can't see that we can do this >> if we set obj.ttl. >> >> Has anyone any recommendations as to how best to deal with purges >> like this? > > If you're issuing a PURGE across the entire site, why not simply > restart Varnish with an empty cache? > > --Michael > Because Varnish is also working for other hosts which don't need purging at the same time... Rob ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory spreading, then stop responding
On Jul 28, 2009, at 2:35 PM, Rob S wrote: > Thanks Darryl. However, I don't think this solution will work in our > usage. We're running a blog. Administrators get un-cached access, > straight through varnish. Then, when they publish, we issue a purge > across the entire site. We need to do this as there's various bits of > navigation that'd need to be updated. I can't see that we can do this > if we set obj.ttl. > > Has anyone any recommendations as to how best to deal with purges > like this? If you're issuing a PURGE across the entire site, why not simply restart Varnish with an empty cache? --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory spreading, then stop responding
Darryl Dixon - Winterhouse Consulting wrote: >> Darryl Dixon - Winterhouse Consulting wrote: >> Can anyone suggest why varnish is using more memory than it's allocated, and why varnishlog would stop returning any output? Varnishlog was writing to disk, so I can probably extract the end of that, if it's of use. >>> Hi Rob, >>> >>> There have been a few threads about this now on this mailing list. >>> Probably it relates to the use of purge_url in your VCL. Are you using >>> this function at all? >>> >>> regards, >>> Darryl >>> >> Darryl, >> >> Thanks for your reply. Yes we are using purge_url, but I was under the >> impression that since http://varnish.projects.linpro.no/changeset/3329, >> there wasn't a problem. I've not succeeded in finding the threads you >> mentioned in your email. Can you either point me at them, or let me >> know their conclusion? >> >> > > Hi Rob, > > See the thread concluding here (the solution to swap purge_url for > obj.ttl=0 is the correct one): > http://projects.linpro.no/pipermail/varnish-misc/2009-April/002743.html > > And also the thread concluding here: > http://projects.linpro.no/pipermail/varnish-misc/2009-June/002840.html > > regards, > Darryl Dixon > Winterhouse Consulting Ltd > http://www.winterhouseconsulting.com > > > Thanks Darryl. However, I don't think this solution will work in our usage. We're running a blog. Administrators get un-cached access, straight through varnish. Then, when they publish, we issue a purge across the entire site. We need to do this as there's various bits of navigation that'd need to be updated. I can't see that we can do this if we set obj.ttl. Has anyone any recommendations as to how best to deal with purges like this? Rob ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory spreading, then stop responding
> Darryl Dixon - Winterhouse Consulting wrote: >>> >>> Can anyone suggest why varnish is using more memory than it's >>> allocated, >>> and why varnishlog would stop returning any output? Varnishlog was >>> writing to disk, so I can probably extract the end of that, if it's of >>> use. >>> >>> >> >> Hi Rob, >> >> There have been a few threads about this now on this mailing list. >> Probably it relates to the use of purge_url in your VCL. Are you using >> this function at all? >> >> regards, >> Darryl > Darryl, > > Thanks for your reply. Yes we are using purge_url, but I was under the > impression that since http://varnish.projects.linpro.no/changeset/3329, > there wasn't a problem. I've not succeeded in finding the threads you > mentioned in your email. Can you either point me at them, or let me > know their conclusion? > Hi Rob, See the thread concluding here (the solution to swap purge_url for obj.ttl=0 is the correct one): http://projects.linpro.no/pipermail/varnish-misc/2009-April/002743.html And also the thread concluding here: http://projects.linpro.no/pipermail/varnish-misc/2009-June/002840.html regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory spreading, then stop responding
Darryl Dixon - Winterhouse Consulting wrote: >> >> Can anyone suggest why varnish is using more memory than it's allocated, >> and why varnishlog would stop returning any output? Varnishlog was >> writing to disk, so I can probably extract the end of that, if it's of >> use. >> >> > > Hi Rob, > > There have been a few threads about this now on this mailing list. > Probably it relates to the use of purge_url in your VCL. Are you using > this function at all? > > regards, > Darryl Darryl, Thanks for your reply. Yes we are using purge_url, but I was under the impression that since http://varnish.projects.linpro.no/changeset/3329, there wasn't a problem. I've not succeeded in finding the threads you mentioned in your email. Can you either point me at them, or let me know their conclusion? Thanks, Rob ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Memory spreading, then stop responding
> Hi, > > Here's my setup: > > [r...@varnish1 ~]# rpm -qa |grep varnish > varnish-libs-2.0.4-1.el5 > varnish-2.0.4-1.el5 > [r...@varnish1 ~]# uname -a > Linux varnish1.example.com 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST > 2009 x86_64 x86_64 x86_64 GNU/Linux > [r...@varnish1 ~]# ps aux |grep varnishd > root 27993 0.0 0.0 106472 816 ?Ss 17:42 0:00 > /usr/sbin/varnishd -P /var/run/varnish.pid -a 10.1.2.51:80 -T :6082 -f > /etc/varnish/default.vcl -u varnish -g varnish -s > file,/var/lib/varnish/varnish_storage.bin,1G > varnish 28063 0.9 1.0 1474728 62860 ? Sl 17:43 0:06 > /usr/sbin/varnishd -P /var/run/varnish.pid -a 10.1.2.51:80 -T :6082 -f > /etc/varnish/default.vcl -u varnish -g varnish -s > file,/var/lib/varnish/varnish_storage.bin,1G > root 28799 0.0 0.0 61192 732 pts/3S+ 17:56 0:00 grep > varnishd > > The problem that I've encountered twice now is the following: > > 1) Varnish spreads to use over 8GB of swap, despite appearing to be > configured to only use 1GB of storage > 2) Our automated monitoring indicates that we're running out of swap > space. > 3) Restart varnish > 4) From this point, varnishlog and varnishncsa return no output. > > Can anyone suggest why varnish is using more memory than it's allocated, > and why varnishlog would stop returning any output? Varnishlog was > writing to disk, so I can probably extract the end of that, if it's of > use. > Hi Rob, There have been a few threads about this now on this mailing list. Probably it relates to the use of purge_url in your VCL. Are you using this function at all? regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Memory spreading, then stop responding
Hi, Here's my setup: [r...@varnish1 ~]# rpm -qa |grep varnish varnish-libs-2.0.4-1.el5 varnish-2.0.4-1.el5 [r...@varnish1 ~]# uname -a Linux varnish1.example.com 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 GNU/Linux [r...@varnish1 ~]# ps aux |grep varnishd root 27993 0.0 0.0 106472 816 ?Ss 17:42 0:00 /usr/sbin/varnishd -P /var/run/varnish.pid -a 10.1.2.51:80 -T :6082 -f /etc/varnish/default.vcl -u varnish -g varnish -s file,/var/lib/varnish/varnish_storage.bin,1G varnish 28063 0.9 1.0 1474728 62860 ? Sl 17:43 0:06 /usr/sbin/varnishd -P /var/run/varnish.pid -a 10.1.2.51:80 -T :6082 -f /etc/varnish/default.vcl -u varnish -g varnish -s file,/var/lib/varnish/varnish_storage.bin,1G root 28799 0.0 0.0 61192 732 pts/3S+ 17:56 0:00 grep varnishd The problem that I've encountered twice now is the following: 1) Varnish spreads to use over 8GB of swap, despite appearing to be configured to only use 1GB of storage 2) Our automated monitoring indicates that we're running out of swap space. 3) Restart varnish 4) From this point, varnishlog and varnishncsa return no output. Can anyone suggest why varnish is using more memory than it's allocated, and why varnishlog would stop returning any output? Varnishlog was writing to disk, so I can probably extract the end of that, if it's of use. Very grateful to anyone who can point me in the right direction. Rob ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
