Failure cenarios?
I have come to understand that in some builds under some conditions varnish may hang or a crash. (we run 1.0.4-3el4.i386.rpm) I have now routed all our ~180 sites troug varnish, pipe by default, cached for selected hostnames. Talk a bout all ones eggs in one basket :) The way it is all set up, we have varnsih on port 80 on one IP and apache on 80 on another whch is not in use for anything directly. If anything should hapen to varnish, or we need to upgrade or anything, 3 lines to netfilter wil reroute all the trafic directly to apache. Now the question is, how do I best detect if varnsih should have a problem? Would it be reasonably reliable to just chek if the pid from /var/run/varnish.pid is running, do I need to fetch a page, or is there some better way? Gaute ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Failure cenarios?
- Gaute Amundsen [EMAIL PROTECTED] wrote: I have come to understand that in some builds under some conditions varnish may hang or a crash. (we run 1.0.4-3el4.i386.rpm) Hi Gaute, I'll just say that in my experience Varnish has proven itself to be extremely stable. We actually run 1.0.3 across the board (yes I know there are known bugs, however we do not experience them at all) and Varnish currently serves up all requests at www.startsiden.no and www.abcnyheter.no. If anything breaks, it has not so far been varnish. However our scenario is pretty different from yours, we have very few vhosts but each has a very high amount of traffic. There is little or no advanced VCL configuration at all on our sites. We're pretty close to the default. The two sites have a different setup with regards to placement of Varnish. One site runs with dedicated varnish servers, the other has varnish and apache2 on the same box. Now the question is, how do I best detect if varnsih should have a problem? Would it be reasonably reliable to just chek if the pid from /var/run/varnish.pid is running, do I need to fetch a page, or is there some better way? Well, we always monitor as high up as possible to make sure everything works on all levels. Lower level monitoring is useful too, but for pinpointing with more accuracy where the problem is. Regards -- Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS http://www.startsiden.no ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Failure cenarios?
Gaute Amundsen [EMAIL PROTECTED] writes: Now the question is, how do I best detect if varnsih should have a problem? Would it be reasonably reliable to just chek if the pid from /var/run/varnish.pid is running, do I need to fetch a page, or is there some better way? I would recommend retrieving a page (or a set of pages). Simply checking the pid won't help you if Varnish has gone off into la-la land, or been SIGSTOPped or something. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish Dirty Caching
Christoph [EMAIL PROTECTED] writes: So what is dirty caching and why use it? Think of a very unreliable backend. If varnish can't reach it's backend, it will simply return the last content it has (even if the content is stale). That way i can cover hickups. It's on our list for 2.0, and will probably hit trunk in late July. DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish Dirty Caching
- Dag-Erling Smørgrav [EMAIL PROTECTED] wrote: Christoph [EMAIL PROTECTED] writes: So what is dirty caching and why use it? Think of a very unreliable backend. If varnish can't reach it's backend, it will simply return the last content it has (even if the content is stale). That way i can cover hickups. It's on our list for 2.0, and will probably hit trunk in late July. Way cool, really looking forward to that feature! As I stated in a different thread it's more often other parts of the system that break than varnish so having such a failsafe enables us to do a lot more fault tolerant setups. Regards -- Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS http://www.startsiden.no ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Problem with varnish and caching
- Anup Shukla [EMAIL PROTECTED] wrote: Manuel Amador (Rudd-O) wrote: site is cached according to Varnish default policies. You have not provided a single counterexample or a single snippet of VCL that could solve the problems I have, or a single snippet of VCL that you guys are actually using on production servers. man vcl does provide an example about how to cache even if Cookies are present. Did i miss something? Yeah, I think you missed the fact that Manuel isn't really interested in anything else than spewing out insults and bulls*** on the list. Nothing to see / hear here people move on, just an angry troll. Regards -- Denis Braekhus - Teknisk Ansvarlig ABC Startsiden AS http://www.startsiden.no ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish Dirty Caching
In message [EMAIL PROTECTED], Christoph writes: i'd like to implement dirty-caching using varnish. I'm busy twisting the variable visibility in VCL into proper shape right now, and that will move us a bit closer to what your want to do. The critical question is how we define backend is down and how fast and efficient we can detect it. Ideas for how to express it in VCL are very welcome. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Varnish Dirty Caching
Poul-Henning Kamp wrote: The critical question is how we define backend is down and how fast and efficient we can detect it. Right. I tend to like the Perlbal approach: Issue a http OPTIONS and check if we get anything back from the backend. It is quite lightweight. /Anton ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Config change.
I need to add more backend servers and req.http.Host definitions quite often, is there a way to change the config without restarting varnish and loose the entire cache? Or is it just simpler to have a backend on loopback and write a separate app to handle the backend distribution without having to restart varnish and loose the cache. This might not be a problem for most uses but i plan to store around 6TB in the cache on each box, so i would realy like not having to refill it each time i add a new site to the caches =) also is it possible to make varnish listen on sevral adresses/ports, i tried with mutiple -a x.x.x.x:y but it did'nt seem to work. // Fredrik Neij ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Config change.
In message [EMAIL PROTECTED], TiAMO writes: I need to add more backend servers and req.http.Host definitions quite often, is there a way to change the config without restarting varnish and loose the entire cache? Yes, you can load a new VCL code from the CLI interface with no disruption to your active traffic. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: logging problems - Pipe Shut
Gaute Amundsen [EMAIL PROTECTED] writes: We have had logging running for about a week now with no apparent problems, but yesterday I routed all our traffic into varnish, and now bad things happen. /var/log/varnish/varnish.log just stops growing after a while and a tail -n 1 gives me pages after pages of: Pipe Shut write(read)Pipe Shut read(read)Pipe Shut write(read)Pipe Shut read(read)Pipe Shut write(read)Pipe Shut read(read)Pipe Shut write(read)Pipe Over 4 megs of it just now actualy. It's not a text file, so tailing it is meaningless. Have you verified that logrotate is correctly set up? DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: logging problems - Pipe Shut
Hm.. I was finding quite a bit of Pipe Shut just running varnishlog -o. I's out of my buffer, so I cant paste it in right now, but could it bee that I was opening to many pipes? I the default action in vcl_recv was pipe, and only a few hosts would get a lookup... Trying with pass now, and it has held upp longer than last time anyway. Gaute ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: logging problems - Pipe Shut
Gaute Amundsen [EMAIL PROTECTED] writes:
I was finding quite a bit of Pipe Shut just running varnishlog -o.
I's out of my buffer, so I cant paste it in right now, but could it
bee that I was opening to many pipes?
pipe shut happens when either the backend or the client closes the
connection, although there is a possibility that the code (rdf() in
cache_pipe.c) is too sensitive and shuts down the pipe as soon as the
client's TCP window fills up. Could you try the attached patch?
DES
--
Dag-Erling Smørgrav
Senior Software Developer
Linpro AS - www.linpro.no
Index: bin/varnishd/cache_pipe.c
===
--- bin/varnishd/cache_pipe.c (revision 1627)
+++ bin/varnishd/cache_pipe.c (working copy)
@@ -50,7 +50,7 @@
rdf(struct pollfd *fds, int idx)
{
int i, j;
- char buf[BUFSIZ];
+ char buf[BUFSIZ], *p;
i = read(fds[idx].fd, buf, sizeof buf);
if (i = 0 || fds[1-idx].events == 0) {
@@ -60,13 +60,16 @@
shutdown(fds[1-idx].fd, SHUT_WR);
fds[idx].events = 0;
} else {
- j = write(fds[1-idx].fd, buf, i);
- if (i != j) {
- VSL(SLT_Debug, fds[idx].fd, Pipe Shut write(write));
- VSL(SLT_Debug, fds[1-idx].fd, Pipe Shut read(write));
- shutdown(fds[idx].fd, SHUT_WR);
- shutdown(fds[1-idx].fd, SHUT_RD);
- fds[1-idx].events = 0;
+ for (p = buf; i 0; i -= j, p += j) {
+ j = write(fds[1-idx].fd, p, i);
+ if (j 0) {
+VSL(SLT_Debug, fds[idx].fd, Pipe Shut write(write));
+VSL(SLT_Debug, fds[1-idx].fd, Pipe Shut read(write));
+shutdown(fds[idx].fd, SHUT_WR);
+shutdown(fds[1-idx].fd, SHUT_RD);
+fds[1-idx].events = 0;
+break;
+ }
}
}
}
___
varnish-misc mailing list
varnish-misc@projects.linpro.no
http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: logging problems - Pipe Shut
On Tuesday 03 July 2007 16:05, Dag-Erling Smørgrav wrote: Gaute Amundsen [EMAIL PROTECTED] writes: Rotation is weekly, and the previous logs have sane dates. Weekly rotation is probably far too seldom, Varnish can easily generate several gigabytes of log data *per hour* under high load. The tought have struck me yes. What would you suggest? Daily ought to stay under 5G by guesstimate, or would it be better to rotate by size? What size is varnishncsa comfortable with? Have you checked the file size limit (ulimit -f) that varnishlog operates under? unlimited Gaute ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: logging problems - Pipe Shut
Gaute Amundsen [EMAIL PROTECTED] writes: On Tuesday 03 July 2007 16:05, Dag-Erling Smørgrav wrote: Gaute Amundsen [EMAIL PROTECTED] writes: Rotation is weekly, and the previous logs have sane dates. Weekly rotation is probably far too seldom, Varnish can easily generate several gigabytes of log data *per hour* under high load. The tought have struck me yes. What would you suggest? Daily ought to stay under 5G by guesstimate, or would it be better to rotate by size? What size is varnishncsa comfortable with? varnishncsa shouldn't care, as it processes the log file linearly, but I generally prefer to rotate by size. Is this a 32-bit machine, BTW? DES -- Dag-Erling Smørgrav Senior Software Developer Linpro AS - www.linpro.no ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: logging problems - Pipe Shut
On Tuesday 03 July 2007 16:30, Dag-Erling Smørgrav wrote: snip varnishncsa shouldn't care, as it processes the log file linearly, but I generally prefer to rotate by size. Hm.. ok. But that would have me running awstats at odd times.. Wil just have to try it out I guess Is this a 32-bit machine, BTW? It is. Quad Xeon 3 GHz. Could you try the attached patch? Having some problems with that. My fault I'm sure, but I installed from the RPM originaly, so I can't just re-run. Compilation goes fine, but the binaries ends up widely different sizes. (5567 vs 168684) before make install that is. After that it's 478984 vs 168684. The source rpms are no different i presume? Is there a way to get the binary made without doing an install? I'd rather not do that on our production box.. Gaute ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: Failure cenarios?
James Quacinella wrote: I use monit for monitoring programs. Here is a snippet I had used when monitoring a varnish install (too bad it never went into production; change values to you liking / environment): A Billion thanks to you :) I have been looking for a tool that could do this for me. Regards A.S ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
