We did some experimentation with saint mode before deciding not to turn it on because we found that we'd be opening ourselves up to exactly this issue.
Once the saint mode threshold is reached for a backend then that backend won't be contacted for anything at all and Varnish will return a 503. It can't return the stale content if it's not in the cache. On 5 Nov 2011, at 11:00, [email protected] wrote: > Send varnish-misc mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of varnish-misc digest..." > > > Today's Topics: > > 1. Re: Intermittent 503's from Varnish (Lei Wang) > 2. Re: Intermittent 503's from Varnish (AD) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 4 Nov 2011 13:26:50 -0700 > From: Lei Wang <[email protected]> > To: "Gresens, August" <[email protected]> > Cc: [email protected] > Subject: Re: Intermittent 503's from Varnish > Message-ID: > <cad_korcxrk3jrtanup8_fgxkt6rwmhhcxqxnpa2wolwjcet...@mail.gmail.com> > Content-Type: text/plain; charset="windows-1252" > > Hi August, > > Thank for the info. I will definitely try that today. > > Have a nice day. > > Lei > > On Fri, Nov 4, 2011 at 12:48 PM, Gresens, August > <[email protected]>wrote: > >> We were able to resolve this. Our Varnish instance is pointing a load >> balanced pool of backend servers (a vip on a hardware load balancer), so >> to Varnish there is only one backend. We had a situation in which certain >> images were returning 500 errors from the backend. Because saint mode was >> enabled, it invalidated the backend (the only back end) after a certain >> number of failures within a particular window. We disabled saint mode and >> this fixed this issue. Best, August**** >> >> ** ** >> ------------------------------ >> >> *From:* Lei Wang [mailto:[email protected]] >> *Sent:* Friday, November 04, 2011 3:40 PM >> *To:* Rob S >> *Cc:* Gresens, August; [email protected] >> *Subject:* Re: Intermittent 503's from Varnish**** >> >> ** ** >> >> I had the same problem. Please help! >> >> /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f >> /etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u >> varnish -g varnish -i varnish81 -S /etc/varnish/secret -p >> thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p >> thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n >> varnish80 -s malloc,6.5G >> >> configuration related to this problem: >> backend eadrm { >> .host = "drm-ea.xxxxxxxx.com"; >> .port = "80"; >> .connect_timeout = 40s; >> .first_byte_timeout = 80s; >> .between_bytes_timeout = 40s; >> } >> >> } elsif (req.http.host ~ "drm-ea") { >> unset beresp.http.set-cookie; >> set beresp.ttl = 86400s; >> log "Adjusting EA-DRM TTL to 86400 seconds"; >> >> Thanks, >> >> Lei**** >> >> On Fri, Nov 4, 2011 at 11:52 AM, Rob S <[email protected]> wrote:**** >> >> Can you send the varnishlog for one of these failing requests? Is >> this a POST or a GET request? Are you using default VCL, or can you >> send the VCL you are using? >> >> Rob**** >> >> On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <[email protected]> >> wrote: >>> Hello >>> >>> >>> >>> We?re having a very nasty production issue. We?re getting intermittent >> 503 >>> from Varnish in front of a healthy backend. >>> >>> >>> >>> This seems to happen on cache-miss, but we?re not seeing requests make >> it to >>> the backend. Varnish immediately returns the 503. After the first 503, >> the >>> page can usually be loaded by requesting it a few more times. This seems >> to >>> indicate that there is some condition within Varnish that prevents it >> from >>> hitting the backend that clears up quickly. >>> >>> >>> >>> Has anyone seen this issue before? Is there anything we can try? >>> >>> >>> >>> This is our startup config: >>> >>> >>> >>> DAEMON_OPTS="-a :80 \ >>> >>> -T localhost:6082 \ >>> >>> -f /etc/varnish/default.vcl \ >>> >>> -S /etc/varnish/secret \ >>> >>> -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \ >>> >>> -p http_headers=128 \ >>> >>> -p sess_workspace=262144 \ >>> >>> -p connect_timeout=60 \ >>> >>> -p max_restarts=6 \ >>> >>> -p log_hashstring=on \ >>> >>> -p cli_timeout=60 \ >>> >>> -t 0" >>> >>> >>> >>> We are on version 2.0.5 >>> >>> >>> >>> Thanks, >>> >>> >>> >>> August >>> >>> >>> >>> >>> **** >> >>> _______________________________________________ >>> varnish-misc mailing list >>> [email protected] >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> _______________________________________________ >> varnish-misc mailing list >> [email protected] >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc**** >> >> ** ** >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20111104/48abd99e/attachment-0001.html> > > ------------------------------ > > Message: 2 > Date: Fri, 4 Nov 2011 16:33:51 -0400 > From: AD <[email protected]> > To: Lei Wang <[email protected]> > Cc: [email protected] > Subject: Re: Intermittent 503's from Varnish > Message-ID: > <canzwjw4jvirfxhlftwu2fwbcppkz2ybi1cwrb4xqchhtbrv...@mail.gmail.com> > Content-Type: text/plain; charset="windows-1252" > > Shouldnt saint mode return a 200 OK (stale cache item) if the backend fails > and you are within the "Saint Window" for that object? I dont think > turning off saint mode is going to correct this issue. > > On Fri, Nov 4, 2011 at 4:26 PM, Lei Wang <[email protected]> wrote: > >> Hi August, >> >> Thank for the info. I will definitely try that today. >> >> Have a nice day. >> >> Lei >> >> >> On Fri, Nov 4, 2011 at 12:48 PM, Gresens, August >> <[email protected]>wrote: >> >>> We were able to resolve this. Our Varnish instance is pointing a load >>> balanced pool of backend servers (a vip on a hardware load balancer), so >>> to Varnish there is only one backend. We had a situation in which certain >>> images were returning 500 errors from the backend. Because saint mode was >>> enabled, it invalidated the backend (the only back end) after a certain >>> number of failures within a particular window. We disabled saint mode and >>> this fixed this issue. Best, August**** >>> >>> ** ** >>> ------------------------------ >>> >>> *From:* Lei Wang [mailto:[email protected]] >>> *Sent:* Friday, November 04, 2011 3:40 PM >>> *To:* Rob S >>> *Cc:* Gresens, August; [email protected] >>> *Subject:* Re: Intermittent 503's from Varnish**** >>> >>> ** ** >>> >>> I had the same problem. Please help! >>> >>> /usr/sbin/varnishd -P /var/run/varnish80.pid -a :80 -f >>> /etc/varnish/varnish80.vcl -T 127.0.0.1:6082 -t 120 -w 1,1000,120 -u >>> varnish -g varnish -i varnish81 -S /etc/varnish/secret -p >>> thread_pool_add_delay 2 -p thread_pools 1 -p thread_pool_min 800 -p >>> thread_pool_max 2000 -p session_linger 50 -p sess_workspace 4096 -n >>> varnish80 -s malloc,6.5G >>> >>> configuration related to this problem: >>> backend eadrm { >>> .host = "drm-ea.xxxxxxxx.com"; >>> .port = "80"; >>> .connect_timeout = 40s; >>> .first_byte_timeout = 80s; >>> .between_bytes_timeout = 40s; >>> } >>> >>> } elsif (req.http.host ~ "drm-ea") { >>> unset beresp.http.set-cookie; >>> set beresp.ttl = 86400s; >>> log "Adjusting EA-DRM TTL to 86400 seconds"; >>> >>> Thanks, >>> >>> Lei**** >>> >>> On Fri, Nov 4, 2011 at 11:52 AM, Rob S <[email protected]> wrote:**** >>> >>> Can you send the varnishlog for one of these failing requests? Is >>> this a POST or a GET request? Are you using default VCL, or can you >>> send the VCL you are using? >>> >>> Rob**** >>> >>> On Fri, Nov 4, 2011 at 6:44 PM, Gresens, August <[email protected]> >>> wrote: >>>> Hello >>>> >>>> >>>> >>>> We?re having a very nasty production issue. We?re getting intermittent >>> 503 >>>> from Varnish in front of a healthy backend. >>>> >>>> >>>> >>>> This seems to happen on cache-miss, but we?re not seeing requests make >>> it to >>>> the backend. Varnish immediately returns the 503. After the first 503, >>> the >>>> page can usually be loaded by requesting it a few more times. This >>> seems to >>>> indicate that there is some condition within Varnish that prevents it >>> from >>>> hitting the backend that clears up quickly. >>>> >>>> >>>> >>>> Has anyone seen this issue before? Is there anything we can try? >>>> >>>> >>>> >>>> This is our startup config: >>>> >>>> >>>> >>>> DAEMON_OPTS="-a :80 \ >>>> >>>> -T localhost:6082 \ >>>> >>>> -f /etc/varnish/default.vcl \ >>>> >>>> -S /etc/varnish/secret \ >>>> >>>> -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G \ >>>> >>>> -p http_headers=128 \ >>>> >>>> -p sess_workspace=262144 \ >>>> >>>> -p connect_timeout=60 \ >>>> >>>> -p max_restarts=6 \ >>>> >>>> -p log_hashstring=on \ >>>> >>>> -p cli_timeout=60 \ >>>> >>>> -t 0" >>>> >>>> >>>> >>>> We are on version 2.0.5 >>>> >>>> >>>> >>>> Thanks, >>>> >>>> >>>> >>>> August >>>> >>>> >>>> >>>> >>>> **** >>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> [email protected] >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> [email protected] >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc**** >>> >>> ** ** >>> >> >> >> _______________________________________________ >> varnish-misc mailing list >> [email protected] >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20111104/8fe01fe8/attachment-0001.html> > > ------------------------------ > > _______________________________________________ > varnish-misc mailing list > [email protected] > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > End of varnish-misc Digest, Vol 68, Issue 9 > ******************************************* http://www.bbc.co.uk/ This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. _______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
