Hi Mattias, On Sun, 6 Mar 2011, Mattias Geniar wrote:
> Not sure if you've managed to test this yet, but Google seem to run with > "Accept-Encoding: gzip". Perhaps there's a problem serving the > compressed version, whereas your manual wget's don't use this > accept-encoding? You're spot on. Adding an Accept-Encoding header to my wget requests resulted in failures. The content length reported being longer than that actually retrieved. I tracked the fault down to PHP doing compression via zlib.compression. Thanks for your help. -Ronan > -----Original Message----- > >From: [email protected] > [mailto:[email protected]] On Behalf Of Ronan > Mullally > Sent: zaterdag 5 maart 2011 10:48 > To: [email protected] > Subject: Varnish returning 503s for Googlebot requests (Bug #813?) > > Hi, > > I'm a varnish noob. I've only just started rolling out a cache in front > of a VBulletin site running Apache that is currently using pound for > load > balancing. > > I'm running 2.1.5 on a debian lenny box. Testing is going well, apart > from one problem. The site runs VBSEO to generate sitemap files. > Without excpetion, every time Googlebot tries to request these files > Varnish returns a 503: > > 66.249.66.246 - - [05/Mar/2011:09:33:53 +0000] "GET > http://www.sitename.net/sitemap_151.xml.gz HTTP/1.1" 503 419 "-" > "Mozilla/5.0 (compatible; Googlebot/2.1; > +http://www.google.com/bot.html)" > > I can request these files via wget direct from the backend as well as > direct from varnish without a problem: > > --2011-03-05 09:23:39-- http://www.sitename.net/sitemap_362.xml.gz > > HTTP request sent, awaiting response... > HTTP/1.1 200 OK > Server: Apache > Content-Type: application/x-gzip > Content-Length: 130283 > Date: Sat, 05 Mar 2011 09:23:38 GMT > X-Varnish: 1282440127 > Age: 0 > Via: 1.1 varnish > Connection: keep-alive > Length: 130283 (127K) [application/x-gzip] > Saving to: `/dev/null' > > 2011-03-05 09:23:39 (417 KB/s) - `/dev/null' saved [130283/130283] > > I've reverted back to default.vcl, the only changes being to define my > own > backends. Varnishlog output is below. Having googled a bit the only > thing I've found is bug #813, but that was apparently fixed prior to > 2.1.5. Am I missing something obvious? > > > -Ronan > > > Varnishlog output > > 18 ReqStart c 66.249.66.246 63009 1282436348 > 18 RxRequest c GET > 18 RxURL c /sitemap_362.xml.gz > 18 RxProtocol c HTTP/1.1 > 18 RxHeader c Host: www.sitename.net > 18 RxHeader c Connection: Keep-alive > 18 RxHeader c Accept: */* > 18 RxHeader c From: googlebot(at)googlebot.com > 18 RxHeader c User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; > +http://www.google.com/bot.html) > 18 RxHeader c Accept-Encoding: gzip,deflate > 18 RxHeader c If-Modified-Since: Sat, 05 Mar 2011 08:40:46 GMT > 18 VCL_call c recv > 18 VCL_return c lookup > 18 VCL_call c hash > 18 VCL_return c hash > 18 VCL_call c miss > 18 VCL_return c fetch > 18 Backend c 40 sitename sitename1 > 40 TxRequest b GET > 40 TxURL b /sitemap_362.xml.gz > 40 TxProtocol b HTTP/1.1 > 40 TxHeader b Host: www.sitename.net > 40 TxHeader b Accept: */* > 40 TxHeader b From: googlebot(at)googlebot.com > 40 TxHeader b User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; > +http://www.google.com/bot.html) > 40 TxHeader b Accept-Encoding: gzip,deflate > 40 TxHeader b X-Forwarded-For: 66.249.66.246 > 40 TxHeader b X-Varnish: 1282436348 > 40 RxProtocol b HTTP/1.1 > 40 RxStatus b 200 > 40 RxResponse b OK > 40 RxHeader b Date: Sat, 05 Mar 2011 09:17:37 GMT > 40 RxHeader b Server: Apache > 40 RxHeader b Content-Length: 130327 > 40 RxHeader b Content-Encoding: gzip > 40 RxHeader b Vary: Accept-Encoding > 40 RxHeader b Content-Type: application/x-gzip > 18 TTL c 1282436348 RFC 10 1299316657 0 0 0 0 > 18 VCL_call c fetch > 18 VCL_return c deliver > 18 ObjProtocol c HTTP/1.1 > 18 ObjStatus c 200 > 18 ObjResponse c OK > 18 ObjHeader c Date: Sat, 05 Mar 2011 09:17:37 GMT > 18 ObjHeader c Server: Apache > 18 ObjHeader c Content-Encoding: gzip > 18 ObjHeader c Vary: Accept-Encoding > 18 ObjHeader c Content-Type: application/x-gzip > 18 FetchError c straight read_error: 0 > 40 Fetch_Body b 4 4294967295 1 > 40 BackendClose b sitename1 > 18 VCL_call c error > 18 VCL_return c deliver > 18 VCL_call c deliver > 18 VCL_return c deliver > 18 TxProtocol c HTTP/1.1 > 18 TxStatus c 503 > 18 TxResponse c Service Unavailable > 18 TxHeader c Server: Varnish > 18 TxHeader c Retry-After: 0 > 18 TxHeader c Content-Type: text/html; charset=utf-8 > 18 TxHeader c Content-Length: 419 > 18 TxHeader c Date: Sat, 05 Mar 2011 09:17:38 GMT > 18 TxHeader c X-Varnish: 1282436348 > 18 TxHeader c Age: 1 > 18 TxHeader c Via: 1.1 varnish > 18 TxHeader c Connection: close > 18 Length c 419 > 18 ReqEnd c 1282436348 1299316657.660784483 > 1299316658.684726000 0.478523970 1.023897409 0.000044107 > 18 SessionClose c error > 18 StatSess c 66.249.66.246 63009 6 1 5 0 0 4 2984 32012 > > > _______________________________________________ > varnish-misc mailing list > [email protected] > http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ varnish-misc mailing list [email protected] http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
