On Thu, 2006-09-28 at 08:02 +1200, Volker Kuhlmann wrote:
> Anyone else finding that google DoSes servers on a regular basis?
> Downloading the same pdf as fast as they can. If it's only 30 times it's
> lucky, bringing the net plan over quota is not unheard of.
Just Seeing Steve's reply to this... Using a dynamic dns might be part
of the issue there? or is your cable IP static?
You can also set the expiry for document types in your apache config per
file type.. Google (and other bots I assume) use a transproxy(s) so
that their server farms can access objects from your site. If a given
file has a very short ttl it may re-request the file multiple times from
multiple machines in the farm while trying to index the pdf..
Try this below, can't remember where I got it from, but I use this on a
couple of sites, brings the traffic down by 30-40% as ISP transproxies
honour it.
<---------------------Blah Blah Blah ------------------->
ExpiresActive On
ExpiresDefault "access plus 300 seconds"
<Directory "/home/website/public_html">
ExpiresByType text/html "access plus 1 day"
ExpiresByType text/css "access plus 1 day"
ExpiresByType text/javascript "access plus 1 day"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/jpg "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
</Directory>
<---------------------Blah Blah Blah ------------------->
Cheers, Chris H.
