Re: [PHP] page suck attack

[Michael McGlothlin]

robert wrote:

Hi





thank you
robert

Every so often my site is "attacked" in which all URLS on my site are retrieved in the span of minutes. What is this called?? I mean what do I google for? I don't know where to begin. I'm not sure if I am going to implement such at thing but I would like to be able to research it to know my options.

Each of our websites get a page sucked about every 5 seconds from Google alone. Sometimes seems a little aggressive but I'd rather they be indexing our websites than not. Other spiders are usually more moderate in my experience. I usually make sure my robots.txt blocks downloading css, scripting, images, and files by spiders so at least all they're getting is the HTML content that is actually useful.

--
Michael McGlothlin
Southwest Plumbing Supply

smime.p7s
Description: S/MIME Cryptographic Signature