Fred Atkinson wrote:
After reading this, I did a search on Google using the advanced
section and listing all sites on one of my domains. They have listed URLs
that I have explicitly blocked in my robots.txt file.
It appears that Googlebot is not a well behaved Crawler.
It would be
-
From: Klaus Johannes Rusch [EMAIL PROTECTED]
To: [EMAIL PROTECTED]; Internet robots, spiders, web-walkers, etc.
robots@mccmedia.com
Sent: Friday, August 26, 2005 6:09 PM
Subject: Re: [Robots] Googlebot complaint (anyone from Google reading?)
[EMAIL PROTECTED] wrote:
Over at Simpy.com I've been
