Fred Atkinson wrote:
After reading this, I did a search on Google using the advanced
section and listing all sites on one of my domains. They have listed URLs
that I have explicitly blocked in my robots.txt file.
It appears that Googlebot is not a well behaved Crawler.
It would be helpful if you included some examples.
Just guessing, Google does include pages in search results that have not
actually crawled but identified based on links from other sites.
You can identify these by the fact that they do not show details, such
as an extract from the page.
--
Klaus Johannes Rusch
[EMAIL PROTECTED]
http://www.atmedia.net/KlausRusch/
_______________________________________________
Robots mailing list
Robots@mccmedia.com
http://www.mccmedia.com/mailman/listinfo/robots
