>Lycos' spider's agent name is Lycos_Spider_(T-Rex)
>and its host name is: *.sjc.lycos.com (e.g. sjc-fe4-1.sjc.lycos.com)
>
>A great table that lists most of the majors is available at:
>http://searchenginewatch.com/webmasters/spiderchart.html
>
>If you have a robot.txt file on your site you could grep out robot.txt
>requests which would tell you which robots are looking and enable you
>to remove the offending hits.
Or you could just do a Browser Report, with FILEINCLUDE /robots.txt
Aengus
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/[email protected]/
------------------------------------------------------------------------
