Indra,
Have extensively used in the past. Quite well
respected, bots that don't follow it invariably end up
on deny lists. Have miles of logs where the first
thing a bot does a GET on is robots.txt
Arup
--- Indranil Das Gupta <[EMAIL PROTECTED]>
wrote:
> Hi all,
>
> I know this is kinda off-topic, but I was wondering
> if someone could help me by sharing their wisdom...
> I am interested to know if anyone here has been
> using the Robot Exclusion Standard and its
> usefulness/effectiveness....
>
> TIA,
> --Indra.
>
>
> --
> To unsubscribe, send mail to [EMAIL PROTECTED]
> with the body
> "unsubscribe ilug-cal" and an empty subject line.
> FAQ: http://www.ilug-cal.org/help/faq_list.html
__________________________________________________
Do You Yahoo!?
Kick off your party with Yahoo! Invites.
http://invites.yahoo.com/
--
To unsubscribe, send mail to [EMAIL PROTECTED] with the body
"unsubscribe ilug-cal" and an empty subject line.
FAQ: http://www.ilug-cal.org/help/faq_list.html
