>Brings up an interesting point I've been sort of researching. What's
>the best way to write up a robots.txt file that allows the local htdig
>to index, but nobody else? I know using a user-agent of htdig should
>allow it to index, but seems to allow remote htdig's also. I guess I
>could disable robot.txt looking in my local copy, but even though I
>trust myself not to abuse that...
Well you can set the robotstxt_name that htdig uses when it parses the
robots.txt file. So try something like:
robotstxt_name: plaidworks-htdig
Then use "plaidworks-htdig" in your robots.txt file for your "local" htdig
and you're set.
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
