In the directives list, this is explained...
robotstxt_name
type: string
used by: htdig
default: htdig
description: Sets the name that htdig will look for when parsing robots.txt files. This
can be used to make htdig appear as a different spider than ht://Dig. Useful to
distinguish between a private and a global index.
example: robotstxt_name: myhtdig
However, I am still not clear on this because of the mention os what htdig can be made
to
look like. Is this directive so that htdig will know to look for robots.txt or a custom
name file that does the same. Or, is this to make our digger look like something else?
Mike
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
FAQ: <http://www.htdig.org/FAQ.html>
