At 4:54 PM -0500 12/14/98, Ryan Scott wrote:
>I still cannot index geocities pages. Here's what they put in their
>robots.txt file:
>
># htdig knows where to go.
>User-agent: htdig/3.1.0b1
>I'm not too familiar with how it is all supposed to be but it appears this
>doesn't cut it. I'm trying to index various neighborhoods on request of the
>folks running those neighborhoods, in case you were a wonderin.
Either get Geocities to set the User-agent: htdig or set the robotstxt_name
option to "htdig/3.1.0b1" in your conf file. The problem is that the
robots.txt name must match *exactly* and it doesn't.
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
----------------------------------------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word "unsubscribe" in
the body of the message.
