Re: [htdig] Going for the big dig

[Geoff Hutchison]

At 10:14 AM +1100 12/19/00, Terry Collins wrote:
>And make sure you don't ignore robots.txt

Yes, though someone would need to alter the code to do this.

--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/

------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
List archives:  <http://www.htdig.org/mail/menu.html>
FAQ:            <http://www.htdig.org/FAQ.html>