On Thu, 6 Dec 2001 [EMAIL PROTECTED] wrote: > I think you've answered me already, but this question was suppose to > be directed at excluding all engines except htdig (?). > > There is no other way (the last response threw me a bit) to do that > except the robots.txt -is that correct?
That's correct. Moreover, some engines seem to ignore the robots.txt file completely. So unless you have other means of protecting the server (e.g. requiring authorization or using local_urls indexing), there's little you can do to exclude other engines. -- --Geoff Hutchison Williams Students Online http://wso.williams.edu/ _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
