On Tue, 1 Jan 2002, Dan Langille wrote: > I've found an instance where a document contains in robots.txt is > included in the final index. Not sure if this is a bug or a feature.
I believe I know what you're talking about. At the moment, if a document is already in the database and the robots.txt forbids it, then it will be left in the database. This could happen if you only update old databases. I don't think I'd call that a feature. Unfortunately, as things are currently set up, it's tricky to fix this. -- -Geoff Hutchison Williams Students Online http://wso.williams.edu/ _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
