In my htdig configuration, the start_url directive is set to a file which contains links to all the files I'd like to have indexed. How can I prevent htdig from following the links in the other documents, without adding a robots meta tag to every file?
Thanks, Claus (I'm (still) using htdig 3.1.5) _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
