When using a very large list of URL's to index, it can get pretty tough to keep track
of
which sites to index or not. In other words, a site that I remove from the list could
end
up back in by error and be indexed again.
I can think of a simple way to take care of that, some sort of database program to
maintain that list but would it be possible to add some smarts to htdig to take care of
that?
Just a thought.
Mike
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
List archives: <http://www.htdig.org/mail/menu.html>
FAQ: <http://www.htdig.org/FAQ.html>
