[EMAIL PROTECTED] said: > Generally, checking to see if the documents have been modified is much > quicker than re-fetching and re-parsing all documents every time.
if you want to speed up update digs even more, you could record the expiration of a document and only check for changes after it has expired and so avoid the http request entirely. Old mailing list archives, for example, never change and the http server can give those documents a very long expiration time. This would dramatically cut the work htdig performs every night or so. Roman Maeder _______________________________________________ htdig-general mailing list <[EMAIL PROTECTED]> To unsubscribe, send a message to <[EMAIL PROTECTED]> with a subject of unsubscribe FAQ: http://htdig.sourceforge.net/FAQ.html
