Re: [htdig3-dev] Htdig database backend

[Geoff Hutchison]

On Tue, 14 Dec 1999 [EMAIL PROTECTED] wrote:

>  I think URL state description, robots.txt content, cookies are all 
> candidates to be stored on disk. One *very* interesting feature would
> be to have a restartable crawler. htdig + ^C + htdig restart where it
> stopped. Once you store the state of your crawler in a database, you
> get that advantage. 

You can actually get some amount of restart with the -l flag contributed
by Didier Gautheron. It stores the current state of the retriever to a
file which it re-reads on next invocation.

-Geoff


------------------------------------
To unsubscribe from the htdig3-dev mailing list, send a message to
[EMAIL PROTECTED] 
You will receive a message to confirm this.