[htdig] Maintaining a database

[Conor Stapleton]

Hi All,   Im using htdig on iis 5 using the cygwin dll. I can enter one url no problem in my conf file and htdig works fine. However when I use say 10 urls, it takes a lot of time to run. A few questions.   If htdig is ended prematurely and the restarted, does it begin from scratch, as the database it had created just gets bigger?   I hope to have 4 databases in the end, each in a language containing about 300 domains crawled in each. I have a 2 meg link from the server which im presuming is enough. I would like to update these daily or as often as possible anyway. What is my best approach to take?   Has anyone any experience as to whether i should split these tasks into maybe 16 / 32 / 64 smaller sets or urls and schedule these to run, with a htmerge at the end of each?   Thanking you in advance, Conor Stapleton