I'm having this very frustrating problem when indexing our web site.
At random times, the indexing process just halts. Sometimes it waits a
little while and then resumes, other times it just sits there.
I'm pretty sure that I know what the cause is. Our ISP isn't necessarily
the most reliable in the world and on occasion a page may time out.
But I'm kind of expecting htdig to handle this situation...
I've looked through all the configuration options and played with all the
ones that I can see may apply (server timeout, etc). I've also tried to
limit the type of page htdig looks at so it gets through the site faster.
Finally, I've tried to "nice" the command to make it go a little slower.
None of this seems to have any effect. At random, on any given page, the
process stops. As I said, sometimes it resumes, but usually the page gets
reported as "not found" or something like that. Often, the process stays
stopped. In either case, I end up with an index that I can't use.
What can I do to get the indexing process to go smoothly? Also, is there a
way to get htdig to register somewhere those pages that it wasn't able to
access, and then have it go back and try to get just those again? In our
case, I KNOW it will be able to retrieve them when it tries a second or
third time.
Thanks for any help. It's been a very frustrating week.
Adolfo "Chago" Santiago
Principle of Minimum Access: "That which is not explicitly permitted is
denied."
Public Key: http://pgpkeys.mit.edu:11371/pks/lookup?op=get&search=0x4E867630
Fingerprint: 0EDB 438E 1222 6DFD B80F 4686 484D 7312 4E86 7630
_______________________________________________
htdig-general mailing list <[EMAIL PROTECTED]>
To unsubscribe, send a message to <[EMAIL PROTECTED]> with a
subject of unsubscribe
FAQ: http://htdig.sourceforge.net/FAQ.html
