I have two questions:
Is there any way to benchmark htdig? Performance seems to degrade as it
crawls through more and more URLs. I've tried to use db.urls as a way to
benchmark, but I realize it's too inaccurate because the URLs that are in
there were just "seen," htdig wasn't there.
If it is true that htdig slows down after a while, do any of you know anyway
to make it faster? (I honestly don't care about the searching use for htdig.
All I'm using it for is getting a list of unique URLs from a site, and
passing it through my parser)
________________________________________________________________________
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
