At 6:22 PM -0400 5/15/00, gil cohen wrote:
>Is there any way to benchmark htdig? Performance seems to degrade as
>it crawls through more and more URLs. I've tried to use db.urls as a
>way to benchmark,
Performance is not linear. So it will slow down as the number of URLs
increase. If you want to test this, I'd probably use the -v flag to
spit up a nice progress report.
>If it is true that htdig slows down after a while, do any of you
>know anyway to make it faster? (I honestly don't care about the
>searching use for htdig. All I'm using it for is getting a list of
>unique URLs from a site, and passing it through my parser)
This seems like a rather inefficient way of doing it. There are any
number of Perl scripts that will happily do this for you. Of course
they don't provide any searching functionality, which in your case
would be a "freebie."
Cheers,
--
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You will receive a message to confirm this.
