At 8:35 AM -0500 11/24/99, [EMAIL PROTECTED] wrote:
>I've got a rather large page (over 200,000 links to very simple pages with
>no links), that I am trying to index. Whenever I run htdig against it, I
>eventually get the error "virtual memory exceeded in 'new'".
>
>I really don't want to split the page into lots of smaller ones because it's
>dynamically generated from a database so... is there any way I can index
>these files without getting that error?
Yes, but only if you increase the amount of swap space...
Here's the problem--in order to do the indexing, htdig must assemble
a TODO list of links. Since indexing that page just added 200,000
links to the TODO list, it needs to grab a bunch more memory. But
that's the problem it mentioned--it ran out of memory (in this case
swap), so it's dead.
Even splitting the links a little might help. Even better, you could
break the links into "batches" and use htmerge to merge the database
together later.
-Geoff Hutchison
Williams Students Online
http://wso.williams.edu/
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED]
You'll receive a message confirming the unsubscription.
