hi list,
i have a spidering script that pulls links/urls from some of our sites
using LWP
while doing this i store the urls in hashes like this:
$ok_in_site{$url} = $response;
$not_ok_in_site{$url} = $response;
$ok_out_site{$url} = $response;
$not_ok_out_site{$url} = $response;
$url = up to 255 characters
$response = about 15 characters
i do really need to store those values somehow, because eventually
these need to be sorted for their final html-output
all this is behaving very well and fast on sites with few links (less
than 2000), but when i run the script on a large site (more than 8000
unique links) i can see the memory is increasing dramtically, so much
that i can't even stop the program normally.
so what i'm looking for is some advice. i am thinking off two
alternatives for the above hashes:
a) store the urls / responses in a temp txtfile
b) store the urls / responses in a temp db-table
c) someone with a better idea
i prefer a) of the two first because of the systems involved [sql
server on winXP] but if performance really is much better using a
database then be it.
../allan
