Hi,
I'm currently using wget to mirror a rather extensive website. So
far I've mirrored 10 gigs and counting. wget has been running for roughly
24 hours. My problem is the huge amount of memory wget has alloted to
it. Since I have it converting links, I was wondering if maybe all the
webpages I have been mirroring are being stored in memory for later
conversion to relative links. This doesn't seem to be the case since some
links have been converted. Unforunately a lot of the links on each page
point to each other. It seems that wget may be downloading pages,
converting them, and then re-downloading them and converting them
again. My main concern is that my machine will run out of physical and
swap memory before the mirroring is complete.
Here's the output of top:
5874 jolan 2 4 44M 34M sleep select 581:40 10.99% wget
And the command line with which I ran it with.
jolan 5874 24.9 54.1 44980 35236 C5 RN+ Mon12AM 582:56.30 wget -t
0 -nH -k -m
I'm using version 1.6.
Thanks,
Jolan
