Re: wget memory leak or expected operation

2001-06-06 Thread Hrvoje Niksic

jolan [EMAIL PROTECTED] writes:

 I'm currently using wget to mirror a rather extensive website. So
 far I've mirrored 10 gigs and counting. wget has been running for roughly
 24 hours. My problem is the huge amount of memory wget has alloted to
 it. Since I have it converting links, I was wondering if maybe all the
 webpages I have been mirroring are being stored in memory for later
 conversion to relative links.

No, this is not the case.

Yes, Wget can allocate a lot of memory when mirroring huge sites, but
I have not been able to detect a memory leaks, and I tried it with
many memory debugging tools.

So if Wget has allocated 44M so far, it will probably not allocate
much more.

I still haven't investigated where all the memory goes and if it is a
bug, but I plan to do so.



wget memory leak or expected operation

2001-06-05 Thread jolan

Hi, 
I'm currently using wget to mirror a rather extensive website. So
far I've mirrored 10 gigs and counting. wget has been running for roughly
24 hours. My problem is the huge amount of memory wget has alloted to
it. Since I have it converting links, I was wondering if maybe all the
webpages I have been mirroring are being stored in memory for later
conversion to relative links. This doesn't seem to be the case since some
links have been converted. Unforunately a lot of the links on each page
point to each other. It seems that wget may be downloading pages,
converting them, and then re-downloading them and converting them
again. My main concern is that my machine will run out of physical and
swap memory before the mirroring is complete. 

Here's the output of top:
 5874 jolan  24   44M   34M sleep select 581:40 10.99% wget 

And the command line with which I ran it with.
jolan 5874 24.9 54.1 44980 35236 C5  RN+   Mon12AM  582:56.30 wget -t
0 -nH -k -m

I'm using version 1.6.

Thanks,
Jolan