> [EMAIL PROTECTED] - Sat Mar 12 16:59:56 2005]:
>
> The following code fragment, from the FAQ, exhibits a memory leak for
> large numbers of input files:
>
> open(FIND, "find /terrabyte1/ro -type f -print |") or die;;
> while (<FIND>)
> {
> chop;
> open(PLAIN_FILE, $_) or die;
> $checksum = do
> {
> # something here leaks memory
> local $/; # slurp!
> unpack("%32C*", <PLAIN_FILE>) % 0xFFFF;
> };
> print "$checksum $_\n";
> close(PLAIN_FILE);
> }
> close(FIND);
>
> My data set is approx 161,000 files totaling slightly over 1 Tb. When
> this script attempts to checksum each file, I notice the process size
> and RSS grows monotonically until vm is exhausted.
>
> If I refactor the do {} block to not "slurp" the file in but compute
> the checksum via a loop with multiple calls to sysread(), the process
> maintains a stable size (approx 3Mb) throughout its entire life.
>
Valgrind seems to confirm the leaks.
==9320== warning: Valgrind's siglongjmp is incomplete
==9320== (it ignores cleanup handlers)
==9320== your program may misbehave as a result
==9320==
==9320== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 20 from 1)
--9320--
--9320-- supp: 20 dl_relocate_object/dl_main
==9320== malloc/free: in use at exit: 225479 bytes in 607 blocks.
==9320== malloc/free: 16541 allocs, 15934 frees, 131130187 bytes allocated.
==9320==
==9320== searching for pointers to 607 not-freed blocks.
==9320== checked 3329276 bytes.
==9320==
==9320== 5 bytes in 1 blocks are definitely lost in loss record 1 of 9
==9320== at 0x1B904A90: malloc (vg_replace_malloc.c:131)
==9320== by 0x32C6E94: Perl_savesharedpv (in
/usr/lib/perl5/5.8.5/i386-linux-
thread-multi/CORE/libperl.so)
==9320== by 0x3274353: (within
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/C
ORE/libperl.so)
==9320== by 0x327827B: perl_parse (in
/usr/lib/perl5/5.8.5/i386-linux-thread-
multi/CORE/libperl.so)
==9320==
==9320==
==9320== 1192 bytes in 1 blocks are definitely lost in loss record 5 of 9
==9320== at 0x1B9054FA: realloc (vg_replace_malloc.c:197)
==9320== by 0x32C892B: Perl_safesysrealloc (in
/usr/lib/perl5/5.8.5/i386-linu
x-thread-multi/CORE/libperl.so)
==9320== by 0x330B818: Perl_savestack_grow (in
/usr/lib/perl5/5.8.5/i386-linu
x-thread-multi/CORE/libperl.so)
==9320== by 0x330B9B8: Perl_save_sptr (in
/usr/lib/perl5/5.8.5/i386-linux-thr
ead-multi/CORE/libperl.so)
==9320==
==9320==
==9320== 8968 bytes in 9 blocks are possibly lost in loss record 7 of 9
==9320== at 0x1B904A90: malloc (vg_replace_malloc.c:131)
==9320== by 0x32C7C1A: Perl_safesysmalloc (in
/usr/lib/perl5/5.8.5/i386-linux
-thread-multi/CORE/libperl.so)
==9320== by 0x32E4FCA: (within
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/C
ORE/libperl.so)
==9320== by 0x32E70F8: Perl_sv_upgrade (in
/usr/lib/perl5/5.8.5/i386-linux-th
read-multi/CORE/libperl.so)
==9320==
==9320==
==9320== 132726 bytes in 11 blocks are definitely lost in loss record 9 of 9
==9320== at 0x1B904A90: malloc (vg_replace_malloc.c:131)
==9320== by 0x32C7C1A: Perl_safesysmalloc (in
/usr/lib/perl5/5.8.5/i386-linux
-thread-multi/CORE/libperl.so)
==9320== by 0x330B4BF: Perl_new_stackinfo (in
/usr/lib/perl5/5.8.5/i386-linux
-thread-multi/CORE/libperl.so)
==9320== by 0x3271DAA: Perl_init_stacks (in
/usr/lib/perl5/5.8.5/i386-linux-t
hread-multi/CORE/libperl.so)
==9320==
==9320== LEAK SUMMARY:
==9320== definitely lost: 133923 bytes in 13 blocks.
==9320== possibly lost: 8968 bytes in 9 blocks.
==9320== still reachable: 82388 bytes in 584 blocks.
==9320== suppressed: 200 bytes in 1 blocks.
==9320== Reachable blocks (those to which a pointer was found) are not
shown.
==9320== To see them, rerun with: --show-reachable=yes
--9320-- TT/TC: 0 tc sectors discarded.
--9320-- 56488 tt_fast misses.
--9320-- translate: new 9010 (153873 -> 1997185; ratio 129:10)
--9320-- discard 1 (23 -> 320; ratio 139:10).
--9320-- chainings: 6458 chainings, 2 unchainings.
--9320-- dispatch: 373000000 jumps (bb entries); of them 16024882 (4%)
unchaine
d.
--9320-- 121827/1722921 major/minor sched events.
--9320-- reg-alloc: 1835 t-req-spill, 367423+12618 orig+spill uis,
--9320-- 46852 total-reg-rank
--9320-- sanity: 117523 cheap, 4701 expensive checks.
--9320-- ccalls: 36820 C calls, 56% saves+restores avoided (123274 bytes)
--9320-- 49527 args, avg 0.89 setup instrs each (10700 bytes)
--9320-- 0% clear the stack (110208 bytes)
--9320-- 15121 retvals, 34% of reg-reg movs avoided (10196 bytes)
