John Alvord wrote: > I am an interested bystander. Last year I diagnosed a 2+ year long storage > growth issue where there was no storage leak... firmly proven. > > The diagnosis was a case of extreme fragmentation. The process needed large > chunks of contiguous storage [2-3 megs] and when released the system storage > allocator [AIX] did not segregate the large chunks. Smaller requests populated > the newly freed area "sometimes" and thus a new storage chunk was needed. When > there was no storage available for a large chunk, the process space was > increased in a large chunk. Repeat until the upper limit of a 32 bit AIX > process size [roughly 2 gigs] is exceeded and the process failed, which took > 10-12 days. > > Not sure how you could prove that case - in effect would need to look at the > physical layout of allocated and free storage.
We see this all the time in OpenLDAP slapd with the glibc allocator. I've documented it pretty extensively. Google tcmalloc is much less susceptible to the problem. http://highlandsun.com/hyc/malloc/ > > Just one possibility... > > John Alvord, IBM > > > > ---------- Forwarded message ---------- > From: *Jakub Kubinski* <jkubinsk...@gmail.com <mailto:jkubinsk...@gmail.com>> > Date: Tue, Feb 14, 2012 at 3:23 AM > Subject: [Valgrind-users] RSS memory is increasing but memory leaks are not > reported > To: valgrind-users@lists.sourceforge.net > <mailto:valgrind-users@lists.sourceforge.net> > > > Hello, > > I have a "strange" memory leak in my XML parser which I cannot locate > using valgrind. > > In my testcase I'm creating and deleting a XML node and after the test > valgrind is not showing any memory leaks but rss memory is still > active VmRSS: 11184 kB ( VmRSS after the XML node creation was > 11048 kB so it increased by 136 after executing the XML node > destructor ) > > In the valigirng report I found "--30004-- memcheck: auxmaps: 179 > auxmap entries (11456k, 11M) in use" which fits to the remaining 11048 > kB in the RSS memory.What does this mean ? > > > > ==30004== Memcheck, a memory error detector. > ==30004== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al. > ==30004== Using LibVEX rev 1658, a library for dynamic binary translation. > ==30004== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP. > ==30004== Using valgrind-3.2.1, a dynamic binary instrumentation framework. > ==30004== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al. > ==30004== > ==30004== My PID = 30004, parent PID = 27692. Prog and args are: > ==30004== ./ala > ==30004== > --30004-- > --30004-- Command line > --30004-- ./ala > --30004-- Startup, with flags: > --30004-- --tool=memcheck > --30004-- --leak-check=full > --30004-- --leak-resolution=high > --30004-- --num-callers=40 > --30004-- --show-reachable=yes > --30004-- --error-limit=no > --30004-- -v > --30004-- --log-file=valgrind.log > --30004-- Contents of /proc/version: > --30004-- Linux version 2.6.39.4-1.NSN.kiuas > (r...@trling23.emea.nsn-net.net <mailto:r...@trling23.emea.nsn-net.net>) (gcc > version 4.1.2 20080704 (Red Hat > 4.1.2-46)) #1 SMP Wed Aug 10 12:16:36 EEST 2011 > --30004-- Arch and hwcaps: AMD64, amd64-sse2 > --30004-- Valgrind library directory: /usr/lib64/valgrind > --30004-- Reading syms from /home/kubinski/max/22/ala (0x400000) > --30004-- Reading syms from /usr/lib64/valgrind/amd64-linux/memcheck > (0x38000000) > --30004-- object doesn't have a dynamic symbol table > --30004-- Reading syms from /lib64/ld-2.5.so <http://ld-2.5.so> (0x3734000000) > --30004-- Reading suppressions file: /usr/lib64/valgrind/default.supp > --30004-- Reading syms from > /usr/lib64/valgrind/amd64-linux/vgpreload_core.so (0x4802000) > --30004-- Reading syms from > /usr/lib64/valgrind/amd64-linux/vgpreload_memcheck.so (0x4A03000) > --30004-- REDIR: 0x3734014400 (index) redirected to 0x4A06550 (index) > --30004-- REDIR: 0x37340145B0 (strcmp) redirected to 0x4A067D0 (strcmp) > --30004-- REDIR: 0x37340145E0 (strlen) redirected to 0x4A06700 (strlen) > --30004-- Reading syms from /usr/lib64/libstdc++.so.6.0.8 (0x3739400000) > --30004-- object doesn't have a symbol table > --30004-- Reading syms from /lib64/libm-2.5.so <http://libm-2.5.so> > (0x3735800000) > --30004-- Reading syms from /lib64/libgcc_s-4.1.2-20080825.so.1 (0x3738C00000) > --30004-- object doesn't have a symbol table > --30004-- Reading syms from /lib64/libc-2.5.so <http://libc-2.5.so> > (0x3735000000) > --30004-- REDIR: 0x3735079AB0 (rindex) redirected to 0x4A06400 (rindex) > --30004-- REDIR: 0x373507A970 (memset) redirected to 0x4A06920 (memset) > --30004-- REDIR: 0x37350796C0 (strlen) redirected to 0x4A066C0 (strlen) > --30004-- REDIR: 0x37394BD160 (operator new(unsigned long)) redirected > to 0x4A05F95 (operator new(unsigned long)) > --30004-- REDIR: 0x373507BDB0 (memcpy) redirected to 0x4A06FF0 (memcpy) > ==30004== Conditional jump or move depends on uninitialised value(s) > ==30004== at 0x373949B2BD: (within /usr/lib64/libstdc++.so.6.0.8) > ==30004== by 0x373949B585: std::string::find(char const*, unsigned > long, unsigned long) const (in /usr/lib64/libstdc++.so.6.0.8) > ==30004== by 0x402621: > libXML::CParser::changeCharsToEntitiesInCDATA() (in > /home/kubinski/max/22/ala) > ==30004== by 0x404259: libXML::CParser::parse(std::string&) (in > /home/kubinski/max/22/ala) > ==30004== by 0x4017E3: main (in /home/kubinski/max/22/ala) > --30004-- REDIR: 0x37394BBEF0 (operator delete(void*)) redirected to > 0x4A050A9 (operator delete(void*)) > --30004-- REDIR: 0x373507A1B0 (memchr) redirected to 0x4A06850 (memchr) > ==30004== > ==30004== Conditional jump or move depends on uninitialised value(s) > ==30004== at 0x404167: libXML::CParser::parse() (in > /home/kubinski/max/22/ala) > ==30004== by 0x404262: libXML::CParser::parse(std::string&) (in > /home/kubinski/max/22/ala) > ==30004== by 0x4017E3: main (in /home/kubinski/max/22/ala) > --30004-- REDIR: 0x37350726F0 (free) redirected to 0x4A05397 (free) > ==30004== > ==30004== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 4 from 1) > ==30004== > ==30004== 1 errors in context 1 of 2: > ==30004== Conditional jump or move depends on uninitialised value(s) > ==30004== at 0x404167: libXML::CParser::parse() (in > /home/kubinski/max/22/ala) > ==30004== by 0x404262: libXML::CParser::parse(std::string&) (in > /home/kubinski/max/22/ala) > ==30004== by 0x4017E3: main (in /home/kubinski/max/22/ala) > ==30004== > ==30004== 1 errors in context 2 of 2: > ==30004== Conditional jump or move depends on uninitialised value(s) > ==30004== at 0x373949B2BD: (within /usr/lib64/libstdc++.so.6.0.8) > --30004-- > --30004-- supp: 4 Fedora-Core-6-hack3-ld25 > ==30004== > ==30004== IN SUMMARY: 2 errors from 2 contexts (suppressed: 4 from 1) > ==30004== > ==30004== malloc/free: in use at exit: 0 bytes in 0 blocks. > ==30004== malloc/free: 846,934 allocs, 846,934 frees, 25,055,370 bytes > allocated. > ==30004== > ==30004== All heap blocks were freed -- no leaks are possible. > --30004-- memcheck: sanity checks: 11546 cheap, 462 expensive > --30004-- memcheck: auxmaps: 179 auxmap entries (11456k, 11M) in use > --30004-- memcheck: auxmaps: 27975161 searches, 29272608 comparisons > --30004-- memcheck: SMs: n_issued = 481 (7696k, 7M) > --30004-- memcheck: SMs: n_deissued = 18 (288k, 0M) > --30004-- memcheck: SMs: max_noaccess = 524287 (8388592k, 8191M) > --30004-- memcheck: SMs: max_undefined = 9 (144k, 0M) > --30004-- memcheck: SMs: max_defined = 249 (3984k, 3M) > --30004-- memcheck: SMs: max_non_DSM = 481 (7696k, 7M) > --30004-- memcheck: max sec V bit nodes: 0 (0k, 0M) > --30004-- memcheck: set_sec_vbits8 calls: 0 (new: 0, updates: 0) > --30004-- memcheck: max shadow mem size: 11840k, 11M > --30004-- translate: fast SP updates identified: 3,741 ( 87.8%) > --30004-- translate: generic_known SP updates identified: 307 ( 7.2%) > --30004-- translate: generic_unknown SP updates identified: 209 ( 4.9%) > --30004-- tt/tc: 117,211 tt lookups requiring 186,235 probes > --30004-- tt/tc: 117,211 fast-cache updates, 5 flushes > --30004-- transtab: new 3,358 (81,817 -> 1,555,815; ratio > 190:10) [0 scs] > --30004-- transtab: dumped 0 (0 -> ??) > --30004-- transtab: discarded 12 (257 -> ??) > --30004-- scheduler: 1,154,615,962 jumps (bb entries). > --30004-- scheduler: 11,546/1,819,427 major/minor sched events. > --30004-- sanity: 11547 cheap, 462 expensive checks. > --30004-- exectx: 30,011 lists, 170 contexts (avg 0 per list) > --30004-- exectx: 1,693,874 searches, 21,706,760 full compares > (12,814 per 1000) > --30004-- exectx: 0 cmp2, 15 cmp4, 0 cmpAll > > > > proc status without the valgrind after the XML creation > Name: ala > State: S (sleeping) > Tgid: 889 > Pid: 889 > PPid: 27692 > TracerPid: 0 > Uid: 61368538 61368538 61368538 61368538 > Gid: 300 300 300 300 > FDSize: 256 > Groups: 300 2051 > VmPeak: 21724 kB > VmSize: 21724 kB > VmLck: 0 kB > VmHWM: 11048 kB > VmRSS: 11048 kB > VmData: 9700 kB > VmStk: 136 kB > VmExe: 688 kB > VmLib: 2936 kB > VmPTE: 76 kB > VmSwap: 0 kB > Threads: 1 > SigQ: 0/1033564 > SigPnd: 0000000000000000 > ShdPnd: 0000000000000000 > SigBlk: 0000000000000000 > SigIgn: 0000000000000000 > SigCgt: 0000000000000000 > CapInh: 0000000000000000 > CapPrm: 0000000000000000 > CapEff: 0000000000000000 > CapBnd: ffffffffffffffff > Cpus_allowed: ffffffff > Cpus_allowed_list: 0-31 > voluntary_ctxt_switches: 13 > nonvoluntary_ctxt_switches: 2 > > proc status without the valgrind after deleting the XML node: > Name: ala > State: S (sleeping) > Tgid: 889 > Pid: 889 > PPid: 27692 > TracerPid: 0 > Uid: 61368538 61368538 61368538 61368538 > Gid: 300 300 300 300 > FDSize: 256 > Groups: 300 2051 > VmPeak: 21724 kB > VmSize: 21724 kB > VmLck: 0 kB > VmHWM: 11184 kB > VmRSS: 11184 kB > VmData: 9700 kB > VmStk: 136 kB > VmExe: 688 kB > VmLib: 2936 kB > VmPTE: 76 kB > VmSwap: 0 kB > Threads: 1 > SigQ: 0/1033564 > SigPnd: 0000000000000000 > ShdPnd: 0000000000000000 > SigBlk: 0000000000000000 > SigIgn: 0000000000000000 > SigCgt: 0000000000000000 > CapInh: 0000000000000000 > CapPrm: 0000000000000000 > CapEff: 0000000000000000 > CapBnd: ffffffffffffffff > Cpus_allowed: ffffffff > Cpus_allowed_list: 0-31 > voluntary_ctxt_switches: 14 > nonvoluntary_ctxt_switches: 4 > > Br, > Jakub > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > Valgrind-users mailing list > Valgrind-users@lists.sourceforge.net > <mailto:Valgrind-users@lists.sourceforge.net> > https://lists.sourceforge.net/lists/listinfo/valgrind-users > > > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > > > > _______________________________________________ > Valgrind-users mailing list > Valgrind-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/valgrind-users -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/ ------------------------------------------------------------------------------ Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d _______________________________________________ Valgrind-users mailing list Valgrind-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/valgrind-users
