Hi,
Is there a way to "force" the application to return the freed memory
to the OS ( when using glibc )
Br,
Jakub
On 16 February 2012 04:33, Howard Chu <h...@symas.com> wrote:
> Jakub Kubinski wrote:
>>
>> Hi,
>>
>> I tried tcmalloc but the results are the same. RSS memory usage is
>> even slightly higher when I used tcmalloc
>
>
> This test is inconclusive since tcmalloc never returns freed memory to the
> OS. Also, while tcmalloc has greater overhead in a single-threaded program,
> it has much less overhead than glibc in multi-threaded code.
>
>>
>> RSS after main initialization
>> glibcMalloc : 2080KB
>> tcMalloc: 2756KB
>>
>> RSS after XML node initialization
>> glibcMalloc : 7004KB
>> tcMalloc: 7560KB
>>
>> RSS after removing the XML node
>> glibcMalloc : 7068KB
>> tcMalloc: 7764KB
>>
>>
>> I also created a small test program in which I manage to reproduce the
>> same results:
>>
>> #include<stdlib.h>
>> #include<stdio.h>
>> #include<string>
>> using namespace std;
>>
>> int main()
>> {
>> system("top -b -n 1> result_top.txt");
>> int **t = new int*[100000];
>> for(long i = 0; i< 100000; i++)
>> {
>> t[i] = new int();
>>
>> }
>> system("top -b -n 1>> result_top.txt");
>>
>> for(long i = 100000 - 1; i>= 0; i--)
>> {
>> delete t[i];
>> }
>> delete [] t;
>> system("top -b -n 1>> result_top.txt");
>> return 0;
>> }
>>
>> Top output after main initialization:
>> glibc: 2716
>> tcMalloc: 4948
>>
>> Top output after “t” array initialization:
>> glibc: 2572
>> tcMalloc: 2668
>>
>> Top output after deleting the “t” array:
>> glibc: 2352
>> tcMalloc: 2708
>>
>> Br,
>> Jakub
>>
>>
>>
>> On 14 February 2012 20:20, Howard Chu<h...@symas.com> wrote:
>>>
>>> John Alvord wrote:
>>>>
>>>>
>>>> I am an interested bystander. Last year I diagnosed a 2+ year long
>>>> storage
>>>> growth issue where there was no storage leak... firmly proven.
>>>>
>>>> The diagnosis was a case of extreme fragmentation. The process needed
>>>> large
>>>> chunks of contiguous storage [2-3 megs] and when released the system
>>>> storage
>>>> allocator [AIX] did not segregate the large chunks. Smaller requests
>>>> populated
>>>> the newly freed area "sometimes" and thus a new storage chunk was
>>>> needed.
>>>> When
>>>> there was no storage available for a large chunk, the process space was
>>>> increased in a large chunk. Repeat until the upper limit of a 32 bit AIX
>>>> process size [roughly 2 gigs] is exceeded and the process failed, which
>>>> took
>>>> 10-12 days.
>>>>
>>>> Not sure how you could prove that case - in effect would need to look at
>>>> the
>>>> physical layout of allocated and free storage.
>>>
>>>
>>>
>>> We see this all the time in OpenLDAP slapd with the glibc allocator. I've
>>> documented it pretty extensively. Google tcmalloc is much less
>>> susceptible
>>> to the problem.
>>>
>>> http://highlandsun.com/hyc/malloc/
>>>
>>>>
>>>> Just one possibility...
>>>>
>>>> John Alvord, IBM
>>>>
>>>>
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: *Jakub Kubinski*<jkubinsk...@gmail.com
>>>> <mailto:jkubinsk...@gmail.com>>
>>>> Date: Tue, Feb 14, 2012 at 3:23 AM
>>>> Subject: [Valgrind-users] RSS memory is increasing but memory leaks are
>>>> not
>>>> reported
>>>> To: valgrind-users@lists.sourceforge.net
>>>> <mailto:valgrind-users@lists.sourceforge.net>
>>>>
>>>>
>>>> Hello,
>>>>
>>>> I have a "strange" memory leak in my XML parser which I cannot locate
>>>> using valgrind.
>>>>
>>>> In my testcase I'm creating and deleting a XML node and after the test
>>>> valgrind is not showing any memory leaks but rss memory is still
>>>> active VmRSS: 11184 kB ( VmRSS after the XML node creation was
>>>> 11048 kB so it increased by 136 after executing the XML node
>>>> destructor )
>>>>
>>>> In the valigirng report I found "--30004-- memcheck: auxmaps: 179
>>>> auxmap entries (11456k, 11M) in use" which fits to the remaining 11048
>>>> kB in the RSS memory.What does this mean ?
>>>>
>>>>
>>>>
>>>> ==30004== Memcheck, a memory error detector.
>>>> ==30004== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et
>>>> al.
>>>> ==30004== Using LibVEX rev 1658, a library for dynamic binary
>>>> translation.
>>>> ==30004== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.
>>>> ==30004== Using valgrind-3.2.1, a dynamic binary instrumentation
>>>> framework.
>>>> ==30004== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et
>>>> al.
>>>> ==30004==
>>>> ==30004== My PID = 30004, parent PID = 27692. Prog and args are:
>>>> ==30004== ./ala
>>>> ==30004==
>>>> --30004--
>>>> --30004-- Command line
>>>> --30004-- ./ala
>>>> --30004-- Startup, with flags:
>>>> --30004-- --tool=memcheck
>>>> --30004-- --leak-check=full
>>>> --30004-- --leak-resolution=high
>>>> --30004-- --num-callers=40
>>>> --30004-- --show-reachable=yes
>>>> --30004-- --error-limit=no
>>>> --30004-- -v
>>>> --30004-- --log-file=valgrind.log
>>>> --30004-- Contents of /proc/version:
>>>> --30004-- Linux version 2.6.39.4-1.NSN.kiuas
>>>> (r...@trling23.emea.nsn-net.net<mailto:r...@trling23.emea.nsn-net.net>)
>>>> (gcc
>>>>
>>>> version 4.1.2 20080704 (Red Hat
>>>> 4.1.2-46)) #1 SMP Wed Aug 10 12:16:36 EEST 2011
>>>> --30004-- Arch and hwcaps: AMD64, amd64-sse2
>>>> --30004-- Valgrind library directory: /usr/lib64/valgrind
>>>> --30004-- Reading syms from /home/kubinski/max/22/ala (0x400000)
>>>> --30004-- Reading syms from /usr/lib64/valgrind/amd64-linux/memcheck
>>>> (0x38000000)
>>>> --30004-- object doesn't have a dynamic symbol table
>>>> --30004-- Reading syms from /lib64/ld-2.5.so<http://ld-2.5.so>
>>>> (0x3734000000)
>>>>
>>>> --30004-- Reading suppressions file: /usr/lib64/valgrind/default.supp
>>>> --30004-- Reading syms from
>>>> /usr/lib64/valgrind/amd64-linux/vgpreload_core.so (0x4802000)
>>>> --30004-- Reading syms from
>>>> /usr/lib64/valgrind/amd64-linux/vgpreload_memcheck.so (0x4A03000)
>>>> --30004-- REDIR: 0x3734014400 (index) redirected to 0x4A06550 (index)
>>>> --30004-- REDIR: 0x37340145B0 (strcmp) redirected to 0x4A067D0 (strcmp)
>>>> --30004-- REDIR: 0x37340145E0 (strlen) redirected to 0x4A06700 (strlen)
>>>> --30004-- Reading syms from /usr/lib64/libstdc++.so.6.0.8 (0x3739400000)
>>>> --30004-- object doesn't have a symbol table
>>>> --30004-- Reading syms from /lib64/libm-2.5.so<http://libm-2.5.so>
>>>> (0x3735800000)
>>>>
>>>> --30004-- Reading syms from /lib64/libgcc_s-4.1.2-20080825.so.1
>>>> (0x3738C00000)
>>>> --30004-- object doesn't have a symbol table
>>>> --30004-- Reading syms from /lib64/libc-2.5.so<http://libc-2.5.so>
>>>> (0x3735000000)
>>>>
>>>> --30004-- REDIR: 0x3735079AB0 (rindex) redirected to 0x4A06400 (rindex)
>>>> --30004-- REDIR: 0x373507A970 (memset) redirected to 0x4A06920 (memset)
>>>> --30004-- REDIR: 0x37350796C0 (strlen) redirected to 0x4A066C0 (strlen)
>>>> --30004-- REDIR: 0x37394BD160 (operator new(unsigned long)) redirected
>>>> to 0x4A05F95 (operator new(unsigned long))
>>>> --30004-- REDIR: 0x373507BDB0 (memcpy) redirected to 0x4A06FF0 (memcpy)
>>>> ==30004== Conditional jump or move depends on uninitialised value(s)
>>>> ==30004== at 0x373949B2BD: (within /usr/lib64/libstdc++.so.6.0.8)
>>>> ==30004== by 0x373949B585: std::string::find(char const*, unsigned
>>>> long, unsigned long) const (in /usr/lib64/libstdc++.so.6.0.8)
>>>> ==30004== by 0x402621:
>>>> libXML::CParser::changeCharsToEntitiesInCDATA() (in
>>>> /home/kubinski/max/22/ala)
>>>> ==30004== by 0x404259: libXML::CParser::parse(std::string&) (in
>>>> /home/kubinski/max/22/ala)
>>>> ==30004== by 0x4017E3: main (in /home/kubinski/max/22/ala)
>>>> --30004-- REDIR: 0x37394BBEF0 (operator delete(void*)) redirected to
>>>> 0x4A050A9 (operator delete(void*))
>>>> --30004-- REDIR: 0x373507A1B0 (memchr) redirected to 0x4A06850 (memchr)
>>>> ==30004==
>>>> ==30004== Conditional jump or move depends on uninitialised value(s)
>>>> ==30004== at 0x404167: libXML::CParser::parse() (in
>>>> /home/kubinski/max/22/ala)
>>>> ==30004== by 0x404262: libXML::CParser::parse(std::string&) (in
>>>> /home/kubinski/max/22/ala)
>>>> ==30004== by 0x4017E3: main (in /home/kubinski/max/22/ala)
>>>> --30004-- REDIR: 0x37350726F0 (free) redirected to 0x4A05397 (free)
>>>> ==30004==
>>>> ==30004== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 4 from 1)
>>>> ==30004==
>>>> ==30004== 1 errors in context 1 of 2:
>>>> ==30004== Conditional jump or move depends on uninitialised value(s)
>>>> ==30004== at 0x404167: libXML::CParser::parse() (in
>>>> /home/kubinski/max/22/ala)
>>>> ==30004== by 0x404262: libXML::CParser::parse(std::string&) (in
>>>> /home/kubinski/max/22/ala)
>>>> ==30004== by 0x4017E3: main (in /home/kubinski/max/22/ala)
>>>> ==30004==
>>>> ==30004== 1 errors in context 2 of 2:
>>>> ==30004== Conditional jump or move depends on uninitialised value(s)
>>>> ==30004== at 0x373949B2BD: (within /usr/lib64/libstdc++.so.6.0.8)
>>>> --30004--
>>>> --30004-- supp: 4 Fedora-Core-6-hack3-ld25
>>>> ==30004==
>>>> ==30004== IN SUMMARY: 2 errors from 2 contexts (suppressed: 4 from 1)
>>>> ==30004==
>>>> ==30004== malloc/free: in use at exit: 0 bytes in 0 blocks.
>>>> ==30004== malloc/free: 846,934 allocs, 846,934 frees, 25,055,370 bytes
>>>> allocated.
>>>> ==30004==
>>>> ==30004== All heap blocks were freed -- no leaks are possible.
>>>> --30004-- memcheck: sanity checks: 11546 cheap, 462 expensive
>>>> --30004-- memcheck: auxmaps: 179 auxmap entries (11456k, 11M) in use
>>>> --30004-- memcheck: auxmaps: 27975161 searches, 29272608 comparisons
>>>> --30004-- memcheck: SMs: n_issued = 481 (7696k, 7M)
>>>> --30004-- memcheck: SMs: n_deissued = 18 (288k, 0M)
>>>> --30004-- memcheck: SMs: max_noaccess = 524287 (8388592k, 8191M)
>>>> --30004-- memcheck: SMs: max_undefined = 9 (144k, 0M)
>>>> --30004-- memcheck: SMs: max_defined = 249 (3984k, 3M)
>>>> --30004-- memcheck: SMs: max_non_DSM = 481 (7696k, 7M)
>>>> --30004-- memcheck: max sec V bit nodes: 0 (0k, 0M)
>>>> --30004-- memcheck: set_sec_vbits8 calls: 0 (new: 0, updates: 0)
>>>> --30004-- memcheck: max shadow mem size: 11840k, 11M
>>>> --30004-- translate: fast SP updates identified: 3,741 (
>>>> 87.8%)
>>>> --30004-- translate: generic_known SP updates identified: 307 ( 7.2%)
>>>> --30004-- translate: generic_unknown SP updates identified: 209 ( 4.9%)
>>>> --30004-- tt/tc: 117,211 tt lookups requiring 186,235 probes
>>>> --30004-- tt/tc: 117,211 fast-cache updates, 5 flushes
>>>> --30004-- transtab: new 3,358 (81,817 -> 1,555,815; ratio
>>>> 190:10) [0 scs]
>>>> --30004-- transtab: dumped 0 (0 -> ??)
>>>> --30004-- transtab: discarded 12 (257 -> ??)
>>>> --30004-- scheduler: 1,154,615,962 jumps (bb entries).
>>>> --30004-- scheduler: 11,546/1,819,427 major/minor sched events.
>>>> --30004-- sanity: 11547 cheap, 462 expensive checks.
>>>> --30004-- exectx: 30,011 lists, 170 contexts (avg 0 per list)
>>>> --30004-- exectx: 1,693,874 searches, 21,706,760 full compares
>>>> (12,814 per 1000)
>>>> --30004-- exectx: 0 cmp2, 15 cmp4, 0 cmpAll
>>>>
>>>>
>>>>
>>>> proc status without the valgrind after the XML creation
>>>> Name: ala
>>>> State: S (sleeping)
>>>> Tgid: 889
>>>> Pid: 889
>>>> PPid: 27692
>>>> TracerPid: 0
>>>> Uid: 61368538 61368538 61368538 61368538
>>>> Gid: 300 300 300 300
>>>> FDSize: 256
>>>> Groups: 300 2051
>>>> VmPeak: 21724 kB
>>>> VmSize: 21724 kB
>>>> VmLck: 0 kB
>>>> VmHWM: 11048 kB
>>>> VmRSS: 11048 kB
>>>> VmData: 9700 kB
>>>> VmStk: 136 kB
>>>> VmExe: 688 kB
>>>> VmLib: 2936 kB
>>>> VmPTE: 76 kB
>>>> VmSwap: 0 kB
>>>> Threads: 1
>>>> SigQ: 0/1033564
>>>> SigPnd: 0000000000000000
>>>> ShdPnd: 0000000000000000
>>>> SigBlk: 0000000000000000
>>>> SigIgn: 0000000000000000
>>>> SigCgt: 0000000000000000
>>>> CapInh: 0000000000000000
>>>> CapPrm: 0000000000000000
>>>> CapEff: 0000000000000000
>>>> CapBnd: ffffffffffffffff
>>>> Cpus_allowed: ffffffff
>>>> Cpus_allowed_list: 0-31
>>>> voluntary_ctxt_switches: 13
>>>> nonvoluntary_ctxt_switches: 2
>>>>
>>>> proc status without the valgrind after deleting the XML node:
>>>> Name: ala
>>>> State: S (sleeping)
>>>> Tgid: 889
>>>> Pid: 889
>>>> PPid: 27692
>>>> TracerPid: 0
>>>> Uid: 61368538 61368538 61368538 61368538
>>>> Gid: 300 300 300 300
>>>> FDSize: 256
>>>> Groups: 300 2051
>>>> VmPeak: 21724 kB
>>>> VmSize: 21724 kB
>>>> VmLck: 0 kB
>>>> VmHWM: 11184 kB
>>>> VmRSS: 11184 kB
>>>> VmData: 9700 kB
>>>> VmStk: 136 kB
>>>> VmExe: 688 kB
>>>> VmLib: 2936 kB
>>>> VmPTE: 76 kB
>>>> VmSwap: 0 kB
>>>> Threads: 1
>>>> SigQ: 0/1033564
>>>> SigPnd: 0000000000000000
>>>> ShdPnd: 0000000000000000
>>>> SigBlk: 0000000000000000
>>>> SigIgn: 0000000000000000
>>>> SigCgt: 0000000000000000
>>>> CapInh: 0000000000000000
>>>> CapPrm: 0000000000000000
>>>> CapEff: 0000000000000000
>>>> CapBnd: ffffffffffffffff
>>>> Cpus_allowed: ffffffff
>>>> Cpus_allowed_list: 0-31
>>>> voluntary_ctxt_switches: 14
>>>> nonvoluntary_ctxt_switches: 4
>>>>
>>>> Br,
>>>> Jakub
>
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
------------------------------------------------------------------------------
Virtualization & Cloud Management Using Capacity Planning
Cloud computing makes use of virtualization - but cloud computing
also focuses on allowing computing to be delivered as a service.
http://www.accelacomm.com/jaw/sfnl/114/51521223/
_______________________________________________
Valgrind-users mailing list
Valgrind-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/valgrind-users
