Thomas,
> Did you use a buffer larger than 32k on those vmcp commands?
Yes, I always use 1M (vmcpCmd="/sbin/vmcp --buffer=1M") in the event there
is a lot of output from CP.
> Vmcp can fail due to memory fragmentation even on a server with lots of
free memory.
Hmmm, interesting... could this be considered a bug?
As a workaround, is there a command to flush the buffer cache?
Thanks.
-Mike M.
On Thu, Jul 9, 2015 at 10:40 AM, Pavelka, Tomas <[email protected]>
wrote:
> > In the past this server has gone to near zero memory, and vmcp commands
> fail.
>
> Do you have any specifics? Did you use a buffer larger than 32k on those
> vmcp commands? Vmcp can fail due to memory fragmentation even on a server
> with lots of free memory.
>
> Tomas Pavelka
> CA Technologies
> Sr Software Engineer
>
> CA CZ, s.r.o
> V Parku 12,
> 148 00 Praha
> Czech Republic
>
> Office: +25996 | [email protected]
>
>
>
> Id. Císlo 25694073, z obchodního rejstříku, vedeného Městským soudem v
> Praze, oddíl C, vložka 61808 / Id. No. 25694073, registered in the
> Commercial Register maintained by the Municipal Court in Praque, Section C,
> File 61808
>
>
>
> -----Original Message-----
> From: Linux on 390 Port [mailto:[email protected]] On Behalf Of
> Michael MacIsaac
> Sent: Thursday, July 09, 2015 4:15 PM
> To: [email protected]
> Subject: Re: How to find a memory leak?
>
> Thanks Richard for the joke :))
>
> Thanks Thomas for the input. I changed the ps command flag to '--sort
> -rss', and restarted memusage - will continue to monitor.
>
> Thanks Dave for the pointer, but I don't have any of my own C/C++ programs
> running, just many bash scripts (if they do no 'malloc's, can they still
> cause memory leaks?).
>
> In the past this server has gone to near zero memory, and vmcp commands
> fail. I'm guessing the OOM killer was invoked, but by then it's already
> too late ...
>
> -Mike
>
> On Thu, Jul 9, 2015 at 9:54 AM, Dave Jones <[email protected]>
> wrote:
>
> > Hi, Mike.
> >
> > if the package AddressSanitizer (ASan) is available, you might want to
> > ive it a go. It is a fast memory error detector. that can find
> > use-after-free and {heap,stack,global}-buffer overflow bugs in C/C++
> > programs. it's here:
> >
> > https://code.google.com/p/address-sanitizer/
> >
> > Good luck....I still think C/C++ will be the death of us all. :-)
> >
> > DJ
> >
> > On 07/09/2015 07:50 AM, Pavelka, Tomas wrote:
> > > Look at the " -/+ buffers/cache" line in the free output:
> > >
> > > Before:
> > > -/+ buffers/cache: 41 450
> > > After:
> > > -/+ buffers/cache: 48 443
> > >
> > > (First number used, second free)
> > >
> > > Linux has various buffers and caches that are allocated if there is
> > > free
> > memory. For example for disk reads. These are dropped if the memory is
> > needed by processes. The " -/+ buffers/cache" line shows what memory
> > is actually used by processes and not the buffers. In your case the
> > used memory rose only by 7 MB.
> > >
> > > BTW I would not look at the virtual memory size of proceses, this
> > > may be
> > allocated way over the virtual memory size of your machine. The more
> > interesting metric is RSS which is how much memory is actually used.
> > >
> > > HTH,
> > > Tomas
> > >
> > > Tomas Pavelka
> > > CA Technologies
> > > Sr Software Engineer
> > >
> > > CA CZ, s.r.o
> > > V Parku 12,
> > > 148 00 Praha
> > > Czech Republic
> > >
> > > Office: +25996 | [email protected]
> > >
> > >
> > >
> > > Id. Císlo 25694073, z obchodního rejstříku, vedeného Městským soudem
> > > v
> > Praze, oddíl C, vložka 61808 / Id. No. 25694073, registered in the
> > Commercial Register maintained by the Municipal Court in Praque,
> > Section C, File 61808
> > >
> > >
> > > -----Original Message-----
> > > From: Linux on 390 Port [mailto:[email protected]] On Behalf
> > > Of
> > Michael MacIsaac
> > > Sent: Thursday, July 09, 2015 2:19 PM
> > > To: [email protected]
> > > Subject: How to find a memory leak?
> > >
> > > Hello list,
> > >
> > > I have a SLES 11 SP3 system that is leaking memory, but I don't know
> > > how
> > or where.
> > >
> > > I find a script on the Internet that runs forever, adapt it
> > > somewhat,
> > and start logging some info to a temp file. Here's the script:
> > >
> > > # cat memusage
> > > #!/bin/bash
> > > #
> > > # track memory usage
> > > #
> > > outFile="/tmp/memusage"
> > > while true
> > > do
> > > echo "---------------------------------------------------" >>
> $outFile
> > > date >> $outFile
> > > ps aux --sort -vsz | head -22 >> $outFile
> > > echo >> $outFile
> > > free -m >> $outFile
> > > sleep 300
> > > done
> > >
> > > After a fresh reboot of a 512 MB virtual machine, I start the script
> > > and
> > the first entry in the temp file shows about 20 MB (512 - 492) used by
> > Linux and 97 MB used by processes:
> > >
> > > Wed Jul 8 12:37:45 EDT 2015
> > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
> COMMAND
> > > root 2181 0.0 0.2 115404 1024 ? Ssl 12:36 0:00
> > > /usr/sbin/nscd
> > > root 1851 0.0 0.1 11512 692 ? S<sl 12:36 0:00
> > > /sbin/auditd -s disable
> > > root 2556 0.3 0.7 11456 4004 ? Ss 12:37 0:00 sshd:
> > > root@pts/0
> > > root 2306 0.0 0.7 10720 3700 ? Ss 12:36 0:00
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2307 0.0 0.4 10720 2204 ? S 12:36 0:00
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2308 0.0 0.4 10720 2204 ? S 12:36 0:00
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2309 0.0 0.4 10720 2204 ? S 12:36 0:00
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2310 0.0 0.4 10720 2204 ? S 12:36 0:00
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2311 0.0 0.4 10720 2204 ? S 12:36 0:00
> > > /usr/sbin/httpd2-prefork
> > > root 1853 0.0 0.1 10428 824 ? S<sl 12:36 0:00
> > > /sbin/audispd
> > > root 997 0.0 0.6 9036 3224 ? Ssl 12:36 0:00
> > > /usr/sbin/console-kit-da
> > > root 2265 0.0 0.5 8136 2532 ? Ss 12:36 0:00
> > > /usr/lib/postfix/master
> > > postfix 2277 0.0 0.4 8004 2372 ? S 12:36 0:00 qmgr
> -l
> > -t
> > > fifo -u
> > > postfix 2276 0.0 0.4 7948 2352 ? S 12:36 0:00 pickup
> > -l
> > > -t fifo -u
> > > root 2172 0.0 0.3 7916 1532 ? Ss 12:36 0:00
> > > /usr/sbin/sshd -o PidFi
> > > 101 994 0.0 0.5 7852 2804 ? Ss 12:36 0:00
> > > /usr/sbin/hald --daemon
> > > root 1869 0.0 0.8 6464 4504 ? Ss 12:36 0:00
> > > /sbin/haveged -w 1024 -
> > > root 2559 1.0 0.6 6056 3076 pts/0 Ss 12:37 0:00 -bash
> > > root 998 0.0 0.2 3980 1332 ? S 12:36 0:00
> > hald-runner
> > > root 2591 0.0 0.3 3652 1604 pts/0 S+ 12:37 0:00
> > /bin/bash
> > > /usr/local/sb
> > > root 2343 0.0 0.1 3508 944 ? Ss 12:36 0:00
> > > /usr/sbin/xinetd -pidfi
> > >
> > > total used free shared buffers
> cached
> > > Mem: *492 97* 394 0 5
> > 50
> > > -/+ buffers/cache: 41 450
> > > Swap: 898 0 898
> > >
> > > This morning the last entry shows 156 MB used by processes: ~59 MB
> > > of
> > memory lost in less than a day. But the 'VSZ' of the top 22 processes
> > seems to be about the same:
> > >
> > > Thu Jul 9 07:57:47 EDT 2015
> > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
> COMMAND
> > > root 2181 0.0 0.2 115404 1088 ? Ssl Jul08 0:00
> > > /usr/sbin/nscd
> > > root 1851 0.0 0.1 11512 692 ? S<sl Jul08 0:00
> > > /sbin/auditd -s disable
> > > root 44879 0.0 0.7 11456 4008 ? Ss 07:36 0:00 sshd:
> > > root@pts/1
> > > root 44913 0.0 0.7 11456 4008 ? Ss 07:37 0:00 sshd:
> > > root@pts/2
> > > root 2306 0.0 0.7 10720 3700 ? Ss Jul08 0:01
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2307 0.0 0.4 10720 2204 ? S Jul08 0:00
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2308 0.0 0.4 10720 2204 ? S Jul08 0:00
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2309 0.0 0.4 10720 2204 ? S Jul08 0:00
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2310 0.0 0.4 10720 2204 ? S Jul08 0:00
> > > /usr/sbin/httpd2-prefork
> > > wwwrun 2311 0.0 0.4 10720 2204 ? S Jul08 0:00
> > > /usr/sbin/httpd2-prefork
> > > root 1853 0.0 0.1 10428 824 ? S<sl Jul08 0:00
> > > /sbin/audispd
> > > root 997 0.0 0.6 9036 3224 ? Ssl Jul08 0:00
> > > /usr/sbin/console-kit-da
> > > root 2265 0.0 0.5 8136 2532 ? Ss Jul08 0:00
> > > /usr/lib/postfix/master
> > > postfix 2277 0.0 0.4 8004 2372 ? S Jul08 0:00 qmgr
> -l
> > -t
> > > fifo -u
> > > postfix 44778 0.0 0.4 7948 2356 ? S 06:56 0:00 pickup
> > -l
> > > -t fifo -u
> > > root 2172 0.0 0.3 7916 1664 ? Ss Jul08 0:00
> > > /usr/sbin/sshd -o PidFi
> > > 101 994 0.0 0.5 7852 2852 ? Ss Jul08 0:00
> > > /usr/sbin/hald --daemon
> > > root 1869 0.0 0.8 6464 4504 ? Ss Jul08 0:00
> > > /sbin/haveged -w 1024 -
> > > root 44882 0.0 0.6 6056 3052 pts/1 Ss 07:36 0:00 -bash
> > > root 44916 0.0 0.5 6056 2876 pts/2 Ss+ 07:37 0:00 -bash
> > > mike 44992 0.0 0.5 6056 3004 pts/1 S+ 07:56 0:00 -bash
> > >
> > > total used free shared buffers
> cached
> > > Mem: *492 156 * 335 0 24
> > 83
> > > -/+ buffers/cache: 48 443
> > > Swap: 898 0 898
> > >
> > > Is there a better way to detect memory leaks? Are there any known
> > 'culprits'?
> > >
> > > Any help will be appreciated. Thanks.
> > >
> > > -Mike MacIsaac
> > >
> > > --------------------------------------------------------------------
> > > -- For LINUX-390 subscribe / signoff / archive access instructions,
> > > send
> > email to [email protected] with the message: INFO LINUX-390 or
> > visit
> > > http://www.marist.edu/htbin/wlvindex?LINUX-390
> > > --------------------------------------------------------------------
> > > -- For more information on Linux on System z, visit
> > http://wiki.linuxvm.org/
> > >
> >
> > --
> > Dave Jones
> > Houston, TX
> > 281.578.7544
> >
> > ----------------------------------------------------------------------
> > For LINUX-390 subscribe / signoff / archive access instructions, send
> > email to [email protected] with the message: INFO LINUX-390 or
> > visit
> > http://www.marist.edu/htbin/wlvindex?LINUX-390
> > ----------------------------------------------------------------------
> > For more information on Linux on System z, visit
> > http://wiki.linuxvm.org/
> >
>
> ----------------------------------------------------------------------
> For LINUX-390 subscribe / signoff / archive access instructions, send
> email to [email protected] with the message: INFO LINUX-390 or visit
> http://www.marist.edu/htbin/wlvindex?LINUX-390
> ----------------------------------------------------------------------
> For more information on Linux on System z, visit http://wiki.linuxvm.org/
>
----------------------------------------------------------------------
For LINUX-390 subscribe / signoff / archive access instructions,
send email to [email protected] with the message: INFO LINUX-390 or visit
http://www.marist.edu/htbin/wlvindex?LINUX-390
----------------------------------------------------------------------
For more information on Linux on System z, visit
http://wiki.linuxvm.org/
