Re: How do you accurately determine a process' RAM usage?
Mauricio Lin wrote: Hi Brady, On 7/20/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: The following shell gets the shared values for the first httpd process: FIRST_HTTPD=`ps -C httpd -o pid= | head -1 | tr -d ' '` HTTPD_STATM_SHARED=$(expr 4 '*' `cut -f3 -d' ' /proc/$FIRST_HTTPD/statm`) HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc) This shows that "smaps" reports 3060 KB more shared mem than "statm". However adding up all the anon sections in smaps only gives 2456 KB? You are adding up all Shared_Clean and Shared_Dirty as Shared, right? yes. Look at the command I posted. When doing this I also noticed that there are duplicate entries in smaps. Any ideas why? Each pair of address per line indicates the start and end address of a memory area (VMA) such as: b7f7d000-b7f7e000 This means that an specific memory area start on virtual address b7f7d000 and end on b7f7e000 . An mapped file like /lib/ld-2.3.3.so is organized in different memory areas. The memory area can be a text section, data section or bss. So it is normal you find the same filename mapped in more than one memory area. yes. Look at the command I posted. There are multiple entries with the _same addresses_ grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c -- Pádraig Brady - http://www.pixelbeat.org -- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Hi Brady, On 7/20/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > Mauricio Lin wrote: > > Hi, > > > > On 7/12/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > > > >>Andrew Morton wrote: > >> > >>>OK, please let us know how it goes. > >> > >>It went very well. I could find no problems at all. > >>I've updated my script to use the new method, so please merge smaps :) > >>http://www.pixelbeat.org/scripts/ps_mem.py > >> > >>Usually the shared mem reported by /proc/$$/statm > >>is the same as summing all the shared values in in /proc/$$/smaps > >>but there can be large discrepancies. > > > > > > Have you checked how the statm shared is calculated? I guess it does > > something like: > > shared = mm->rss - mm->anon_rss > > yes > > > But in smaps output you can have anonymous area like: > > > > b6e0e000-b6e13000 rw-p > > Size:20 KB > > Rss: 4 KB > > Shared_Clean: 0 KB > > Shared_Dirty: 4 KB > > Private_Clean:0 KB > > Private_Dirty:0 KB > > > > Look that it presents 4 KB of shared value in area considered anonymous. > > > > ANDREW: anon_rss is the rss for anonymous area, right? > > I see your point and I'm not sure. > The following shell gets the shared values for the > first httpd process: > > FIRST_HTTPD=`ps -C httpd -o pid= | head -1 | tr -d ' '` > HTTPD_STATM_SHARED=$(expr 4 '*' `cut -f3 -d' ' /proc/$FIRST_HTTPD/statm`) > HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' > | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc) > > > This shows that "smaps" reports 3060 KB more shared mem than "statm". > However adding up all the anon sections in smaps only gives 2456 KB? You are adding up all Shared_Clean and Shared_Dirty as Shared, right? > > When doing this I also noticed that there are duplicate > entries in smaps. Any ideas why? Each pair of address per line indicates the start and end address of a memory area (VMA) such as: b7f7d000-b7f7e000 This means that an specific memory area start on virtual address b7f7d000 and end on b7f7e000 . An mapped file like /lib/ld-2.3.3.so is organized in different memory areas. The memory area can be a text section, data section or bss. So it is normal you find the same filename mapped in more than one memory area. You can find more information about VMA on Linux Kernel Development book (chapter 14) written by Robert Love. For instance: > grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c > >2 b7f7d000-b7f7e000 r-xp 03:05 246646 > /usr/lib/httpd/modules/mod_auth_anon.so This is a text section. >2 b7f7e000-b7f7f000 rwxp 03:05 246646 > /usr/lib/httpd/modules/mod_auth_anon.so This should be a data section. IMHO, bss section corresponds to the anonymous area where the mapping is not backed by a file. BR, Mauricio Lin. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Mauricio Lin wrote: Hi, On 7/12/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: Andrew Morton wrote: OK, please let us know how it goes. It went very well. I could find no problems at all. I've updated my script to use the new method, so please merge smaps :) http://www.pixelbeat.org/scripts/ps_mem.py Usually the shared mem reported by /proc/$$/statm is the same as summing all the shared values in in /proc/$$/smaps but there can be large discrepancies. Have you checked how the statm shared is calculated? I guess it does something like: shared = mm->rss - mm->anon_rss yes But in smaps output you can have anonymous area like: b6e0e000-b6e13000 rw-p Size:20 KB Rss: 4 KB Shared_Clean: 0 KB Shared_Dirty: 4 KB Private_Clean:0 KB Private_Dirty:0 KB Look that it presents 4 KB of shared value in area considered anonymous. ANDREW: anon_rss is the rss for anonymous area, right? I see your point and I'm not sure. The following shell gets the shared values for the first httpd process: FIRST_HTTPD=`ps -C httpd -o pid= | head -1 | tr -d ' '` HTTPD_STATM_SHARED=$(expr 4 '*' `cut -f3 -d' ' /proc/$FIRST_HTTPD/statm`) HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc) This shows that "smaps" reports 3060 KB more shared mem than "statm". However adding up all the anon sections in smaps only gives 2456 KB? When doing this I also noticed that there are duplicate entries in smaps. Any ideas why? grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c 2 b7f7d000-b7f7e000 r-xp 03:05 246646 /usr/lib/httpd/modules/mod_auth_anon.so 2 b7f7e000-b7f7f000 rwxp 03:05 246646 /usr/lib/httpd/modules/mod_auth_anon.so 2 b7f7f000-b7f81000 r-xp 03:05 246645 /usr/lib/httpd/modules/mod_auth.so 2 b7f81000-b7f82000 rwxp 1000 03:05 246645 /usr/lib/httpd/modules/mod_auth.so 2 b7f82000-b7f84000 r-xp 03:05 246641 /usr/lib/httpd/modules/mod_access.so 2 b7f84000-b7f85000 rwxp 1000 03:05 246641 /usr/lib/httpd/modules/mod_access.so 2 b7f85000-b7f9a000 r-xp 03:05 361234 /lib/ld-2.3.3.so 2 b7f9a000-b7f9b000 r-xp 00014000 03:05 361234 /lib/ld-2.3.3.so 2 b7f9b000-b7f9c000 rwxp 00015000 03:05 361234 /lib/ld-2.3.3.so 2 bfb85000-bfb9a000 rw-p bfb85000 00:00 0 [stack] 2 e000-f000 ---p 00:00 0 [vdso] If you factor that in, it means that smaps will report 156 KB too much shared mem in this example. In the real world you can see this with a newly started apache. On my system statm reported that apache was using 35MB, whereas smaps reported the correct amount of 11MB. How dou you know that 11MB is the correct shared value and the 35MB is the wrong value? Well I'm quite sure that COW pages ar not accounted for in the statm shared value, which can be easily seen with my previously posted test program. Also putting the machine into swap, and then using httpd again causes the value reported to be very close to 11MB (i.e. the unused pages are not swapped back in). -- Pádraig Brady - http://www.pixelbeat.org -- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Mauricio Lin wrote: Hi, On 7/12/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Andrew Morton wrote: OK, please let us know how it goes. It went very well. I could find no problems at all. I've updated my script to use the new method, so please merge smaps :) http://www.pixelbeat.org/scripts/ps_mem.py Usually the shared mem reported by /proc/$$/statm is the same as summing all the shared values in in /proc/$$/smaps but there can be large discrepancies. Have you checked how the statm shared is calculated? I guess it does something like: shared = mm-rss - mm-anon_rss yes But in smaps output you can have anonymous area like: b6e0e000-b6e13000 rw-p Size:20 KB Rss: 4 KB Shared_Clean: 0 KB Shared_Dirty: 4 KB Private_Clean:0 KB Private_Dirty:0 KB Look that it presents 4 KB of shared value in area considered anonymous. ANDREW: anon_rss is the rss for anonymous area, right? I see your point and I'm not sure. The following shell gets the shared values for the first httpd process: FIRST_HTTPD=`ps -C httpd -o pid= | head -1 | tr -d ' '` HTTPD_STATM_SHARED=$(expr 4 '*' `cut -f3 -d' ' /proc/$FIRST_HTTPD/statm`) HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc) This shows that smaps reports 3060 KB more shared mem than statm. However adding up all the anon sections in smaps only gives 2456 KB? When doing this I also noticed that there are duplicate entries in smaps. Any ideas why? grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c 2 b7f7d000-b7f7e000 r-xp 03:05 246646 /usr/lib/httpd/modules/mod_auth_anon.so 2 b7f7e000-b7f7f000 rwxp 03:05 246646 /usr/lib/httpd/modules/mod_auth_anon.so 2 b7f7f000-b7f81000 r-xp 03:05 246645 /usr/lib/httpd/modules/mod_auth.so 2 b7f81000-b7f82000 rwxp 1000 03:05 246645 /usr/lib/httpd/modules/mod_auth.so 2 b7f82000-b7f84000 r-xp 03:05 246641 /usr/lib/httpd/modules/mod_access.so 2 b7f84000-b7f85000 rwxp 1000 03:05 246641 /usr/lib/httpd/modules/mod_access.so 2 b7f85000-b7f9a000 r-xp 03:05 361234 /lib/ld-2.3.3.so 2 b7f9a000-b7f9b000 r-xp 00014000 03:05 361234 /lib/ld-2.3.3.so 2 b7f9b000-b7f9c000 rwxp 00015000 03:05 361234 /lib/ld-2.3.3.so 2 bfb85000-bfb9a000 rw-p bfb85000 00:00 0 [stack] 2 e000-f000 ---p 00:00 0 [vdso] If you factor that in, it means that smaps will report 156 KB too much shared mem in this example. In the real world you can see this with a newly started apache. On my system statm reported that apache was using 35MB, whereas smaps reported the correct amount of 11MB. How dou you know that 11MB is the correct shared value and the 35MB is the wrong value? Well I'm quite sure that COW pages ar not accounted for in the statm shared value, which can be easily seen with my previously posted test program. Also putting the machine into swap, and then using httpd again causes the value reported to be very close to 11MB (i.e. the unused pages are not swapped back in). -- Pádraig Brady - http://www.pixelbeat.org -- - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Hi Brady, On 7/20/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Mauricio Lin wrote: Hi, On 7/12/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Andrew Morton wrote: OK, please let us know how it goes. It went very well. I could find no problems at all. I've updated my script to use the new method, so please merge smaps :) http://www.pixelbeat.org/scripts/ps_mem.py Usually the shared mem reported by /proc/$$/statm is the same as summing all the shared values in in /proc/$$/smaps but there can be large discrepancies. Have you checked how the statm shared is calculated? I guess it does something like: shared = mm-rss - mm-anon_rss yes But in smaps output you can have anonymous area like: b6e0e000-b6e13000 rw-p Size:20 KB Rss: 4 KB Shared_Clean: 0 KB Shared_Dirty: 4 KB Private_Clean:0 KB Private_Dirty:0 KB Look that it presents 4 KB of shared value in area considered anonymous. ANDREW: anon_rss is the rss for anonymous area, right? I see your point and I'm not sure. The following shell gets the shared values for the first httpd process: FIRST_HTTPD=`ps -C httpd -o pid= | head -1 | tr -d ' '` HTTPD_STATM_SHARED=$(expr 4 '*' `cut -f3 -d' ' /proc/$FIRST_HTTPD/statm`) HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc) This shows that smaps reports 3060 KB more shared mem than statm. However adding up all the anon sections in smaps only gives 2456 KB? You are adding up all Shared_Clean and Shared_Dirty as Shared, right? When doing this I also noticed that there are duplicate entries in smaps. Any ideas why? Each pair of address per line indicates the start and end address of a memory area (VMA) such as: b7f7d000-b7f7e000 This means that an specific memory area start on virtual address b7f7d000 and end on b7f7e000 . An mapped file like /lib/ld-2.3.3.so is organized in different memory areas. The memory area can be a text section, data section or bss. So it is normal you find the same filename mapped in more than one memory area. You can find more information about VMA on Linux Kernel Development book (chapter 14) written by Robert Love. For instance: grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c 2 b7f7d000-b7f7e000 r-xp 03:05 246646 /usr/lib/httpd/modules/mod_auth_anon.so This is a text section. 2 b7f7e000-b7f7f000 rwxp 03:05 246646 /usr/lib/httpd/modules/mod_auth_anon.so This should be a data section. IMHO, bss section corresponds to the anonymous area where the mapping is not backed by a file. BR, Mauricio Lin. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Mauricio Lin wrote: Hi Brady, On 7/20/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: The following shell gets the shared values for the first httpd process: FIRST_HTTPD=`ps -C httpd -o pid= | head -1 | tr -d ' '` HTTPD_STATM_SHARED=$(expr 4 '*' `cut -f3 -d' ' /proc/$FIRST_HTTPD/statm`) HTTPD_SMAPS_SHARED=$(grep Shared /proc/$FIRST_HTTPD/smaps | tr -s ' ' | cut -f2 -d' ' | ( tr '\n' +; echo 0 ) | bc) This shows that smaps reports 3060 KB more shared mem than statm. However adding up all the anon sections in smaps only gives 2456 KB? You are adding up all Shared_Clean and Shared_Dirty as Shared, right? yes. Look at the command I posted. When doing this I also noticed that there are duplicate entries in smaps. Any ideas why? Each pair of address per line indicates the start and end address of a memory area (VMA) such as: b7f7d000-b7f7e000 This means that an specific memory area start on virtual address b7f7d000 and end on b7f7e000 . An mapped file like /lib/ld-2.3.3.so is organized in different memory areas. The memory area can be a text section, data section or bss. So it is normal you find the same filename mapped in more than one memory area. yes. Look at the command I posted. There are multiple entries with the _same addresses_ grep -F - /proc/$FIRST_HTTPD/smaps | sort | uniq -d -c -- Pádraig Brady - http://www.pixelbeat.org -- - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Hi, On 7/12/05, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > Andrew Morton wrote: > > OK, please let us know how it goes. > > It went very well. I could find no problems at all. > I've updated my script to use the new method, so please merge smaps :) > http://www.pixelbeat.org/scripts/ps_mem.py > > Usually the shared mem reported by /proc/$$/statm > is the same as summing all the shared values in in /proc/$$/smaps > but there can be large discrepancies. Have you checked how the statm shared is calculated? I guess it does something like: shared = mm->rss - mm->anon_rss But in smaps output you can have anonymous area like: b6e0e000-b6e13000 rw-p Size:20 KB Rss: 4 KB Shared_Clean: 0 KB Shared_Dirty: 4 KB Private_Clean:0 KB Private_Dirty:0 KB Look that it presents 4 KB of shared value in area considered anonymous. ANDREW: anon_rss is the rss for anonymous area, right? > In the real world you can see this with a newly started apache. > On my system statm reported that apache was using 35MB, > whereas smaps reported the correct amount of 11MB. How dou you know that 11MB is the correct shared value and the 35MB is the wrong value? BR, Mauricio Lin. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Hi, On 7/12/05, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Andrew Morton wrote: OK, please let us know how it goes. It went very well. I could find no problems at all. I've updated my script to use the new method, so please merge smaps :) http://www.pixelbeat.org/scripts/ps_mem.py Usually the shared mem reported by /proc/$$/statm is the same as summing all the shared values in in /proc/$$/smaps but there can be large discrepancies. Have you checked how the statm shared is calculated? I guess it does something like: shared = mm-rss - mm-anon_rss But in smaps output you can have anonymous area like: b6e0e000-b6e13000 rw-p Size:20 KB Rss: 4 KB Shared_Clean: 0 KB Shared_Dirty: 4 KB Private_Clean:0 KB Private_Dirty:0 KB Look that it presents 4 KB of shared value in area considered anonymous. ANDREW: anon_rss is the rss for anonymous area, right? In the real world you can see this with a newly started apache. On my system statm reported that apache was using 35MB, whereas smaps reported the correct amount of 11MB. How dou you know that 11MB is the correct shared value and the 35MB is the wrong value? BR, Mauricio Lin. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Andrew Morton wrote:
OK, please let us know how it goes.
It went very well. I could find no problems at all.
I've updated my script to use the new method, so please merge smaps :)
http://www.pixelbeat.org/scripts/ps_mem.py
Usually the shared mem reported by /proc/$$/statm
is the same as summing all the shared values in in /proc/$$/smaps
but there can be large discrepancies.
In the real world you can see this with a newly started apache.
On my system statm reported that apache was using 35MB,
whereas smaps reported the correct amount of 11MB.
These numbers will converge over time as the unused pages
of the apache processes were swapped out.
Attached is a mem.py script that triggers and reports
the memory discrepancy on an smaps enabled system.
As for the overhead of calling smaps, well comparing to
/proc/$$/maps which does essentially the same thing only
not walking mem, on my 1.6GHz P4, 384MB laptop, gives:
time read_10K_smaps
real0m3.323s
user0m0.636s
sys 0m2.660s
time read_10K_maps
real0m1.742s
user0m0.300s
sys 0m1.428s
I've also attached a patch to make smaps output match the maps output,
as smaps was not showing info like which map was heap and stack etc.
The code is smaller also, as show_smap() now just calls show_map().
Note this patch is to be applied _after_ Hugh's changes.
--
Pádraig Brady - http://www.pixelbeat.org
--
#!/usr/bin/python
import os, sys, time
PAGESIZE=os.sysconf("SC_PAGE_SIZE")/1024 #KiB
#stat = {Shared,Rss}
#file = {smaps,statm}
def getMemStat(pid,stat,file):
if file=="smaps":
shared_lines=[line
for line in open("/proc/"+str(pid)+"/smaps").readlines()
if line.find(stat)!=-1]
return sum([int(line.split()[1]) for line in shared_lines])
else:
if stat == "Shared": stat_index=2
elif stat == "Rss": stat_index=1
return int(open("/proc/"+str(pid)+"/statm").readline().split()[stat_index])*PAGESIZE
a="\x00"*(32*1024*1024) #alloc 32MiB
pid=os.fork()
if(not pid):
time.sleep(1)
sys.exit()
for file in ["statm", "smaps"]:
rss=float(getMemStat(pid,"Rss",file))
shr=float(getMemStat(pid,"Shared",file))
print "/proc/%d/%s reports %.1f%% shared" % (pid,file,((shr/rss)*100))
os.wait()
linux-2.6.13-rc2-mm1-hd-smaps.diff
Description: TeXInfo document
Re: How do you accurately determine a process' RAM usage?
Andrew Morton wrote:
OK, please let us know how it goes.
It went very well. I could find no problems at all.
I've updated my script to use the new method, so please merge smaps :)
http://www.pixelbeat.org/scripts/ps_mem.py
Usually the shared mem reported by /proc/$$/statm
is the same as summing all the shared values in in /proc/$$/smaps
but there can be large discrepancies.
In the real world you can see this with a newly started apache.
On my system statm reported that apache was using 35MB,
whereas smaps reported the correct amount of 11MB.
These numbers will converge over time as the unused pages
of the apache processes were swapped out.
Attached is a mem.py script that triggers and reports
the memory discrepancy on an smaps enabled system.
As for the overhead of calling smaps, well comparing to
/proc/$$/maps which does essentially the same thing only
not walking mem, on my 1.6GHz P4, 384MB laptop, gives:
time read_10K_smaps
real0m3.323s
user0m0.636s
sys 0m2.660s
time read_10K_maps
real0m1.742s
user0m0.300s
sys 0m1.428s
I've also attached a patch to make smaps output match the maps output,
as smaps was not showing info like which map was heap and stack etc.
The code is smaller also, as show_smap() now just calls show_map().
Note this patch is to be applied _after_ Hugh's changes.
--
Pádraig Brady - http://www.pixelbeat.org
--
#!/usr/bin/python
import os, sys, time
PAGESIZE=os.sysconf(SC_PAGE_SIZE)/1024 #KiB
#stat = {Shared,Rss}
#file = {smaps,statm}
def getMemStat(pid,stat,file):
if file==smaps:
shared_lines=[line
for line in open(/proc/+str(pid)+/smaps).readlines()
if line.find(stat)!=-1]
return sum([int(line.split()[1]) for line in shared_lines])
else:
if stat == Shared: stat_index=2
elif stat == Rss: stat_index=1
return int(open(/proc/+str(pid)+/statm).readline().split()[stat_index])*PAGESIZE
a=\x00*(32*1024*1024) #alloc 32MiB
pid=os.fork()
if(not pid):
time.sleep(1)
sys.exit()
for file in [statm, smaps]:
rss=float(getMemStat(pid,Rss,file))
shr=float(getMemStat(pid,Shared,file))
print /proc/%d/%s reports %.1f%% shared % (pid,file,((shr/rss)*100))
os.wait()
linux-2.6.13-rc2-mm1-hd-smaps.diff
Description: TeXInfo document
Re: How do you accurately determine a process' RAM usage?
On Thu, 7 Jul 2005, Andrew Morton wrote:
> [EMAIL PROTECTED] wrote:
>
> > Looks like it's been stable for 4 months?
>
> yup, although I don't think it's been used much.
Just a sidenote to say I should be sending you an update to it
(the /proc/$pid/smaps code) in the next couple of days, but merely
cosmetic ("map" -> "vma", cond_resched_lock, p?d_addr_next ptwalking).
I'm still pretty sceptical about it, but it's probably a useful
framework for people to hack on, to report whatever kind of page
numbers they're interested in for this or that investigation.
Hugh
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
[EMAIL PROTECTED] wrote: > > Andrew Morton wrote: > > Calculating this stuff accurately is very expensive. You'll get a better > > answer using proc-pid-smaps.patch from -mm, but even that won't tell you > > things about sharing levels of the pages. > > Great, thanks! I'll play around with this: > http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc1/2.6.13-rc1-mm1/broken-out/proc-pid-smaps.patch OK, please let us know how it goes. > Looks like it's been stable for 4 months? yup, although I don't think it's been used much. > Given that it's an independent /proc/$pid/smaps file, > it only needs to be queried when required and so > I wouldn't worry too much about cost. `top` wouldn't use it > for e.g., but specialised tools like mine would. I agree, but people get upset ;) Plus some userspace tool developer might see it and start using it without knowing the cost on big iron. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Andrew Morton wrote: Calculating this stuff accurately is very expensive. You'll get a better answer using proc-pid-smaps.patch from -mm, but even that won't tell you things about sharing levels of the pages. Great, thanks! I'll play around with this: http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc1/2.6.13-rc1-mm1/broken-out/proc-pid-smaps.patch Looks like it's been stable for 4 months? Given that it's an independent /proc/$pid/smaps file, it only needs to be queried when required and so I wouldn't worry too much about cost. `top` wouldn't use it for e.g., but specialised tools like mine would. -- Pádraig Brady - http://www.pixelbeat.org -- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
Andrew Morton wrote: Calculating this stuff accurately is very expensive. You'll get a better answer using proc-pid-smaps.patch from -mm, but even that won't tell you things about sharing levels of the pages. Great, thanks! I'll play around with this: http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc1/2.6.13-rc1-mm1/broken-out/proc-pid-smaps.patch Looks like it's been stable for 4 months? Given that it's an independent /proc/$pid/smaps file, it only needs to be queried when required and so I wouldn't worry too much about cost. `top` wouldn't use it for e.g., but specialised tools like mine would. -- Pádraig Brady - http://www.pixelbeat.org -- - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
[EMAIL PROTECTED] wrote: Andrew Morton wrote: Calculating this stuff accurately is very expensive. You'll get a better answer using proc-pid-smaps.patch from -mm, but even that won't tell you things about sharing levels of the pages. Great, thanks! I'll play around with this: http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc1/2.6.13-rc1-mm1/broken-out/proc-pid-smaps.patch OK, please let us know how it goes. Looks like it's been stable for 4 months? yup, although I don't think it's been used much. Given that it's an independent /proc/$pid/smaps file, it only needs to be queried when required and so I wouldn't worry too much about cost. `top` wouldn't use it for e.g., but specialised tools like mine would. I agree, but people get upset ;) Plus some userspace tool developer might see it and start using it without knowing the cost on big iron. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
On Thu, 7 Jul 2005, Andrew Morton wrote: [EMAIL PROTECTED] wrote: Looks like it's been stable for 4 months? yup, although I don't think it's been used much. Just a sidenote to say I should be sending you an update to it (the /proc/$pid/smaps code) in the next couple of days, but merely cosmetic (map - vma, cond_resched_lock, p?d_addr_next ptwalking). I'm still pretty sceptical about it, but it's probably a useful framework for people to hack on, to report whatever kind of page numbers they're interested in for this or that investigation. Hugh - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
[EMAIL PROTECTED] wrote: > > I wrote a tool to report how much RAM a > particular program (apache for e.g.) was using: > http://www.pixelbeat.org/scripts/ps_mem.py > > I was then pointed at the following: > http://wiki.apache.org/spamassassin/TopSharedMemoryBug > which describes how copy-on-write pages are > not counted as shared since 2.6. > > So how can one determine how much RAM a process is using? > Seems like a fundamental requirement to me. > Could we add a "SharedTotal" column to /proc/$$/statm for e.g. ? Calculating this stuff accurately is very expensive. You'll get a better answer using proc-pid-smaps.patch from -mm, but even that won't tell you things about sharing levels of the pages. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
How do you accurately determine a process RAM usage
I wrote a tool to (try to) report how much RAM a
particular program (apache for e.g.) was using:
http://www.pixelbeat.org/scripts/ps_mem.py
I was then pointed at the following:
http://wiki.apache.org/spamassassin/TopSharedMemoryBug
which describes how copy-on-write pages are
not counted as shared since 2.6.
So how can one determine how much RAM a process is using?
Seems like a fundamental requirement to me.
Could we add a "SharedTotal" column to /proc/$$/statm for e.g. ?
Here's a little python script to demo the problem:
#!/usr/bin/python
import os, sys, time
a="\x00"*(32*1024*1024) #alloc 32MiB
pid=os.fork()
if(not pid):
time.sleep(1)
sys.exit()
sizel=map(float,open("/proc/"+str(pid)+"/statm").readlines()[0].split()[1:3])
print "only %.1f%% reported as shared" % ((sizel[1]/sizel[0])*100)
os.wait()
--
Pádraig Brady - http://www.pixelbeat.org
--
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
How do you accurately determine a process' RAM usage?
I wrote a tool to report how much RAM a
particular program (apache for e.g.) was using:
http://www.pixelbeat.org/scripts/ps_mem.py
I was then pointed at the following:
http://wiki.apache.org/spamassassin/TopSharedMemoryBug
which describes how copy-on-write pages are
not counted as shared since 2.6.
So how can one determine how much RAM a process is using?
Seems like a fundamental requirement to me.
Could we add a "SharedTotal" column to /proc/$$/statm for e.g. ?
Here's a little python script to demo the problem:
#!/usr/bin/python
import os, sys, time
a="\x00"*(32*1024*1024) #alloc 32MiB
pid=os.fork()
if(not pid):
time.sleep(1)
sys.exit()
sizel=map(float,open("/proc/"+str(pid)+"/statm").readlines()[0].split()[1:3])
print "only %.1f%% reported as shared" % ((sizel[1]/sizel[0])*100)
os.wait()
--
Pádraig Brady - http://www.pixelbeat.org
--
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
How do you accurately determine a process' RAM usage?
I wrote a tool to report how much RAM a particular program (apache for e.g.) was using: http://www.pixelbeat.org/scripts/ps_mem.py I was then pointed at the following: http://wiki.apache.org/spamassassin/TopSharedMemoryBug which describes how copy-on-write pages are not counted as shared since 2.6. So how can one determine how much RAM a process is using? Seems like a fundamental requirement to me. Could we add a SharedTotal column to /proc/$$/statm for e.g. ? Here's a little python script to demo the problem: #!/usr/bin/python import os, sys, time a=\x00*(32*1024*1024) #alloc 32MiB pid=os.fork() if(not pid): time.sleep(1) sys.exit() sizel=map(float,open(/proc/+str(pid)+/statm).readlines()[0].split()[1:3]) print only %.1f%% reported as shared % ((sizel[1]/sizel[0])*100) os.wait() -- Pádraig Brady - http://www.pixelbeat.org -- - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
How do you accurately determine a process RAM usage
I wrote a tool to (try to) report how much RAM a particular program (apache for e.g.) was using: http://www.pixelbeat.org/scripts/ps_mem.py I was then pointed at the following: http://wiki.apache.org/spamassassin/TopSharedMemoryBug which describes how copy-on-write pages are not counted as shared since 2.6. So how can one determine how much RAM a process is using? Seems like a fundamental requirement to me. Could we add a SharedTotal column to /proc/$$/statm for e.g. ? Here's a little python script to demo the problem: #!/usr/bin/python import os, sys, time a=\x00*(32*1024*1024) #alloc 32MiB pid=os.fork() if(not pid): time.sleep(1) sys.exit() sizel=map(float,open(/proc/+str(pid)+/statm).readlines()[0].split()[1:3]) print only %.1f%% reported as shared % ((sizel[1]/sizel[0])*100) os.wait() -- Pádraig Brady - http://www.pixelbeat.org -- - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: How do you accurately determine a process' RAM usage?
[EMAIL PROTECTED] wrote: I wrote a tool to report how much RAM a particular program (apache for e.g.) was using: http://www.pixelbeat.org/scripts/ps_mem.py I was then pointed at the following: http://wiki.apache.org/spamassassin/TopSharedMemoryBug which describes how copy-on-write pages are not counted as shared since 2.6. So how can one determine how much RAM a process is using? Seems like a fundamental requirement to me. Could we add a SharedTotal column to /proc/$$/statm for e.g. ? Calculating this stuff accurately is very expensive. You'll get a better answer using proc-pid-smaps.patch from -mm, but even that won't tell you things about sharing levels of the pages. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
