Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
on 02/12/2011 19:13 Steven Hartland said the following: [snip] Given this starting point the following links provided me with addtional information:- http://www.freebsd.org/doc/en/books/arch-handbook/vm.html http://www.freebsd.org/doc/en/books/design-44bsd/overview-memory-management.html http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/ http://www.cse.chalmers.se/edu/course/EDA203/unix4.pdf I was under the incorrect impression that Virtual Memory (VM) was so named as it was a unified physical memory and swap (virtual memory), but its not that simple, as other items such as file-backed objects also count to this total which would never show in physical or swap allocation of other tools such as top and swapinfo. So what I believe is now the big cause of virtual memory uplift vs the memory totals shown by ps / top is that the vm totals include things like file backed memory mapped process binaries, shared libs etc many multiple times. This would explain why this specific machine shows the applification more than others here as it runs thousands of very small lightweight processes. Thanks for pointer Andy, I now understand a lot more about the BSD VMS :) What do people think about expanding that entry in the man page of vmstat to clarify just what active virtual pages really means? Thank you for the excellent additional research. I think that the expanded explanation would be very appreciated, but only if it is completely correct and unambiguous from technical perspective. That might not be trivial to achieve, but is realistic :-) P.S. I think that it would also be nice to further breakdown VM stats by type of object e.g. anonymous (swap-backed) memory vs file-backed backed memory, etc. Proportional RSS reporting is another nice to have feature. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
Yeah On Fri, Dec 02, 2011 at 03:19:53PM +0800, Adrian Chadd wrote: .. where are these statistics coming from? top? Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
on 30/11/2011 14:39 Steven Hartland said the following: We're seeing some impossible memory usage stats reported on machines here from vmstat and sysctl vm.vmtotal. We have machines reporting to be using 31GB total when they only have 8GB physical and are not using any swap. Here's an output from one of our machines:- vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 31768M 2112M 2 0 0 0 0 0 370 8139 3996 0 1 99 The raw output is:- vmstat -c 2 -w 1 -n 0 -H procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 32530228 2162524 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 32530228 2162524 2 0 0 0 0 0 286 8234 4347 0 1 99 Top shows:- last pid: 6665; load averages: 0.00, 0.00, 0.01 up 80+01:24:12 09:35:28 1893 processes:1 running, 1892 sleeping CPU: 0.0% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.7% idle Mem: 3754M Active, 84M Inact, 1976M Wired, 4K Cache, 2109M Free Swap: 4096M Total, 4096M Free sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) === Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1893) Virtual Memory: (Total: 1106403532K Active: 32540260K) Real Memory:(Total: 4563648K Active: 3921644K) Shared Virtual Memory: (Total: 19976K Active: 16396K) Shared Real Memory: (Total: 9040K Active: 8436K) Free Memory Pages: 2161740K As mentioned this machine has 8GB of ram and according to both top and swapinfo is using no swap at all From dmesg:- real memory = 8589934592 (8192 MB) avail memory = 823536 (7873 MB) swapinfo Device 1K-blocks UsedAvail Capacity /dev/gptid/09f211f7-39ce-11e0-8 41943040 4194304 0% uname -a FreeBSD test 8.2-RELEASE FreeBSD 8.2-RELEASE #2: Thu Mar 24 17:28:55 UTC 2011 root@test:/usr/obj/usr/src/sys/MULTIPLAY amd64 sysctl hw.pagesize hw.pagesize: 4096 It looks like it may be out by a factor of 4, possibly due to the fact the its a 4k page size not 1k as indicated by the vmstat man page:- memory Information about the usage of virtual and real memory. Virtual pages (reported in units of 1024 bytes) are considered active if they belong to processes which are running or have run in the last 20 seconds. avm active virtual pages fre size of the free list I think that the description of avm is not sufficiently complete as it doesn't explain meaning of the virtual memory it refers too. Reference to units of 1024 bytes when talking about pages also sounds a bit vague. The values reported are actually byte sizes, non-humanized numbers are in KB (units of 1024 bytes in the poetic speech), humanized numbers carry an appropriate suffix. Totalling up RSS from ps axo rss gives a total in the region of that if the vm stats are out by a factor of 4, in this case it should be: 8132557 which is 7.75GB a much more realistic value. Am I totally missing something or is there problem here? Likely more of the former than of latter. Those virtual sizes are not sufficiently explained, but you have been warned that those are not physical sizes, so I am not sure why you try to compare the virtual figures with the physical figures. Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, here you immediately increased your virtual size counts by 1GB, even if you hadn't accessed any bytes in the file yet and so none of them were in physical memory. The same applies to anonymous memory. P.S. the above is reveled by a cursory look through the code (which is publicly available btw) :-) -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
On 12/1/11 1:05 PM, Jason Hellenthal wrote: On Thu, Dec 01, 2011 at 10:44:58AM -, Steven Hartland wrote: - Original Message - From: Jason Hellenthaljh...@dataix.net This goes along with the thoughts I had about 4 months ago tending to some zfs statistics as well top showing greater than 100% actual CPU usage. This is a big pet peave of mine. Its like saying you ate 134% of a bannanna when in all reallity it is impossible. You can never have more than 100% usage of anything and when seen is a clear notice that some math is considerably incorrect leading to other such miscalculations to be performed. Things like the above already have checks in place that ensure no boundries are being crossed/overflowed or underrun but it surely makes processing results building future products a bitch. One instance is the calculation of threads for example firefox can be seen using upto or more 338% of the CPU. Thats impossible its like saying anyones CPU grew by 400%. I could understand a bit of overflow as stats are snapshots which may not be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / overflow. I agree With respect to top showing greater than 100% by how much are you talking? Do your realise that each core = 100%? So if you have a quad core your system total will be 400% not 100%? Yeah I realize that but it still would lead you to believe that if a proccessor has 4 cores on the same die then total for each core could only be 25% usage. And the usage for a proccess only consuming full usage of 1 core is 100%. But you can start firefox on a single uniproccessor and like stated above see large usage percents near 338% or greater which is impossible and leads me to believe were forcing calculation for the entire proccess of threads onto tthread 0. This makes accounting pretty difficult. A single-package machine with 4 cores on the die is not a uniprocessor machine. It is an SMP machine. Try booting a kernel without SMP, _that_ will give you a UP machine. The %CPU usage is actually very simple if you stop trying to make it so complicated. Each potential concurrent thread of execution is mapped to 100%. Thus, if you have a system with 16 potential threads (either due to 4 quad-core packages, or 2 quad-core package where each core has 2 threads via HTT, etc.), then you have a total CPU usage of 1600%. Why does this make sense? Because a machine with 16 concurrent threads can (theoretically) do 16x the work of an otherwise identical machine with a single thread of execution within a given unit of wall time. Another way to look at it is that in FreeBSD, each thread of execution is treated as a CPU in top, etc. Thus, 100% CPU means that a given thread is using all of the available cycles on a CPU. If you have a multithreaded app (like Firefox) that is using all of the available cycles on 4 CPUs, then that would be 400% CPU (it's using 4 CPUs). It may only be using 25% of the available system-wide CPU cycles, but that is not what %CPU measures. Anyways, pretty much everyone I've ever talked to about this gets it right away, so the current arrangement is fine for the majority of folks. It has also been that way since FreeBSD first added SMP over a decade ago. It is also true on other OS's such as OS X, so the current arrangement is here to stay. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
- Original Message - From: Jason Hellenthal jh...@dataix.net Just to put some visuals to this... . `-- DIE |-- Core1 [Idle] |-- Core2 [35% ] | `-- thread127 |-- Core3 [40% ] | `-- thread127 `-- Core4 [100%] `-- thread127 In this case you would say the DIE should be at a total of 175% ? I think your getting confused there; it sounds like your referring to a single CPU capable of multiple tasks via either multiple cores or HTT as an UP machine? If so that's your problem this isn't UP it SMP. Have a look on your machine in /var/run/dmesg.boot if your see it reporting more than one core then your SMP not UP hence the confusion as each of these cores be they real, virtual or physical represents a possible thread of 100% so even if you have a single physical CPU with 4 cores that still represents a possible total of 400% e.g the following shows a machine capable of 1600% if all cores are busy. FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 16 cpu9 (AP): APIC ID: 17 cpu10 (AP): APIC ID: 18 cpu11 (AP): APIC ID: 19 cpu12 (AP): APIC ID: 20 cpu13 (AP): APIC ID: 21 cpu14 (AP): APIC ID: 22 cpu15 (AP): APIC ID: 23 If you want proper UP which will total 100% you could remove SMP from your kernel but I wouldnt advise that ;-) Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
- Original Message - From: Andriy Gapon a...@freebsd.org Totalling up RSS from ps axo rss gives a total in the region of that if the vm stats are out by a factor of 4, in this case it should be: 8132557 which is 7.75GB a much more realistic value. Am I totally missing something or is there problem here? Likely more of the former than of latter. Those virtual sizes are not sufficiently explained, but you have been warned that those are not physical sizes, so I am not sure why you try to compare the virtual figures with the physical figures. My miss-understanding was due to what virtual actually meant. Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, here you immediately increased your virtual size counts by 1GB, even if you hadn't accessed any bytes in the file yet and so none of them were in physical memory. The same applies to anonymous memory. P.S. the above is reveled by a cursory look through the code (which is publicly available btw) :-) Yer I did have a dig around before posting and ended up the code for vm.vmtotal, which is where vmstat gets its info from but that's just a summation of each object's size from vm_object_list. Thats where I got lost without an insight into what a vm_object.size actually represents. Your info about mmap'ed files helped point me in the right direction as it identified space that shows as virtual but doesn't show in swap or real ram, which is what I was missing. Given this starting point the following links provided me with addtional information:- http://www.freebsd.org/doc/en/books/arch-handbook/vm.html http://www.freebsd.org/doc/en/books/design-44bsd/overview-memory-management.html http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/ http://www.cse.chalmers.se/edu/course/EDA203/unix4.pdf I was under the incorrect impression that Virtual Memory (VM) was so named as it was a unified physical memory and swap (virtual memory), but its not that simple, as other items such as file-backed objects also count to this total which would never show in physical or swap allocation of other tools such as top and swapinfo. So what I believe is now the big cause of virtual memory uplift vs the memory totals shown by ps / top is that the vm totals include things like file backed memory mapped process binaries, shared libs etc many multiple times. This would explain why this specific machine shows the applification more than others here as it runs thousands of very small lightweight processes. Thanks for pointer Andy, I now understand a lot more about the BSD VMS :) What do people think about expanding that entry in the man page of vmstat to clarify just what active virtual pages really means? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
On Fri, Dec 02, 2011 at 05:13:05PM -, Steven Hartland wrote: - Original Message - From: Andriy Gapon a...@freebsd.org Totalling up RSS from ps axo rss gives a total in the region of that if the vm stats are out by a factor of 4, in this case it should be: 8132557 which is 7.75GB a much more realistic value. Am I totally missing something or is there problem here? Likely more of the former than of latter. Those virtual sizes are not sufficiently explained, but you have been warned that those are not physical sizes, so I am not sure why you try to compare the virtual figures with the physical figures. My miss-understanding was due to what virtual actually meant. Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, here you immediately increased your virtual size counts by 1GB, even if you hadn't accessed any bytes in the file yet and so none of them were in physical memory. The same applies to anonymous memory. P.S. the above is reveled by a cursory look through the code (which is publicly available btw) :-) Yer I did have a dig around before posting and ended up the code for vm.vmtotal, which is where vmstat gets its info from but that's just a summation of each object's size from vm_object_list. Thats where I got lost without an insight into what a vm_object.size actually represents. Your info about mmap'ed files helped point me in the right direction as it identified space that shows as virtual but doesn't show in swap or real ram, which is what I was missing. Given this starting point the following links provided me with addtional information:- http://www.freebsd.org/doc/en/books/arch-handbook/vm.html http://www.freebsd.org/doc/en/books/design-44bsd/overview-memory-management.html http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/ http://www.cse.chalmers.se/edu/course/EDA203/unix4.pdf I was under the incorrect impression that Virtual Memory (VM) was so named as it was a unified physical memory and swap (virtual memory), but its not that simple, as other items such as file-backed objects also count to this total which would never show in physical or swap allocation of other tools such as top and swapinfo. So what I believe is now the big cause of virtual memory uplift vs the memory totals shown by ps / top is that the vm totals include things like file backed memory mapped process binaries, shared libs etc many multiple times. This would explain why this specific machine shows the applification more than others here as it runs thousands of very small lightweight processes. Thanks for pointer Andy, I now understand a lot more about the BSD VMS :) What do people think about expanding that entry in the man page of vmstat to clarify just what active virtual pages really means? Regards Steve Thanks for your research Steve. That makes perfect sense and additions to the documentation are surely needed. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
- Original Message - From: Jason Hellenthal jh...@dataix.net This goes along with the thoughts I had about 4 months ago tending to some zfs statistics as well top showing greater than 100% actual CPU usage. This is a big pet peave of mine. Its like saying you ate 134% of a bannanna when in all reallity it is impossible. You can never have more than 100% usage of anything and when seen is a clear notice that some math is considerably incorrect leading to other such miscalculations to be performed. Things like the above already have checks in place that ensure no boundries are being crossed/overflowed or underrun but it surely makes processing results building future products a bitch. One instance is the calculation of threads for example firefox can be seen using upto or more 338% of the CPU. Thats impossible its like saying anyones CPU grew by 400%. I could understand a bit of overflow as stats are snapshots which may not be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / overflow. With respect to top showing greater than 100% by how much are you talking? Do your realise that each core = 100%? So if you have a quad core your system total will be 400% not 100%? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
On 12/1/11 11:44 AM, Steven Hartland wrote: - Original Message - From: Jason Hellenthal jh...@dataix.net This goes along with the thoughts I had about 4 months ago tending to some zfs statistics as well top showing greater than 100% actual CPU usage. This is a big pet peave of mine. Its like saying you ate 134% of a bannanna when in all reallity it is impossible. You can never have more than 100% usage of anything and when seen is a clear notice that some math is considerably incorrect leading to other such miscalculations to be performed. Things like the above already have checks in place that ensure no boundries are being crossed/overflowed or underrun but it surely makes processing results building future products a bitch. One instance is the calculation of threads for example firefox can be seen using upto or more 338% of the CPU. Thats impossible its like saying anyones CPU grew by 400%. I could understand a bit of overflow as stats are snapshots which may not be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / overflow. With respect to top showing greater than 100% by how much are you talking? Do your realise that each core = 100%? So if you have a quad core your system total will be 400% not 100%? That's his point, you cannot use 400% of a system as a whole, his point is that top should report 100% where each core accounts for 25% ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
- Original Message - From: Damien Fleuriot m...@my.gd I could understand a bit of overflow as stats are snapshots which may not be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / overflow. With respect to top showing greater than 100% by how much are you talking? Do your realise that each core = 100%? So if you have a quad core your system total will be 400% not 100%? That's his point, you cannot use 400% of a system as a whole, his point is that top should report 100% where each core accounts for 25% Then I would have to disagree, keeping 100% to mean 100% of a single core is much easier to manage than 100% of a machines total capacity. If you went to 100% = the machine total capacity processes could be using a lot of cpu without even registering 1% on today's machines where 24 cores is common place. It also makes detecting single process / thread bottlenecks easier as if your seeing 100% you know its maxing a core, instead of having to calculate it once you know how many cores the machine has. If your looking for total machine usage then that's also their in the summary at the top of the screen e.g. CPU states: 13.6% user, 0.0% nice, 1.3% system, 0.0% interrupt, 85.1% idle Anyway this is quite off topic, and I don't want to loose sight of the threads goal which is to determine why we can see 31GB of usage on an 8GB machine with very little shared memory usage an no swap usage. Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
On Wed, 30 Nov 2011 12:39:10 - Steven Hartland wrote: We're seeing some impossible memory usage stats reported on machines here from vmstat and sysctl vm.vmtotal. We have machines reporting to be using 31GB total when they only have 8GB physical and are not using any swap. Here's an output from one of our machines:- vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 31768M 2112M 2 0 0 0 0 0 370 8139 3996 0 1 99 ... It looks like it may be out by a factor of 4, possibly due to the fact the its a 4k page size not 1k as indicated by the vmstat man page:- I don't think so, I have 16GB and tops shows: Mem: 817M Active, 396M Inact, 1364M Wired, 82M Cache, 1282M Buf, 13G Free but vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 9750M13G 450 5 3 1 560 0 234 50201 5206 2 2 96 1 0 0 9742M13G79 4 0 0 573 0 239 51886 4700 0 1 99 ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
- Original Message - From: RW rwmailli...@googlemail.com To: freebsd-hackers@freebsd.org Sent: Thursday, December 01, 2011 1:58 PM Subject: Re: Invalid memory stats from vmstat and sysctl vm.vmtotal? On Wed, 30 Nov 2011 12:39:10 - Steven Hartland wrote: We're seeing some impossible memory usage stats reported on machines here from vmstat and sysctl vm.vmtotal. We have machines reporting to be using 31GB total when they only have 8GB physical and are not using any swap. Here's an output from one of our machines:- vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 31768M 2112M 2 0 0 0 0 0 370 8139 3996 0 1 99 ... It looks like it may be out by a factor of 4, possibly due to the fact the its a 4k page size not 1k as indicated by the vmstat man page:- I don't think so, I have 16GB and tops shows: Mem: 817M Active, 396M Inact, 1364M Wired, 82M Cache, 1282M Buf, 13G Free but vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 9750M13G 450 5 3 1 560 0 234 50201 5206 2 2 96 1 0 0 9742M13G79 4 0 0 573 0 239 51886 4700 0 1 99 Hmm, yes on another machine same OS with 16GB memory we see:- dmesg | grep memory real memory = 17179869184 (16384 MB) avail memory = 16536604672 (15770 MB) vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 1948M 666M 395 14 0 0 398 1966 3237 5980 14085 2 4 93 1 0 0 1948M 636M 4 0 0 016 0 8215 18660 35322 2 7 92 sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) === Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 194) Virtual Memory: (Total: 1075975332K Active: 1979056K) Real Memory:(Total: 309048K Active: 160804K) Shared Virtual Memory: (Total: 75112K Active: 18860K) Shared Real Memory: (Total: 15656K Active: 10572K) Free Memory Pages: 731960K but with top:- last pid: 38187; load averages: 0.15, 0.16, 0.16 up 2+23:33:55 15:13:02 195 processes: 1 running, 194 sleeping CPU: 0.0% user, 0.0% nice, 4.1% system, 0.0% interrupt, 95.9% idle Mem: 886M Active, 12G Inact, 2194M Wired, 599M Cache, 1630M Buf, 138M Free Swap: 8192M Total, 1512K Used, 8190M Free So I've no idea whats going on? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
On Thu, Dec 01, 2011 at 10:44:58AM -, Steven Hartland wrote: - Original Message - From: Jason Hellenthal jh...@dataix.net This goes along with the thoughts I had about 4 months ago tending to some zfs statistics as well top showing greater than 100% actual CPU usage. This is a big pet peave of mine. Its like saying you ate 134% of a bannanna when in all reallity it is impossible. You can never have more than 100% usage of anything and when seen is a clear notice that some math is considerably incorrect leading to other such miscalculations to be performed. Things like the above already have checks in place that ensure no boundries are being crossed/overflowed or underrun but it surely makes processing results building future products a bitch. One instance is the calculation of threads for example firefox can be seen using upto or more 338% of the CPU. Thats impossible its like saying anyones CPU grew by 400%. I could understand a bit of overflow as stats are snapshots which may not be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / overflow. I agree With respect to top showing greater than 100% by how much are you talking? Do your realise that each core = 100%? So if you have a quad core your system total will be 400% not 100%? Yeah I realize that but it still would lead you to believe that if a proccessor has 4 cores on the same die then total for each core could only be 25% usage. And the usage for a proccess only consuming full usage of 1 core is 100%. But you can start firefox on a single uniproccessor and like stated above see large usage percents near 338% or greater which is impossible and leads me to believe were forcing calculation for the entire proccess of threads onto tthread 0. This makes accounting pretty difficult. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
On Thu, Dec 01, 2011 at 01:23:35PM -, Steven Hartland wrote: - Original Message - From: Damien Fleuriot m...@my.gd I could understand a bit of overflow as stats are snapshots which may not be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / overflow. With respect to top showing greater than 100% by how much are you talking? Do your realise that each core = 100%? So if you have a quad core your system total will be 400% not 100%? That's his point, you cannot use 400% of a system as a whole, his point is that top should report 100% where each core accounts for 25% Then I would have to disagree, keeping 100% to mean 100% of a single core is much easier to manage than 100% of a machines total capacity. If you went to 100% = the machine total capacity processes could be using a lot of cpu without even registering 1% on today's machines where 24 cores is common place. It also makes detecting single process / thread bottlenecks easier as if your seeing 100% you know its maxing a core, instead of having to calculate it once you know how many cores the machine has. If your looking for total machine usage then that's also their in the summary at the top of the screen e.g. CPU states: 13.6% user, 0.0% nice, 1.3% system, 0.0% interrupt, 85.1% idle Anyway this is quite off topic, and I don't want to loose sight of the threads goal which is to determine why we can see 31GB of usage on an 8GB machine with very little shared memory usage an no swap usage. Just to put some visuals to this... . `-- DIE |-- Core1 [Idle] |-- Core2 [35% ] | `-- thread127 |-- Core3 [40% ] | `-- thread127 `-- Core4 [100%] `-- thread127 In this case you would say the DIE should be at a total of 175% ? $(((25*0)+(25*0.35)+(25*0.40)+(25*1.00))) Out of sanity and each core only being 25% of the total DIE it should be reporting. It is using all together 43.75% of the total DIE. But thats not what I see even on SP machines. 1 DIE 1 core and a report of 338% usage for 8 threads of firefox. If someone was attempting to write a scheduler to launch processes yield back to the scheduler to launch more based on processor usage either by core or by die totals that scheduler would be ineffective at best without alot of kludges put in place to handle all the misinterpretation. Somewhere along the line our math has been distorted or the move from SP - MP, cores, hyperthreading etc.. has just not completed yet and should not be ignored. pgpqrrUzEGGAL.pgp Description: PGP signature
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
.. where are these statistics coming from? top? Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Invalid memory stats from vmstat and sysctl vm.vmtotal?
We're seeing some impossible memory usage stats reported on machines here from vmstat and sysctl vm.vmtotal. We have machines reporting to be using 31GB total when they only have 8GB physical and are not using any swap. Here's an output from one of our machines:- vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 31768M 2112M 2 0 0 0 0 0 370 8139 3996 0 1 99 The raw output is:- vmstat -c 2 -w 1 -n 0 -H procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 32530228 2162524 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 32530228 2162524 2 0 0 0 0 0 286 8234 4347 0 1 99 Top shows:- last pid: 6665; load averages: 0.00, 0.00, 0.01 up 80+01:24:12 09:35:28 1893 processes:1 running, 1892 sleeping CPU: 0.0% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.7% idle Mem: 3754M Active, 84M Inact, 1976M Wired, 4K Cache, 2109M Free Swap: 4096M Total, 4096M Free sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) === Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1893) Virtual Memory: (Total: 1106403532K Active: 32540260K) Real Memory:(Total: 4563648K Active: 3921644K) Shared Virtual Memory: (Total: 19976K Active: 16396K) Shared Real Memory: (Total: 9040K Active: 8436K) Free Memory Pages: 2161740K As mentioned this machine has 8GB of ram and according to both top and swapinfo is using no swap at all From dmesg:- real memory = 8589934592 (8192 MB) avail memory = 823536 (7873 MB) swapinfo Device 1K-blocks UsedAvail Capacity /dev/gptid/09f211f7-39ce-11e0-8 41943040 4194304 0% uname -a FreeBSD test 8.2-RELEASE FreeBSD 8.2-RELEASE #2: Thu Mar 24 17:28:55 UTC 2011 root@test:/usr/obj/usr/src/sys/MULTIPLAY amd64 sysctl hw.pagesize hw.pagesize: 4096 It looks like it may be out by a factor of 4, possibly due to the fact the its a 4k page size not 1k as indicated by the vmstat man page:- memory Information about the usage of virtual and real memory. Virtual pages (reported in units of 1024 bytes) are considered active if they belong to processes which are running or have run in the last 20 seconds. avm active virtual pages fre size of the free list Totalling up RSS from ps axo rss gives a total in the region of that if the vm stats are out by a factor of 4, in this case it should be: 8132557 which is 7.75GB a much more realistic value. Am I totally missing something or is there problem here? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
On Wed, Nov 30, 2011 at 12:39:10PM -, Steven Hartland wrote: We're seeing some impossible memory usage stats reported on machines here from vmstat and sysctl vm.vmtotal. We have machines reporting to be using 31GB total when they only have 8GB physical and are not using any swap. Here's an output from one of our machines:- vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 31768M 2112M 2 0 0 0 0 0 370 8139 3996 0 1 99 The raw output is:- vmstat -c 2 -w 1 -n 0 -H procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 32530228 2162524 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 32530228 2162524 2 0 0 0 0 0 286 8234 4347 0 1 99 Top shows:- last pid: 6665; load averages: 0.00, 0.00, 0.01 up 80+01:24:12 09:35:28 1893 processes:1 running, 1892 sleeping CPU: 0.0% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.7% idle Mem: 3754M Active, 84M Inact, 1976M Wired, 4K Cache, 2109M Free Swap: 4096M Total, 4096M Free sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) === Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1893) Virtual Memory: (Total: 1106403532K Active: 32540260K) Real Memory:(Total: 4563648K Active: 3921644K) Shared Virtual Memory: (Total: 19976K Active: 16396K) Shared Real Memory: (Total: 9040K Active: 8436K) Free Memory Pages: 2161740K As mentioned this machine has 8GB of ram and according to both top and swapinfo is using no swap at all From dmesg:- real memory = 8589934592 (8192 MB) avail memory = 823536 (7873 MB) swapinfo Device 1K-blocks UsedAvail Capacity /dev/gptid/09f211f7-39ce-11e0-8 41943040 4194304 0% uname -a FreeBSD test 8.2-RELEASE FreeBSD 8.2-RELEASE #2: Thu Mar 24 17:28:55 UTC 2011 root@test:/usr/obj/usr/src/sys/MULTIPLAY amd64 sysctl hw.pagesize hw.pagesize: 4096 It looks like it may be out by a factor of 4, possibly due to the fact the its a 4k page size not 1k as indicated by the vmstat man page:- memory Information about the usage of virtual and real memory. Virtual pages (reported in units of 1024 bytes) are considered active if they belong to processes which are running or have run in the last 20 seconds. avm active virtual pages fre size of the free list Totalling up RSS from ps axo rss gives a total in the region of that if the vm stats are out by a factor of 4, in this case it should be: 8132557 which is 7.75GB a much more realistic value. Am I totally missing something or is there problem here? Regards Steve This goes along with the thoughts I had about 4 months ago tending to some zfs statistics as well top showing greater than 100% actual CPU usage. This is a big pet peave of mine. Its like saying you ate 134% of a bannanna when in all reallity it is impossible. You can never have more than 100% usage of anything and when seen is a clear notice that some math is considerably incorrect leading to other such miscalculations to be performed. Things like the above already have checks in place that ensure no boundries are being crossed/overflowed or underrun but it surely makes processing results building future products a bitch. One instance is the calculation of threads for example firefox can be seen using upto or more 338% of the CPU. Thats impossible its like saying anyones CPU grew by 400%. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
