Re: Memory problems
On Wed, Jun 2, 2010 at 8:46 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Wed, Jun 02, 2010 at 07:47:06PM +0200, Jonatan Soto wrote: I fact I am setting this VM's for production and staging purposes. ESXi allows to create resources pools so it's possible to prioritize the usage of resources and that fits my needs very well. So I believe you noticed that is a very large amount of RAM (I have only 4 physical GB) for too less daemons and you are right, but with the advantages I explained before allows me to play fine with the memory I've defined at present. In near future I will install more software on each server, like databases, JVM's, etc. so the system will be fully loaded for sure. That's why I've got alarmed when I saw the memory consumptions for each server. I am totally conscious that I should upgrade RAM before I publish the server. Since you are running ESXi, it is actually quite possible for it to tell the guest (through the balloon driver) to give back some memory, which will show in the guest as used memory. Sorry Lennart, I don't get your point. Perhaps is because I am using the free version of ESXi and this option is not available. Check the vmware memory statistics and see if it has done that to the guests. I checked it out yesterday and I found a more 'realistic' memory usage than the shown with top or free. I would like to post links containing the exported images of the charts but I don't know why some fields (the important ones) are not shown. I must go to the traditional way, printScreen + gimp...I also can post memory usage data exported in excel format if it is the interest of somebody. I will do it ASAP. Well, I'm actually a bit confused here. I don't know in what direction should I go, I mean if I should investigate further and try to find an explanation for this or reinstall everything using a backported kernel or continuing the installation of the services and let's see what happens. Note that I am running out of time that's why I tried the official stable release because I thougth it will be pretty straightforward. For now I am thinking to reboot every server, except Server4 which is running correctly (at least the memory usage showed seems reasonable to me), and then observe how they behave. Thanks a lot for your help. -- Len Sorensen
Re: Memory problems
On Thu, Jun 03, 2010 at 10:27:53AM +0200, Jonatan Soto wrote: I checked it out yesterday and I found a more 'realistic' memory usage than the shown with top or free. I would like to post links containing the exported images of the charts but I don't know why some fields (the important ones) are not shown. I must go to the traditional way, printScreen + gimp...I also can post memory usage data exported in excel format if it is the interest of somebody. I will do it ASAP. Well, I'm actually a bit confused here. I don't know in what direction should I go, I mean if I should investigate further and try to find an explanation for this or reinstall everything using a backported kernel or continuing the installation of the services and let's see what happens. Note that I am running out of time that's why I tried the official stable release because I thougth it will be pretty straightforward. For now I am thinking to reboot every server, except Server4 which is running correctly (at least the memory usage showed seems reasonable to me), and then observe how they behave. Thanks a lot for your help. So any idea what is using the memory? By the way does your guest have the vmware tools installed? If you do, then you probably have /proc/vmmemctl. Check what that file says if it exists. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100603142249.gt17...@caffeine.csclub.uwaterloo.ca
Re: Memory problems
Hi there, On Thu, Jun 3, 2010 at 4:22 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Thu, Jun 03, 2010 at 10:27:53AM +0200, Jonatan Soto wrote: I checked it out yesterday and I found a more 'realistic' memory usage than the shown with top or free. I would like to post links containing the exported images of the charts but I don't know why some fields (the important ones) are not shown. I must go to the traditional way, printScreen + gimp...I also can post memory usage data exported in excel format if it is the interest of somebody. I will do it ASAP. Well, I'm actually a bit confused here. I don't know in what direction should I go, I mean if I should investigate further and try to find an explanation for this or reinstall everything using a backported kernel or continuing the installation of the services and let's see what happens. Note that I am running out of time that's why I tried the official stable release because I thougth it will be pretty straightforward. For now I am thinking to reboot every server, except Server4 which is running correctly (at least the memory usage showed seems reasonable to me), and then observe how they behave. Thanks a lot for your help. So any idea what is using the memory? No idea at all By the way does your guest have the vmware tools installed? Yes they do. But good to mention this because I don't remember if a restarted the servers since I installed vmware-tools Besides that I prefer to wait a little bit more in order to check how behaves Server4 (the only one that I've rebooted recently) If you do, then you probably have /proc/vmmemctl. Check what that file says if it exists. Here it is (eg: Server1): target: 503386 pages current: 503386 pages rateNoSleepAlloc: 16384 pages/sec rateSleepAlloc: 2048 pages/sec rateFree: 16384 pages/sec timer: 1135686 start: 1 ( 0 failed) guestType: 1 ( 0 failed) lock:2518801 ( 54 failed) unlock: 2015361 ( 0 failed) target: 1135686 ( 0 failed) primNoSleepAlloc:2518801 ( 0 failed) primCanSleepAlloc: 0 ( 0 failed) primFree:2015361 errAlloc: 51 errFree: 51 Is primFree the total amount of free memory available? -- Len Sorensen Thanks Len, I really appreciate your help!
Re: Memory problems
On Thu, Jun 03, 2010 at 04:56:21PM +0200, Jonatan Soto wrote: Here it is (eg: Server1): target: 503386 pages current: 503386 pages So that looks like the vmware balloon driver has aquired 503386 pages (at 4KB each) which would be your 2GB missing memory. So because your host is low on ram it has asked your guest to give back 2GB. So if you want your guests to not have their ram borrowed back to the host, add more ram to the host, or make sure the guests run identical code so that vmware can merge more identical pages (not very likely to be possible). rateNoSleepAlloc: 16384 pages/sec rateSleepAlloc: 2048 pages/sec rateFree: 16384 pages/sec timer: 1135686 start: 1 ( 0 failed) guestType: 1 ( 0 failed) lock:2518801 ( 54 failed) unlock: 2015361 ( 0 failed) target: 1135686 ( 0 failed) primNoSleepAlloc:2518801 ( 0 failed) primCanSleepAlloc: 0 ( 0 failed) primFree:2015361 errAlloc: 51 errFree: 51 Is primFree the total amount of free memory available? No idea what units those are in. So the vmware balloon driver is in fact the thing that made your ram disppear. Since it is a kernel driver you of course won't see a process using the ram, it will simply be marked in use. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100603150628.gu17...@caffeine.csclub.uwaterloo.ca
Memory problems
Hi list, I'm facing a problem with lenny regarding to memory usage. I have 4 VM lenny based on top of a VMWare ESXi. The system is running for a few days and top command shows a very high amount of memory consumption for each server. I have a little knowledge of how linux (kernel 2.6) manages memory. A nice resource I found is this: http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm So, I understand that cached memory may be free if some application requires it but I don't understand why lenny is consuming 2GB of physical memory. It's worth to mention that all the systems are running with only the standard package installed and few additional daemons for each server. I post what top command shows in order to provide better clues of what's going on: Server1: top - 18:47:12 up 12 days, 3:57, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 53 total, 2 running, 51 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2901684k used, 196080k free, 156460k buffers Swap: 578300k total,0k used, 578300k free, 592736k cached additional daemons - apache2, bind9, sshd Server2: top - 18:48:30 up 12 days, 2:29, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 55 total, 1 running, 54 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2412008k used, 685756k free, 145204k buffers Swap: 915664k total,0k used, 915664k free, 155112k cached additional daemons - apache2, sshd Server3: top - 18:52:10 up 12 days, 2:32, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 72 total, 1 running, 71 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2263200k used, 834564k free,49152k buffers Swap: 2928632k total,0k used, 2928632k free, 107700k cached additional daemons - lvm, sshd Server4: top - 16:50:19 up 1 day, 6:42, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 272300k used, 2825464k free, 126164k buffers Swap: 1341388k total,0k used, 1341388k free,53196k cached additional daemons - bind9, sldap, samba, sshd Note that I've recently rebooted Server4 and it has lower memory consumption rather than the others and it is running more daemons. May be this issue is a misconfiguration of my servers or a memory leak? Should I tweak something in order to improve memory management? Any help would be much appreciated. PD: Apache2 is installed using default configuration of the Apache2 official Debian release.
Re: Memory problems
On Wed, Jun 02, 2010 at 05:06:30PM +0200, Jonatan Soto wrote: Hi list, I'm facing a problem with lenny regarding to memory usage. I have 4 VM lenny based on top of a VMWare ESXi. The system is running for a few days and top command shows a very high amount of memory consumption for each server. I have a little knowledge of how linux (kernel 2.6) manages memory. A nice resource I found is this: http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm So, I understand that cached memory may be free if some application requires it but I don't understand why lenny is consuming 2GB of physical memory. It's worth to mention that all the systems are running with only the standard package installed and few additional daemons for each server. I post what top command shows in order to provide better clues of what's going on: Server1: top - 18:47:12 up 12 days, 3:57, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 53 total, 2 running, 51 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2901684k used, 196080k free, 156460k buffers Swap: 578300k total,0k used, 578300k free, 592736k cached additional daemons - apache2, bind9, sshd Server2: top - 18:48:30 up 12 days, 2:29, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 55 total, 1 running, 54 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2412008k used, 685756k free, 145204k buffers Swap: 915664k total,0k used, 915664k free, 155112k cached additional daemons - apache2, sshd Server3: top - 18:52:10 up 12 days, 2:32, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 72 total, 1 running, 71 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2263200k used, 834564k free,49152k buffers Swap: 2928632k total,0k used, 2928632k free, 107700k cached additional daemons - lvm, sshd Server4: top - 16:50:19 up 1 day, 6:42, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 272300k used, 2825464k free, 126164k buffers Swap: 1341388k total,0k used, 1341388k free,53196k cached additional daemons - bind9, sldap, samba, sshd Note that I've recently rebooted Server4 and it has lower memory consumption rather than the others and it is running more daemons. May be this issue is a misconfiguration of my servers or a memory leak? Should I tweak something in order to improve memory management? Any help would be much appreciated. PD: Apache2 is installed using default configuration of the Apache2 official Debian release. Well I would run top, hit 'M' to sort by memory usage, and see which processes are using the ram. For example: top - 11:22:12 up 12 days, 23:25, 11 users, load average: 1.52, 1.51, 1.67 Tasks: 257 total, 1 running, 256 sleeping, 0 stopped, 0 zombie Cpu(s): 3.5%us, 0.3%sy, 0.0%ni, 86.4%id, 9.8%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16473836k total, 16311892k used, 161944k free, 5624832k buffers Swap: 16777208k total, 168k used, 16777040k free, 6619416k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 8998 bacula20 0 193m 123m 1792 S 28 0.8 1611:17 bacula-dir 21000 snmp 20 0 41976 34m 980 S0 0.2 2:30.73 snmpd 20123 bind 20 0 117m 27m 860 S0 0.2 0:00.22 named 20768 mysql 20 0 124m 12m 1060 S0 0.1 1:26.30 mysqld 23350 root 20 0 34336 9988 7404 S0 0.1 0:00.76 bat 21302 root 20 0 23564 6772 2312 S0 0.0 0:02.66 apache2
Re: Memory problems
On 06/02/2010 09:06 AM, Jonatan Soto wrote: Hi list, I'm facing a problem with lenny regarding to memory usage. I have 4 VM lenny based on top of a VMWare ESXi. The system is running for a few days and top command shows a very high amount of memory consumption for each server. I have a little knowledge of how linux (kernel 2.6) manages memory. A nice resource I found is this: http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm So, I understand that cached memory may be free if some application requires it but I don't understand why lenny is consuming 2GB of physical memory. It's worth to mention that all the systems are running with only the standard package installed and few additional daemons for each server. I post what top command shows in order to provide better clues of what's going on: Server1: top - 18:47:12 up 12 days, 3:57, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 53 total, 2 running, 51 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2901684k used, 196080k free, 156460k buffers Swap: 578300k total,0k used, 578300k free, 592736k cached additional daemons - apache2, bind9, sshd Server2: top - 18:48:30 up 12 days, 2:29, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 55 total, 1 running, 54 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2412008k used, 685756k free, 145204k buffers Swap: 915664k total,0k used, 915664k free, 155112k cached additional daemons - apache2, sshd Server3: top - 18:52:10 up 12 days, 2:32, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 72 total, 1 running, 71 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2263200k used, 834564k free,49152k buffers Swap: 2928632k total,0k used, 2928632k free, 107700k cached additional daemons - lvm, sshd Server4: top - 16:50:19 up 1 day, 6:42, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 272300k used, 2825464k free, 126164k buffers Swap: 1341388k total,0k used, 1341388k free,53196k cached additional daemons - bind9, sldap, samba, sshd I like to think of the drive cache as the computer saying We'll hold onto this information you read from the hard drive in memory just in case you want to use again soon. The memory wouldn't be doing anything if empty anyways. Essentially I count cache as free memory because the OS will drop it in a heartbeat if an application needs it. Also, it is reasonable that the cache increases as the uptime does because the computer reads more from the hard drive that can be cached as time goes on. Brian Oborn Note that I've recently rebooted Server4 and it has lower memory consumption rather than the others and it is running more daemons. May be this issue is a misconfiguration of my servers or a memory leak? Should I tweak something in order to improve memory management? Any help would be much appreciated. PD: Apache2 is installed using default configuration of the Apache2 official Debian release. -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4c0676f7.6070...@iac.isu.edu
Re: Memory problems
Jonatan, In my opinion your virtual machines have a very large memory (4G) if you only want to run the basic and a few daemons cut it and try again I belive the buffers are used because you have memory available to spare... chp...@refugio:~$ free total used free sharedbuffers cached Mem: 20610762019148 41928 0 123156 586100 -/+ buffers/cache:1309892 751184 Swap: 1959800 01959800 I'm running X, firefox, chrome, kde and some daemons... On Wed, Jun 2, 2010 at 10:06 AM, Jonatan Soto seniledemen...@gmail.com wrote: Hi list, I'm facing a problem with lenny regarding to memory usage. I have 4 VM lenny based on top of a VMWare ESXi. The system is running for a few days and top command shows a very high amount of memory consumption for each server. I have a little knowledge of how linux (kernel 2.6) manages memory. A nice resource I found is this: http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm So, I understand that cached memory may be free if some application requires it but I don't understand why lenny is consuming 2GB of physical memory. It's worth to mention that all the systems are running with only the standard package installed and few additional daemons for each server. I post what top command shows in order to provide better clues of what's going on: Server1: top - 18:47:12 up 12 days, 3:57, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 53 total, 2 running, 51 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2901684k used, 196080k free, 156460k buffers Swap: 578300k total, 0k used, 578300k free, 592736k cached additional daemons - apache2, bind9, sshd Server2: top - 18:48:30 up 12 days, 2:29, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 55 total, 1 running, 54 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2412008k used, 685756k free, 145204k buffers Swap: 915664k total, 0k used, 915664k free, 155112k cached additional daemons - apache2, sshd Server3: top - 18:52:10 up 12 days, 2:32, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 72 total, 1 running, 71 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2263200k used, 834564k free, 49152k buffers Swap: 2928632k total, 0k used, 2928632k free, 107700k cached additional daemons - lvm, sshd Server4: top - 16:50:19 up 1 day, 6:42, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 272300k used, 2825464k free, 126164k buffers Swap: 1341388k total, 0k used, 1341388k free, 53196k cached additional daemons - bind9, sldap, samba, sshd Note that I've recently rebooted Server4 and it has lower memory consumption rather than the others and it is running more daemons. May be this issue is a misconfiguration of my servers or a memory leak? Should I tweak something in order to improve memory management? Any help would be much appreciated. PD: Apache2 is installed using default configuration of the Apache2 official Debian release. -- Perhaps the depth of love can be calibrated by the number of different selves that are actively involved in a given relationship. Carl Sagan (Contact) Absolute certainty is a privilege of uneducated minds-and fanatics. It is, for scientific folk, an unattainable ideal. Cassius J. Keyser Jaime Ochoa Malagón Arquitecto de Soluciones Cel: +52 (55) 1021 0774 -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/aanlktimqoamr_knb01x03xsr3oufxas_bl8na6put...@mail.gmail.com
Re: Memory problems
Thank you very much for your quick responses. On Wed, Jun 2, 2010 at 5:26 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Wed, Jun 02, 2010 at 05:06:30PM +0200, Jonatan Soto wrote: Hi list, I'm facing a problem with lenny regarding to memory usage. I have 4 VM lenny based on top of a VMWare ESXi. The system is running for a few days and top command shows a very high amount of memory consumption for each server. I have a little knowledge of how linux (kernel 2.6) manages memory. A nice resource I found is this: http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm So, I understand that cached memory may be free if some application requires it but I don't understand why lenny is consuming 2GB of physical memory. It's worth to mention that all the systems are running with only the standard package installed and few additional daemons for each server. I post what top command shows in order to provide better clues of what's going on: Server1: top - 18:47:12 up 12 days, 3:57, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 53 total, 2 running, 51 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2901684k used, 196080k free, 156460k buffers Swap: 578300k total,0k used, 578300k free, 592736k cached additional daemons - apache2, bind9, sshd Server2: top - 18:48:30 up 12 days, 2:29, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 55 total, 1 running, 54 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2412008k used, 685756k free, 145204k buffers Swap: 915664k total,0k used, 915664k free, 155112k cached additional daemons - apache2, sshd Server3: top - 18:52:10 up 12 days, 2:32, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 72 total, 1 running, 71 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2263200k used, 834564k free,49152k buffers Swap: 2928632k total,0k used, 2928632k free, 107700k cached additional daemons - lvm, sshd Server4: top - 16:50:19 up 1 day, 6:42, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 272300k used, 2825464k free, 126164k buffers Swap: 1341388k total,0k used, 1341388k free,53196k cached additional daemons - bind9, sldap, samba, sshd Note that I've recently rebooted Server4 and it has lower memory consumption rather than the others and it is running more daemons. May be this issue is a misconfiguration of my servers or a memory leak? Should I tweak something in order to improve memory management? Any help would be much appreciated. PD: Apache2 is installed using default configuration of the Apache2 official Debian release. Well I would run top, hit 'M' to sort by memory usage, and see which processes are using the ram. Well, that's what I did and that's why I decided to consult here because I thought it might be a problem since the memory usage shown in top doesn't match the amount of memory consumption of each process. The following corresponds to Server3, I understand that these processes cannot consume 2GB of physical memory 12027 root 20 0 89732 3764 2988 R0 0.1 0:00.30 sshd 22502 root 20 0 163m 2752 1560 S0 0.1 0:42.68 nscd 3608 root 20 0 77784 2164 1668 S0 0.1 0:00.66 login 3009 root 20 0 119m 2080 1004 S0 0.1 0:20.07 rsyslogd 28878 root 20 0 10576 1768 1312 S0 0.1 0:00.36 bash 12030 root 20 0 10528 1676 1264 S0 0.1 0:00.00 bash 3280 root 20 0 23896 1436 1132 S0 0.0 7:36.22 vmware-guestd 28889 root 20 0 42808 1144 672 S0 0.0 0:01.12 sshd 1262 root 16 -4 16912 1124 484 S0 0.0 0:00.04 udevd 12034 root 20 0 10624 1100 848 R0 0.0 0:04.90 top 3553 Debian-e 20 0 42716 1016 612 S0 0.0 0:07.40 exim4 3591 root 20 0 19804 844 652 S0 0.0 0:05.16 cron 2793 statd 20 0 10136 760 632 S0 0.0 0:00.00 rpc.statd 1 root 20 0 10312 756 620 S0 0.0 0:11.76 init 3020 root 20 0 3796 600 476 S0 0.0 0:00.00 acpid 3616 root 20 0 3796 584 484 S0 0.0 0:00.00 getty 3612 root 20 0 3796 580 484 S0 0.0 0:00.00 getty 3617 root 20 0 3796 580 484 S0 0.0 0:00.00 getty 3609 root 20 0 3796 576 484 S0 0.0 0:00.00 getty 3613 root 20 0 3796 576 484 S0 0.0 0:00.00 getty 2782 daemon20 0 8020 536 416 S0 0.0 0:00.00 portmap 3571 daemon20 0 16356 444 296 S0 0.0 0:02.40 atd For example: top - 11:22:12 up 12 days, 23:25, 11 users, load average: 1.52, 1.51, 1.67 Tasks: 257 total, 1 running, 256 sleeping, 0 stopped, 0 zombie Cpu(s): 3.5%us, 0.3%sy, 0.0%ni, 86.4%id,
Re: Memory problems
On 06/02/2010 10:06 AM, Jonatan Soto wrote: Hi list, I'm facing a problem with lenny regarding to memory usage. I have 4 VM lenny based on top of a VMWare ESXi. The system is running for a few days and top command shows a very high amount of memory consumption for each server. I have a little knowledge of how linux (kernel 2.6) manages memory. A nice resource I found is this: http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm So, I understand that cached memory may be free if some application requires it but I don't understand why lenny is consuming 2GB of physical memory. It's worth to mention that all the systems are running with only the standard package installed and few additional daemons for each server. I post what top command shows in order to provide better clues of what's going on: [snip] Note that I've recently rebooted Server4 and it has lower memory consumption rather than the others and it is running more daemons. May be this issue is a misconfiguration of my servers or a memory leak? Should I tweak something in order to improve memory management? Any help would be much appreciated. PD: Apache2 is installed using default configuration of the Apache2 official Debian release. Kernel Sampage Merging (in 2.6.33) might help reduce memory consumption. -- Dissent is patriotic, remember? -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4c068b30.9040...@cox.net
Re: Memory problems
On Wed, Jun 02, 2010 at 06:43:28PM +0200, Jonatan Soto wrote: Well, that's what I did and that's why I decided to consult here because I thought it might be a problem since the memory usage shown in top doesn't match the amount of memory consumption of each process. The following corresponds to Server3, I understand that these processes cannot consume 2GB of physical memory 12027 root 20 0 89732 3764 2988 R0 0.1 0:00.30 sshd 22502 root 20 0 163m 2752 1560 S0 0.1 0:42.68 nscd 3608 root 20 0 77784 2164 1668 S0 0.1 0:00.66 login 3009 root 20 0 119m 2080 1004 S0 0.1 0:20.07 rsyslogd 28878 root 20 0 10576 1768 1312 S0 0.1 0:00.36 bash 12030 root 20 0 10528 1676 1264 S0 0.1 0:00.00 bash 3280 root 20 0 23896 1436 1132 S0 0.0 7:36.22 vmware-guestd 28889 root 20 0 42808 1144 672 S0 0.0 0:01.12 sshd 1262 root 16 -4 16912 1124 484 S0 0.0 0:00.04 udevd 12034 root 20 0 10624 1100 848 R0 0.0 0:04.90 top 3553 Debian-e 20 0 42716 1016 612 S0 0.0 0:07.40 exim4 3591 root 20 0 19804 844 652 S0 0.0 0:05.16 cron 2793 statd 20 0 10136 760 632 S0 0.0 0:00.00 rpc.statd 1 root 20 0 10312 756 620 S0 0.0 0:11.76 init 3020 root 20 0 3796 600 476 S0 0.0 0:00.00 acpid 3616 root 20 0 3796 584 484 S0 0.0 0:00.00 getty 3612 root 20 0 3796 580 484 S0 0.0 0:00.00 getty 3617 root 20 0 3796 580 484 S0 0.0 0:00.00 getty 3609 root 20 0 3796 576 484 S0 0.0 0:00.00 getty 3613 root 20 0 3796 576 484 S0 0.0 0:00.00 getty 2782 daemon20 0 8020 536 416 S0 0.0 0:00.00 portmap 3571 daemon20 0 16356 444 296 S0 0.0 0:02.40 atd Certainly nothing there appears to be using much ram. So I understand there's nothing to worry. I should look at field RES so the sum of all the processes is the correct physical memory usage . Is that right? That's my understanding at least. Another way to see how much memory is consuming the process could be ps -ef, but it's a percentage. Taking this value, probably matches the REST value of top command. Please correct me if I'm wrong. Well if your machine in fact is not using the memory for cache or buffers and none of the processes have it, then I would say one of your drivers or some other kernel part is leaking memory. That would be bad. I wonder it that is the case. What is the output of 'free'? In my case I get: total used free sharedbuffers cached Mem: 16473836 16364008 109828 049435528269128 -/+ buffers/cache:3151328 13322508 Swap: 16777208168 16777040 So out of 16GB, 8GB is cache, 5GB buffers, and 3GB actually used. No problem then. If everything is in fact being used, then that seems wrong, and I can only think there is a kernel bug leaking memory somewhere then. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100602165206.gn17...@caffeine.csclub.uwaterloo.ca
Re: Memory problems
On Wed, Jun 02, 2010 at 11:47:44AM -0500, Ron Johnson wrote: Kernel Sampage Merging (in 2.6.33) might help reduce memory consumption. 2.6.32 has KSM. I am using it now (using 2.6.32 backport kernel). Works very well. /sys/kernel/mm/ksm# grep . * full_scans:2415 max_kernel_pages:0 pages_shared:347205 pages_sharing:2802858 pages_to_scan:100 pages_unshared:2392685 pages_volatile:479345 run:1 sleep_millisecs:20 Seems like a decent ratio of shared pages. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100602165402.go17...@caffeine.csclub.uwaterloo.ca
Re: Memory problems
On Wed, Jun 2, 2010 at 6:52 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Wed, Jun 02, 2010 at 06:43:28PM +0200, Jonatan Soto wrote: Well, that's what I did and that's why I decided to consult here because I thought it might be a problem since the memory usage shown in top doesn't match the amount of memory consumption of each process. The following corresponds to Server3, I understand that these processes cannot consume 2GB of physical memory 12027 root 20 0 89732 3764 2988 R0 0.1 0:00.30 sshd 22502 root 20 0 163m 2752 1560 S0 0.1 0:42.68 nscd 3608 root 20 0 77784 2164 1668 S0 0.1 0:00.66 login 3009 root 20 0 119m 2080 1004 S0 0.1 0:20.07 rsyslogd 28878 root 20 0 10576 1768 1312 S0 0.1 0:00.36 bash 12030 root 20 0 10528 1676 1264 S0 0.1 0:00.00 bash 3280 root 20 0 23896 1436 1132 S0 0.0 7:36.22 vmware-guestd 28889 root 20 0 42808 1144 672 S0 0.0 0:01.12 sshd 1262 root 16 -4 16912 1124 484 S0 0.0 0:00.04 udevd 12034 root 20 0 10624 1100 848 R0 0.0 0:04.90 top 3553 Debian-e 20 0 42716 1016 612 S0 0.0 0:07.40 exim4 3591 root 20 0 19804 844 652 S0 0.0 0:05.16 cron 2793 statd 20 0 10136 760 632 S0 0.0 0:00.00 rpc.statd 1 root 20 0 10312 756 620 S0 0.0 0:11.76 init 3020 root 20 0 3796 600 476 S0 0.0 0:00.00 acpid 3616 root 20 0 3796 584 484 S0 0.0 0:00.00 getty 3612 root 20 0 3796 580 484 S0 0.0 0:00.00 getty 3617 root 20 0 3796 580 484 S0 0.0 0:00.00 getty 3609 root 20 0 3796 576 484 S0 0.0 0:00.00 getty 3613 root 20 0 3796 576 484 S0 0.0 0:00.00 getty 2782 daemon20 0 8020 536 416 S0 0.0 0:00.00 portmap 3571 daemon20 0 16356 444 296 S0 0.0 0:02.40 atd Certainly nothing there appears to be using much ram. So I understand there's nothing to worry. I should look at field RES so the sum of all the processes is the correct physical memory usage . Is that right? That's my understanding at least. Another way to see how much memory is consuming the process could be ps -ef, but it's a percentage. Taking this value, probably matches the REST value of top command. Please correct me if I'm wrong. Well if your machine in fact is not using the memory for cache or buffers and none of the processes have it, then I would say one of your drivers or some other kernel part is leaking memory. That would be bad. I wonder it that is the case. What is the output of 'free'? Sorry, I should posted it before. This is also for server3, but 1 and 2 have the same problem. Server4 is not a valid example because I rebooted a couple of days ago and it seems to behave great for now. totalused freeshared buffers cached Mem: 30977642263292 834472 0 49272 107712 -/+ buffers/cache:2106308 991456 Swap: 2928632 02928632 In this case, out of 3GB, 100MB is cache and 50MB buffers. Plus 1GB free memory it means that the system is taking 2GB for I don't know what... In my case I get: total used free sharedbuffers cached Mem: 16473836 16364008 109828 049435528269128 -/+ buffers/cache:3151328 13322508 Swap: 16777208168 16777040 So out of 16GB, 8GB is cache, 5GB buffers, and 3GB actually used. No problem then. If everything is in fact being used, then that seems wrong, and I can only think there is a kernel bug leaking memory somewhere then. -- Len Sorensen Thank you for your time!
Re: Memory problems
On Wed, Jun 2, 2010 at 6:54 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Wed, Jun 02, 2010 at 11:47:44AM -0500, Ron Johnson wrote: Kernel Sampage Merging (in 2.6.33) might help reduce memory consumption. 2.6.32 has KSM. I am using it now (using 2.6.32 backport kernel). Works very well. I am using 2.6.26-2 which is the default with the latest stable release, isn't? /sys/kernel/mm/ksm# grep . * full_scans:2415 max_kernel_pages:0 pages_shared:347205 pages_sharing:2802858 pages_to_scan:100 pages_unshared:2392685 pages_volatile:479345 run:1 sleep_millisecs:20 Seems like a decent ratio of shared pages. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100602165402.go17...@caffeine.csclub.uwaterloo.ca
Re: Memory problems
Hi Jaime, Thanks for your help. 2010/6/2 Jaime Ochoa Malagón chp...@gmail.com Jonatan, In my opinion your virtual machines have a very large memory (4G) if you only want to run the basic and a few daemons cut it and try again I belive the buffers are used because you have memory available to spare... I fact I am setting this VM's for production and staging purposes. ESXi allows to create resources pools so it's possible to prioritize the usage of resources and that fits my needs very well. So I believe you noticed that is a very large amount of RAM (I have only 4 physical GB) for too less daemons and you are right, but with the advantages I explained before allows me to play fine with the memory I've defined at present. In near future I will install more software on each server, like databases, JVM's, etc. so the system will be fully loaded for sure. That's why I've got alarmed when I saw the memory consumptions for each server. I am totally conscious that I should upgrade RAM before I publish the server. chp...@refugio:~$ free total used free sharedbuffers cached Mem: 20610762019148 41928 0 123156 586100 -/+ buffers/cache:1309892 751184 Swap: 1959800 01959800 I'm running X, firefox, chrome, kde and some daemons... On Wed, Jun 2, 2010 at 10:06 AM, Jonatan Soto seniledemen...@gmail.com wrote: Hi list, I'm facing a problem with lenny regarding to memory usage. I have 4 VM lenny based on top of a VMWare ESXi. The system is running for a few days and top command shows a very high amount of memory consumption for each server. I have a little knowledge of how linux (kernel 2.6) manages memory. A nice resource I found is this: http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm So, I understand that cached memory may be free if some application requires it but I don't understand why lenny is consuming 2GB of physical memory. It's worth to mention that all the systems are running with only the standard package installed and few additional daemons for each server. I post what top command shows in order to provide better clues of what's going on: Server1: top - 18:47:12 up 12 days, 3:57, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 53 total, 2 running, 51 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2901684k used, 196080k free, 156460k buffers Swap: 578300k total,0k used, 578300k free, 592736k cached additional daemons - apache2, bind9, sshd Server2: top - 18:48:30 up 12 days, 2:29, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 55 total, 1 running, 54 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2412008k used, 685756k free, 145204k buffers Swap: 915664k total,0k used, 915664k free, 155112k cached additional daemons - apache2, sshd Server3: top - 18:52:10 up 12 days, 2:32, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 72 total, 1 running, 71 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 2263200k used, 834564k free,49152k buffers Swap: 2928632k total,0k used, 2928632k free, 107700k cached additional daemons - lvm, sshd Server4: top - 16:50:19 up 1 day, 6:42, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 58 total, 1 running, 57 sleeping, 0 stopped, 0 zombie Mem: 3097764k total, 272300k used, 2825464k free, 126164k buffers Swap: 1341388k total,0k used, 1341388k free,53196k cached additional daemons - bind9, sldap, samba, sshd Note that I've recently rebooted Server4 and it has lower memory consumption rather than the others and it is running more daemons. May be this issue is a misconfiguration of my servers or a memory leak? Should I tweak something in order to improve memory management? Any help would be much appreciated. PD: Apache2 is installed using default configuration of the Apache2 official Debian release. -- Perhaps the depth of love can be calibrated by the number of different selves that are actively involved in a given relationship. Carl Sagan (Contact) Absolute certainty is a privilege of uneducated minds-and fanatics. It is, for scientific folk, an unattainable ideal. Cassius J. Keyser Jaime Ochoa Malagón Arquitecto de Soluciones Cel: +52 (55) 1021 0774
Re: Memory problems
On Wed, Jun 02, 2010 at 07:25:34PM +0200, Jonatan Soto wrote: Sorry, I should posted it before. This is also for server3, but 1 and 2 have the same problem. Server4 is not a valid example because I rebooted a couple of days ago and it seems to behave great for now. totalused freeshared buffers cached Mem: 30977642263292 834472 0 49272 107712 -/+ buffers/cache:2106308 991456 Swap: 2928632 02928632 In this case, out of 3GB, 100MB is cache and 50MB buffers. Plus 1GB free memory it means that the system is taking 2GB for I don't know what... Well if it doesn't show in top for any process, then it must be something in the kernel, or it got leaked and lost. Are you using any weird kernel (like a vm specific xen kernel or something) that supports the baloon driver to allow the host to take back memory from the guest if needed, or are these running straight on the hardware? -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100602183719.gp17...@caffeine.csclub.uwaterloo.ca
Re: Memory problems
On Wed, Jun 02, 2010 at 07:28:35PM +0200, Jonatan Soto wrote: On Wed, Jun 2, 2010 at 6:54 PM, Lennart Sorensen lsore...@csclub.uwaterloo.ca wrote: On Wed, Jun 02, 2010 at 11:47:44AM -0500, Ron Johnson wrote: Kernel Sampage Merging (in 2.6.33) might help reduce memory consumption. 2.6.32 has KSM. I am using it now (using 2.6.32 backport kernel). Works very well. I am using 2.6.26-2 which is the default with the latest stable release, isn't? Yes. I am running stable but with the 2.6.32 kernel from backports.org. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100602183759.gq17...@caffeine.csclub.uwaterloo.ca
Re: Memory problems
On Wed, Jun 02, 2010 at 07:47:06PM +0200, Jonatan Soto wrote: I fact I am setting this VM's for production and staging purposes. ESXi allows to create resources pools so it's possible to prioritize the usage of resources and that fits my needs very well. So I believe you noticed that is a very large amount of RAM (I have only 4 physical GB) for too less daemons and you are right, but with the advantages I explained before allows me to play fine with the memory I've defined at present. In near future I will install more software on each server, like databases, JVM's, etc. so the system will be fully loaded for sure. That's why I've got alarmed when I saw the memory consumptions for each server. I am totally conscious that I should upgrade RAM before I publish the server. Since you are running ESXi, it is actually quite possible for it to tell the guest (through the balloon driver) to give back some memory, which will show in the guest as used memory. Check the vmware memory statistics and see if it has done that to the guests. -- Len Sorensen -- To UNSUBSCRIBE, email to debian-amd64-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20100602184651.gr17...@caffeine.csclub.uwaterloo.ca
net join memory problems
When I try to run net join to get samba to talk to the domain controller I get a bunch error messages about being out of memory and killing process. The same samba configuration works fine on other machines with the regular version of Sarge. Has anyone else come across this kind of problem with Samba and AMD64 or has anyone been able to successfully join a domain with AMD64? -- scribles [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
