Re: [gentoo-user] mysqld invoked oom-killer
On Tuesday 26 July 2011 04:54:27 Grant wrote: Is this because I've eselect'ed icedtea6-bin instead of sun-jdk-1.6? BTW, can anyone tell me why I'm using icedtea6-bin instead of icedtea? I don't know. On this box the only java-vm installed is icedtea6-bin. -- Rgds Peter Linux Counter number 5290
Re: [gentoo-user] mysqld invoked oom-killer
Sounds like a case for a swap partition that can be activated when you need it for big emerges. I hit the same thing with firefox-5 oddly enough. I have one smallish swap partition at PRI=10 and a bigger one at PRI=1. As for OOo, long ago I figured the pain wasn't worth the gain so now I use the -bin packages. Or switch to Libre Office. Its compiler requirements seem lighter than OO.o. I tried libreoffice with the same result. Much more strangely, I enabled a 10GB swap file and got the same error. I thought it was a virtual memory problem at first but now that I look more closely it looks like a java problem: Error occurred during initialization of VM Could not reserve enough space for code cache JavaVM: JNI_CreateJavaVM called _exit, caught by abort_handler in javavm.cxx [Java framework] sunjavaplugin.soCan not create JavaVirtualMachine, abort handler was called. [Java framework] The JRE specified by the bootstrap variable UNO_JAVA_JFW_JREHOME or UNO_JAVA_JFW_ENV_JREHOME could not be recognized. Check the values and make sure that you use a plug-in library that can recognize that JRE. register component 'file:///var/tmp/portage/app-office/libreoffice-3.3.1/work/libreoffice-build-3.3.1.2/build/libreoffice-3.3.1.2/testtools/source/bridgetest/../../unxlngx6.pro/class/testComponent.jar' in registry '../../unxlngx6.pro/lib/uno_services.rdb' failed! error (CannotRegisterImplementationException): Could not create Java implementation loader dmake: Error code 1, while making '../../unxlngx6.pro/lib/uno_services.rdb' Is this because I've eselect'ed icedtea6-bin instead of sun-jdk-1.6? BTW, can anyone tell me why I'm using icedtea6-bin instead of icedtea? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
Grant wrote: ... If my main rig starts using swap a lot, I'm going to be very curious. I even used 8Gbs to put portages work directory on tmpfs. I still didn't use any swap. By the way, that doesn't seem to make the compiles any faster. o_O CPU bottleneck? - Grant I sort of doubt it. I have a AMD 4 core 3.2Ghz CPU. A 6 core is about all that beats it. I think they have a 3.3. and a 3.4 now but that's marginal really. It has always been said that reading and writing to the portage work directory would slow down compiles a bit. Well, I tested the theory and it seemed to have taken a few seconds longer with portage's work directory on tmpfs instead of a hard drive. I see the reason it should be faster and I think it should be faster but when I tried it, theory ran up against reality. We all know what happens when theory meets reality. Theory is small and weak but reality is large, strong and a great teacher to boot. Reality always wins even when it don't make sense. :/ Don't worry, I was as surprised as anyone. I found on the forum where someone else came to the same conclusions. I guess it caches stuff or just that drives are faster and smarter nowadays. It was neat to try it tho. ;-) This is the most ram I have ever had. May be more than inside my head. lol Dale :-) :-)
Re: [gentoo-user] mysqld invoked oom-killer
If my main rig starts using swap a lot, I'm going to be very curious. I even used 8Gbs to put portages work directory on tmpfs. I still didn't use any swap. By the way, that doesn't seem to make the compiles any faster. o_O CPU bottleneck? - Grant I sort of doubt it. I have a AMD 4 core 3.2Ghz CPU. A 6 core is about all that beats it. I think they have a 3.3. and a 3.4 now but that's marginal really. I just got a 3.7! :) It has always been said that reading and writing to the portage work directory would slow down compiles a bit. Well, I tested the theory and it seemed to have taken a few seconds longer with portage's work directory on tmpfs instead of a hard drive. I see the reason it should be faster and I think it should be faster but when I tried it, theory ran up against reality. We all know what happens when theory meets reality. Theory is small and weak but reality is large, strong and a great teacher to boot. Reality always wins even when it don't make sense. :/ Don't worry, I was as surprised as anyone. I found on the forum where someone else came to the same conclusions. I guess it caches stuff or just that drives are faster and smarter nowadays. Thanks for sharing your results Dale. Surprising stuff. - Grant
Re: [gentoo-user] mysqld invoked oom-killer
... Next I'd look at tuning your Mysql config. If you've never touched my.cnf, by default it's set to use 64MB IIRC. You may need to raise this to get better performance. key_buffer and innodb_buffer_pool_size are the only two I'd modify without knowing more. kashani I'm running InnoDB and I've changed both key_buffer and innodb_buffer_pool_size to 256MB. Does that sound about right? Does this mean I've allocated a total of 512MB VM to mysql? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
... If my main rig starts using swap a lot, I'm going to be very curious. I even used 8Gbs to put portages work directory on tmpfs. I still didn't use any swap. By the way, that doesn't seem to make the compiles any faster. o_O CPU bottleneck? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
... That all makes perfect sense. So the reason a swap larger than maybe 1GB is not usually implemented is because idle processes don't normally have more than a few hundred MB of pages in memory? That's not entirely true, either. For example, My laptop has 4GB of swap. Why? Well, because I use hibernate and hibernate works on the swap partition and I want to make sure that I have enough swap to write all my memory to swap (actually It's now compressed so actually I probablldon't really need that much). Wouldn't a sufficiently large swap (100GB for example) completely prevent out of memory conditions and the oom-killer? No. oom killer kicks in when your system is out of virtual memory. Consider this example: You have 4GB RAM You have 0 swap. Therefore you have a total of 4GB virtual memory. The second all your processes try to consume more than 4G of virtual memory, oom killer will kick in* Consider the next example You have 4GB RAM You have 100GB swap. Therefore you have a total of 104GB virtual memory The second all your processes try to consume more than 104GB of virtual memory, oom killer will kick in. Oom killer works on virtual memory (RAM + swap). So it doesn't matter how much RAM you have or how much swap you have, when the total virtual memory is consumed, oom killer is called. The secret is to not run out of virtual memory. There is no *easy* way not to run out of virtual memory. You either don't consume as much VM, or you provide more VM (either through RAM or swap). * This is not entirely true, the system also needs memory for the kernel, buffers, hardware drivers, and other things which simply cannot be paged out to disk, so the actual number will be less than the amount of VM. You have all been very patient with me and I truly appreciate it. Let's see if I've got it. When the system is in a healthy state, swap is used to store pages of idle processes (which should not typically amount to more than about 200MB) in order to free up as much RAM as possible for filesystem cache if nothing else. Linux will attempt to fill any unused RAM with filesystem cache. The system should be configured to assure it will not use an amount of memory greater than the amount of physical RAM, otherwise the system may run out of physical RAM. If this happens and the system has no swap, the system will initiate the oom-killer. If this happens and the system has swap, the system will thrash until it runs out of swap at which point it will initiate the oom-killer. Besides providing storage for idle process pages, when available physical RAM is depleted, swap gives the user some time to intervene before the oom-killer is initiated, although at that point the system will be in a state of thrashing. Too much swap will cause the system to thrash for a long time before the oom-killer is initiated which is not ideal. How did I do? Is there anything else I should keep in mind? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday 21 July 2011 21:44:51 Alan McKinnon wrote: Sounds like a case for a swap partition that can be activated when you need it for big emerges. I hit the same thing with firefox-5 oddly enough. I have one smallish swap partition at PRI=10 and a bigger one at PRI=1. As for OOo, long ago I figured the pain wasn't worth the gain so now I use the -bin packages. Or switch to Libre Office. Its compiler requirements seem lighter than OO.o. -- Rgds Peter Linux Counter number 5290
Re: [gentoo-user] mysqld invoked oom-killer
On Thu, 21 Jul 2011 22:16:41 -0400, Albert Hopkins wrote: Think of it this way: You have a house with an attic. Now the attic is not as efficient as say, the middle of your living room. You have a Christmas tree, but you only use that Christmas tree maybe once a year. Now it's much more efficient to keep that Christmas tree in the attic for 11 months of the year and use that reclaimed space in your living room for.. say a coffee table. Then, when you need that Christmas tree in December, you pull it out of the attic and maybe put the coffee table up in the attic for a month. And the winner of the Analogy of the Month award is... This has to be the best explanation of how swap is used I have ever read. -- Neil Bothwick All right, set phasers to deep fat fry! signature.asc Description: PGP signature
Re: [gentoo-user] mysqld invoked oom-killer
On Thu, Jul 21, 2011 at 8:29 PM, Grant emailgr...@gmail.com wrote: Its more how much i/o rather than the size. If you have a bunch of stuff swapped out, but it hardly ever needs to be swapped in, the impact will be low. Keep an eye on the use with vmstat; adam@rix ~ $ vmstat 5 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 56700 351244 79564 207848 0 0 3 3 11 7 1 0 99 0 0 0 56700 351244 79564 207848 0 0 0 8 52 27 0 0 100 0 0 0 56700 351244 79564 207848 0 0 0 0 45 14 0 0 100 0 0 0 56700 351244 79564 207848 0 0 0 0 47 17 0 0 100 0 from the man page; Swap si: Amount of memory swapped in from disk (/s). so: Amount of memory swapped to disk (/s). Exactly! My system is the same way. Right now I've got a 4GB system that's using 708MB swap. But vmstat isn't showing any swap activity. Why? Because some processes that I'm not aware about because I'm obviously not using, got swapped out a long time ago, and Linux is using that reclaimed RAM to compile chromium ;) If/when I need part of that 708MB becomes active, Linux will swap it back in in one short burst that I doubt that I'll even notice. Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? I have 12GB of RAM and 12GB of swap on my main PC. Why? Because... why not? :) After 5 days uptime, it actually has 89M of swap used for some reason. It has over 10GB cached. All of my sysctl vm.* settings have been left to the defaults. So I guess it just pushed some unused stuff out to swap to make room for more caching.
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday 21 July 2011 17:26:33 kashani did opine thusly: On 7/21/2011 4:53 PM, Grant wrote: So swap isn't treated exactly like RAM. It actually has special handling in Linux which makes it beneficial to have on almost any Linux system? According to Alan, things get very bad when a Linux system hits swap. How can behavior like this be beneficial: When a linux machine hits swap, it does so very aggressively, there is nothing nice about it at all. The entire machine slows to a painstaking crawl for easily a minute at a time while the kernel writes pages out to disk, and disk is thousands of times slower than RAM. It gets so bad that you can't even run a shell properly to try and see what's going on and kill the actual memory hog. Also, aren't you likely to wear out your hard disk sooner using swap? 1. swap is good. Unless you have a good reason, leave it there. You do not have a good reason to remove it and neither does anyone else. 2. Don't use the swap that you have. It's slow. It is not a replacement for RAM. 3. If you use a little bit of swap, 100-200MB, that's fine. It's also a sign you need more RAM. 4. If you're using all your RAM and a couple of GB of swap, you're screwed. Avoid this. 5. Swap that you never write to or read from never needs to hit the drives. If you're worried about drive wear, turn off logging. Excellent summary of swap; says a lot of what I was trying to say but didn't succeed. I might argue with your point #1, but then I would be nit-picking and it's very dependant on circumstance anyway. As in all things IT, YMMV -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday 21 July 2011 21:08:49 Albert Hopkins did opine thusly: When a linux machine hits swap, it does so very aggressively, there is nothing nice about it at all. The entire machine slows to a painstaking crawl for easily a minute at a time while the kernel writes pages out to disk, and disk is thousands of times slower than RAM. This is not entirely true. There's regular swapping and there is thrashing. Yes, thrashing is the correct word for what I described. I very seldom see swapping happen as one expects swap to be used, it almost always becomes thrashing shortly thereafter, and we do monitor our machines closely at work. That's what I've observed and it's not a large data set so I could be completely wrong. -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] mysqld invoked oom-killer
Assuming you have the concept right, if I have 'MaxClients 50' and 'MaxSpareServers 10', there should never be more than 60 apache2 processes running and I should be able to serve up to 50 simultaneous TCP sessions? I'd guess it wouldnt go past 50. Can anyone explain why I have 20 apache2 processes running moments after an apache2 restart with 'MaxSpareServers 10' and without more than 1 or 2 simultaneous TCP sessions? Do you have StartServer set to 10 (default is 2 or 3) Have any TCP sessions recently closed? Maybe the Server isn't re-cycled until the *_WAIT TCP states have timed out. Do you know if MaxClients includes clients associated with *_WAIT TCP states? I'm trying to figure out the hard limit on apache2 processes according to my config so I can plan for memory usage accordingly, but I get the feeling I'm going about it the wrong way. Doesn't every sysadmin need to do this to make sure they don't run out of memory? I have: StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 50 MaxRequestsPerChild 1 - Grant
Re: [gentoo-user] mysqld invoked oom-killer
Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? You've not understood what I said, I think. Swap is not useful as filesystem cache. Swap is as efficient (probably a little less) than the files on the disk. It's RAM that's efficient as filesystem cache. Where swap comes in is the kernel can swap out pages from stale processes, and reclaim the RAM as filesystem cache. That all makes perfect sense, but if a small swap is good and a large swap is not any better, I'm missing something. Maybe the pages from stale processes never total more than a small amount? I don't see how that could be Because you're (likely) never going to be using 100GB of memory at one time for all your processes, let alone idle processes, so what's the point of allocating all that swap? Continuing the analogy, it's like getting a stadium-sized attic that's 100x bigger than the house your building it on just to store a Christmas tree and a few other items. Here's another way of looking at it. The kernel wants to use *all* your RAM. RAM is fast (compared to disk). But it wants to use the RAM for stuff that's actually needed most at the present time. So say you have 4G RAM. You're only using maybe 1.5G memory for applications. So the kernel is going to try to use the remaining 2.5G for cache when/if it needs to. But let's say you're hitting the disk a lot because you're compiling something, then the kernel might decide it would like to cache more files than the 2.5G. So it sees you have 300M of paged in process memory that hasn't been used in a long while. A better use of RAM may be to swap out those 300M and use it for more filesystem cache, causing your compilation to run faster. But if you have a 100G swap file and only 300M of idle pages then all that extra swap isn't going to be of any use. Similarly, you don't want to swap out all of the 1.5G RAM because some of it is actually being actively used (e.g. by the compiler). That all makes perfect sense. So the reason a swap larger than maybe 1GB is not usually implemented is because idle processes don't normally have more than a few hundred MB of pages in memory? Wouldn't a sufficiently large swap (100GB for example) completely prevent out of memory conditions and the oom-killer? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
... To confuse you even more, there is a swappiness setting as well. On my old x86 rig, I have 2Gbs of ram. My hard drive is really slow since it is IDE. I set swappiness to 20. That tells the kernel that I have swap space but don't use it unless you must. For what I use the rig for, 2Gbs is plenty of ram. The lower the swappiness setting, the less the kernel will try to use ram. The higher the setting, the more it will try to use swap. I have a new rig that is amd64 and has SATA drives which are pretty fast. I still have swappiness set to 20. Why do I have it set to 20 when the drives are faster you ask? I have it set to 20 because I have 16Gbs of ram here. Even if I have portage's work directory on tmpfs and am compiling OOo, it should not need swap then either. By the way, my swap partition is 1Gb on both systems. Why have it this way since one machine has 2Gbs and one has 16Gbs? As it has been said, you want a little swap and even using a little swap is OK. You just don't want it to be using swap and actually swapping data all the time. On my old rig, it started out with 512Mbs. I use KDE and it got to the point where it was using enough ram that it was not just using swap and letting things sit, it was actively swapping data from swap and doing so a lot. It would only be using a 100Mbs sometimes 200Mbs. The point is, it was slowing the system down because of the swapping process. I bought a stick of ram and all was well again. It would still use a 100Mbs of swap at times but it would not be actively swapping the data back and forth so it wasn't a big deal. I think the point is this, it is good to have a little swap. It is even OK for it to use a little swap when it is mostly sitting there. When you notice it using swap and it is actively swapping and moving things back and forth, you need more memory. Having the swap may can save you from a crash but is can also give you a time to add more ram hint too. If Linux starts using swap a good bit, you need more ram. OK, how can you determine when a machine is actively swapping and moving things back and forth? Do you need to monitor the system with a real-time tool during peak usage? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
... Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? I have 12GB of RAM and 12GB of swap on my main PC. Why? Because... why not? :) After 5 days uptime, it actually has 89M of swap used for some reason. It has over 10GB cached. All of my sysctl vm.* settings have been left to the defaults. So I guess it just pushed some unused stuff out to swap to make room for more caching. That's what I'm curious about. If some swap is good, why isn't more better? Paul has demonstrated that a Linux system will put at least 10GB to use and probably much more given the opportunity. Disk space is so cheap, why isn't everyone running a 10GB or 100GB swap since Linux will actually put it to use? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
... Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? I have 12GB of RAM and 12GB of swap on my main PC. Why? Because... why not? :) After 5 days uptime, it actually has 89M of swap used for some reason. It has over 10GB cached. All of my sysctl vm.* settings have been left to the defaults. So I guess it just pushed some unused stuff out to swap to make room for more caching. Uh oh. Did I misunderstand you Paul? Do you have 10GB cached in swap or RAM? - Grant That's what I'm curious about. If some swap is good, why isn't more better? Paul has demonstrated that a Linux system will put at least 10GB to use and probably much more given the opportunity. Disk space is so cheap, why isn't everyone running a 10GB or 100GB swap since Linux will actually put it to use? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On Friday 22 July 2011 19:13:35 Grant wrote: Wouldn't a sufficiently large swap (100GB for example) completely prevent out of memory conditions and the oom-killer? Of course, on any system with more than a few dozen MB of RAM, but I can't imagine any combination of running programs whose size could add up to even a tenth of that, with or without library sharing (somebody will be along with an example in a moment). For instance I'm running four instances of BOINC projects here, one on each core, with oodles of space to spare and no swapping. Mind you, I do have 16GB RAM :-) Having said that I ought to go and shrink my swap partitions, but my disks are only half-allocated already, so I don't see the point. -- Rgds Peter Linux Counter number 5290
Re: [gentoo-user] mysqld invoked oom-killer
Grant wrote: ... To confuse you even more, there is a swappiness setting as well. On my old x86 rig, I have 2Gbs of ram. My hard drive is really slow since it is IDE. I set swappiness to 20. That tells the kernel that I have swap space but don't use it unless you must. For what I use the rig for, 2Gbs is plenty of ram. The lower the swappiness setting, the less the kernel will try to use ram. The higher the setting, the more it will try to use swap. I have a new rig that is amd64 and has SATA drives which are pretty fast. I still have swappiness set to 20. Why do I have it set to 20 when the drives are faster you ask? I have it set to 20 because I have 16Gbs of ram here. Even if I have portage's work directory on tmpfs and am compiling OOo, it should not need swap then either. By the way, my swap partition is 1Gb on both systems. Why have it this way since one machine has 2Gbs and one has 16Gbs? As it has been said, you want a little swap and even using a little swap is OK. You just don't want it to be using swap and actually swapping data all the time. On my old rig, it started out with 512Mbs. I use KDE and it got to the point where it was using enough ram that it was not just using swap and letting things sit, it was actively swapping data from swap and doing so a lot. It would only be using a 100Mbs sometimes 200Mbs. The point is, it was slowing the system down because of the swapping process. I bought a stick of ram and all was well again. It would still use a 100Mbs of swap at times but it would not be actively swapping the data back and forth so it wasn't a big deal. I think the point is this, it is good to have a little swap. It is even OK for it to use a little swap when it is mostly sitting there. When you notice it using swap and it is actively swapping and moving things back and forth, you need more memory. Having the swap may can save you from a crash but is can also give you a time to add more ram hint too. If Linux starts using swap a good bit, you need more ram. OK, how can you determine when a machine is actively swapping and moving things back and forth? Do you need to monitor the system with a real-time tool during peak usage? - Grant I use gkrellm on mine. It has a little charty thingy. I think iotop will show if anything is being swapped to. There may be others as well for a console or non-X machine. Most people know of top but that really doesn't help a lot. It doesn't show things moving just how much is being used. I think there is a better command that shows this but dang if I can recall it right now. Somebody help a old fart out here. lol Dale :-) :-)
Re: [gentoo-user] mysqld invoked oom-killer
On Friday 22 July 2011 19:46:25 Grant wrote: That's what I'm curious about. If some swap is good, why isn't more better? Paul has demonstrated that a Linux system will put at least 10GB to use and probably much more given the opportunity. Disk space is so cheap, why isn't everyone running a 10GB or 100GB swap since Linux will actually put it to use? On this i5 box I have two 1TB SATA disks in RAID-1 with md and lvm2 on top. There's so much space that I was lavish with swap space: each disk has 1GB at priority 10 and 10GB at priority 1. Far too much: I've never seen more than a few hundred MB swap in use, even when compiling Firefox or OO.o. -- Rgds Peter Linux Counter number 5290
Re: [gentoo-user] mysqld invoked oom-killer
On Friday, July 22 at 11:46 (-0700), Grant said: That's what I'm curious about. If some swap is good, why isn't more better? Paul has demonstrated that a Linux system will put at least 10GB to use and probably much more given the opportunity. Disk space is so cheap, why isn't everyone running a 10GB or 100GB swap since Linux will actually put it to use? Vitamin C is good for you, but if you take a whole bottle of vitamin C tablets you will die :P Seriously... I think you are just not understanding what is being said (or maybe just trying to over-generalize it). There is never a time I'm using 100G of vm at one time, so why do i need 100G of swap? Sure, I could create a 100G swap partition, but the kernel is *never* going to need to use 100G of swap at once (unless I have a *seriously* broken app), so why bother? Moreover, 100G is going to take a LONG time to swap in/out (remember disk is slower than RAM). What we are saying is, swap is good for certain conditions (which I don't feel like explaining again).
Re: [gentoo-user] mysqld invoked oom-killer
On Friday, July 22 at 19:55 (+0100), Peter Humphrey said: Wouldn't a sufficiently large swap (100GB for example) completely prevent out of memory conditions and the oom-killer? Of course, on any system with more than a few dozen MB of RAM, but I can't imagine any combination of running programs whose size could add up to even a tenth of that, with or without library sharing (somebody will be along with an example in a moment). The *prime* example is you have a program with a memory leak (omg we have programs with memory leaks?). On a system with only say 2GB swap, that program will cause oom killer to kick in fairly quickly, on a system with 100GB swap, that system is going to have to use all 100GB of swap before oom kicks in. By then your system will probably be thrashing like hell. There is no way you can complete guarantee a system won't run out of virtual memory, unless you can guarantee that there are no misbehaving applications or that some clueless guy won't isn't going to try to open a database dump in vi.* * Well you could set process/user limits to make sure a process gets an error after it tries to allocate a set limit of memory.
Re: [gentoo-user] mysqld invoked oom-killer
Peter Humphrey wrote: On Friday 22 July 2011 19:13:35 Grant wrote: Wouldn't a sufficiently large swap (100GB for example) completely prevent out of memory conditions and the oom-killer? Of course, on any system with more than a few dozen MB of RAM, but I can't imagine any combination of running programs whose size could add up to even a tenth of that, with or without library sharing (somebody will be along with an example in a moment). For instance I'm running four instances of BOINC projects here, one on each core, with oodles of space to spare and no swapping. Mind you, I do have 16GB RAM :-) Having said that I ought to go and shrink my swap partitions, but my disks are only half-allocated already, so I don't see the point. This sounds like me. I have 16Gbs here too. I have 1Gb of swap . . . . because it has always worked for me. I should have made it 300Mbs tho. The only reason I want swap is to prevent a crash long enough to maybe do something about it. This is just my opinion. Unless you are strapped for memory, mobo can't have that much, you only need a few hundred Mbs really. All you need is enough to prevent a crash and let you know when you are running short. If you have a mobo that maxes out at 1Gb or something, then you may want some swap with enough space to make up for the shortage of ram, realizing of course that it is going to slow things down, most likely a lot. If my main rig starts using swap a lot, I'm going to be very curious. I even used 8Gbs to put portages work directory on tmpfs. I still didn't use any swap. By the way, that doesn't seem to make the compiles any faster. o_O One other thing, don't forget you can adjust swapiness to control how bad thigns get before it starts using swap. A setting of 100 will use swap in a hurry and put about anything in it. I setting of 10, 20 or something means it will only use swap if it is out of ram and it can't make any available. Man it's hot here. Dale :-) :-)
Re: [gentoo-user] mysqld invoked oom-killer
On Fri, Jul 22, 2011 at 1:48 PM, Grant emailgr...@gmail.com wrote: ... Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? I have 12GB of RAM and 12GB of swap on my main PC. Why? Because... why not? :) After 5 days uptime, it actually has 89M of swap used for some reason. It has over 10GB cached. All of my sysctl vm.* settings have been left to the defaults. So I guess it just pushed some unused stuff out to swap to make room for more caching. Uh oh. Did I misunderstand you Paul? Do you have 10GB cached in swap or RAM? - Grant That's what I'm curious about. If some swap is good, why isn't more better? Paul has demonstrated that a Linux system will put at least 10GB to use and probably much more given the opportunity. Disk space is so cheap, why isn't everyone running a 10GB or 100GB swap since Linux will actually put it to use? - Grant In RAM. Total swap usage was only 89M.
Re: [gentoo-user] mysqld invoked oom-killer
On Friday, July 22 at 11:13 (-0700), Grant said: That all makes perfect sense. So the reason a swap larger than maybe 1GB is not usually implemented is because idle processes don't normally have more than a few hundred MB of pages in memory? That's not entirely true, either. For example, My laptop has 4GB of swap. Why? Well, because I use hibernate and hibernate works on the swap partition and I want to make sure that I have enough swap to write all my memory to swap (actually It's now compressed so actually I probablldon't really need that much). Wouldn't a sufficiently large swap (100GB for example) completely prevent out of memory conditions and the oom-killer? No. oom killer kicks in when your system is out of virtual memory. Consider this example: You have 4GB RAM You have 0 swap. Therefore you have a total of 4GB virtual memory. The second all your processes try to consume more than 4G of virtual memory, oom killer will kick in* Consider the next example You have 4GB RAM You have 100GB swap. Therefore you have a total of 104GB virtual memory The second all your processes try to consume more than 104GB of virtual memory, oom killer will kick in. Oom killer works on virtual memory (RAM + swap). So it doesn't matter how much RAM you have or how much swap you have, when the total virtual memory is consumed, oom killer is called. The secret is to not run out of virtual memory. There is no *easy* way not to run out of virtual memory. You either don't consume as much VM, or you provide more VM (either through RAM or swap). * This is not entirely true, the system also needs memory for the kernel, buffers, hardware drivers, and other things which simply cannot be paged out to disk, so the actual number will be less than the amount of VM.
Re: [gentoo-user] mysqld invoked oom-killer
On 7/20/2011 6:29 PM, Michael Mol wrote: Also, run a caching proxy if at all possible. That made the single biggest difference for my server. Other useful things: * Set the MaxRequestsPerChild to something like 450. That's pretty low. You'd barely get your application parsed, cached, and load some data before you'd have to recycle the child process. Most people set it around 1. Large enough to be useful, but still deal with any minor memory leaks. kashani
Re: [gentoo-user] mysqld invoked oom-killer
I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. Can anyone tell how much swap this is: /dev/sda2 80325 1140614 530145 82 Linux swap / Solaris If it's something like 512MB, that may not have prevented me from running out of memory since I have 4GB RAM. Is there any way to find out if there was a memory leak or other problem that should be investigated? To activate swap, put a line in fstab like so: /dev/sda2 none swap sw 0 0 However, you do not want to use it. it is not the life-saver some howto authors on the internet claim it to be. When a linux machine hits swap, it does so very aggressively, there is nothing nice about it at all. The entire machine slows to a painstaking crawl for easily a minute at a time while the kernel writes pages out to disk, and disk is thousands of times slower than RAM. It gets so bad that you can't even run a shell properly to try and see what's going on and kill the actual memory hog. My personal rule of thumb: if you hit swap, the bad thing has already gone very very south, usually to the point where you can't do much about it and it's already too late. Besides, that bastard deomon spawn of satan called the oom-killer is likely about to kick in and REALLY make your day. Anyone else notice how oom-killer seems to be hard coded to zap the most inconvenient process of all?. What you need to be doing is monitor your memory usage during normal conditions and deal with issues before they become problems. Hi Alan, I think it was your advice I took a long time ago when I stopped installing new machines with a swap partition and disabled it on my already-installed machines. Some time later, others on this list caught wind of what I'd done and told me I was an idiot. Is there a consensus on this? If the drawbacks and advantages of using swap cancel each other out, I won't use it. - Grant
Re: [gentoo-user] mysqld invoked oom-killer
I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. ... Switching from prefork to threads and vice versa can be very difficult depending on which modules and libraries your site uses. It is not on the list of things you should try first. Or second. Maybe 37th. I wouldn't expect adding swap to do much in this case. Your site gets hit hard, Mysql is a bit slow, Apache processes start stacking up, the system starts swapping, disk is really slow compared to RAM, and everything grinds to a complete halt possibly locking the machine up. The easiest thing to try is to turn off keepalives so child processes aren't hanging around keeping connections up. Also lower the number of Apache children to 8 * number of processors or a minimum of 32. Test a bit. Turning off keep alive can cause problems for Flash based uploaders to your site and code that expect the connection to stay up. For most sites this shouldn't matter. Next I'd look at tuning your Mysql config. If you've never touched my.cnf, by default it's set to use 64MB IIRC. You may need to raise this to get better performance. key_buffer and innodb_buffer_pool_size are the only two I'd modify without knowing more. I use the default MyISAM tables and it looks like there are three key_buffer definitions in my.cnf. One under [mysqld] is 16M, one under [isamchk] is 20M, and one under [myisamchk] is 20M. All defaults. Should I increase them all to 64M? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. ... Also, run a caching proxy if at all possible. That made the single biggest difference for my server. Other useful things: * Set the MaxRequestsPerChild to something like 450. As part of their caching, things like mod_php will grow the process size a bit as the apache process gets old in the tooth. Setting MaxRequestsPerChild lower causes the process to expire and be replaced sooner. On my server, I see apache processes consume about 60MB towards the end, and then cycle back and consume about 22MB. Default is 1 so 450 might be too low. * On my server, I have MinSpareServers at 10, and MaxSpareServers at 12. I handle spikes pretty well, and free the memory quickly. I use 20 and 40 for Min and Max respectively. * If you're using PHP, set memory_limit in php.ini as low as your applications can survive. I'm assuming you're running on a VPS or similar. At 512MB of RAM with a web server and database server, you need to keep things very tight. I have 4GB RAM. 512MB is the size of my currently unused swap partition. - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On Thu, Jul 21, 2011 at 11:39 AM, Grant emailgr...@gmail.com wrote: Hi Alan, I think it was your advice I took a long time ago when I stopped installing new machines with a swap partition and disabled it on my already-installed machines. Some time later, others on this list caught wind of what I'd done and told me I was an idiot. Is there a consensus on this? If the drawbacks and advantages of using swap cancel each other out, I won't use it. I think it's basically like this: No swap = If you run out of memory, OOM-killer starts killing things randomly and stuff breaks. With Swap = System does not run out of memory, so things don't die, but it runs poetntially much slower during that period of high memory usage depending on your disk speed and how heavily it is leaning on swap at that moment (if it is actively trying to use more data in RAM than you physically have RAM for, it's a total slowdown disaster). If it's a case of run-away memory usage, it'll run out of swap, too, anyway, so having swap in that case only delays the OOM-killer. I think if you have 4GB of RAM you shouldn't need any swap under normal circumstances. I have a gentoo box with just 256MB of RAM that's running web server (apache + php), mail server (postfix + dovecot), and database (mariadb), and it works fine if i disable swap. I do normally have swap enabled on it, though, because emerging sometimes uses a lot of RAM.
Re: [gentoo-user] mysqld invoked oom-killer
Hi Alan, I think it was your advice I took a long time ago when I stopped installing new machines with a swap partition and disabled it on my already-installed machines. Some time later, others on this list caught wind of what I'd done and told me I was an idiot. Is there a consensus on this? If the drawbacks and advantages of using swap cancel each other out, I won't use it. I think it's basically like this: No swap = If you run out of memory, OOM-killer starts killing things randomly and stuff breaks. With Swap = System does not run out of memory, so things don't die, but it runs poetntially much slower during that period of high memory usage depending on your disk speed and how heavily it is leaning on swap at that moment (if it is actively trying to use more data in RAM than you physically have RAM for, it's a total slowdown disaster). If it's a case of run-away memory usage, it'll run out of swap, too, anyway, so having swap in that case only delays the OOM-killer. I think if you have 4GB of RAM you shouldn't need any swap under normal circumstances. I have a gentoo box with just 256MB of RAM that's running web server (apache + php), mail server (postfix + dovecot), and database (mariadb), and it works fine if i disable swap. I do normally have swap enabled on it, though, because emerging sometimes uses a lot of RAM. Thanks Paul. I'm leaning toward leaving swap disabled. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On Thu, Jul 21, 2011 at 12:06 PM, Grant emailgr...@gmail.com wrote: Thanks Paul. I'm leaning toward leaving swap disabled. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? Yep.
Re: [gentoo-user] mysqld invoked oom-killer
I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. ... If you're running any other servers that utilize MySQL like Apache or something, check its access logs to see if you had an abnormal number of connections. Bruteforce hacking or some kind of flooding/DOS attack might cause it to use more memory than it ordinarily would. I don't know why I didn't check the apache2 error log before, but I got the following entry 2 seconds before the server became unresponsive: [error] server reached MaxClients setting, consider raising the MaxClients setting I use the default 256 for MaxClients. This confirms the server was brought down by too many child processes consuming too much memory. Looking back at the access_log, it's clear this condition was caused by the single IP which requested one of my pages about 300 times over the course of 1 minute. This caused my entire server to lock up for hours until I rebooted it. I hesitate to reduce MaxClients from 256. I think my server should be able to handle it since it's the default. So I need to prevent my apache2 child processes from consuming so much memory? apache2 was restarted about an hour before the lockup so it had a pretty fresh start. I do use mod_perl which is a memory hog from what I understand. Do I just need more RAM? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
* Paul Hartman paul.hartman+gen...@gmail.com [110721 12:33]: [..] I think if you have 4GB of RAM you shouldn't need any swap under normal circumstances. I have a gentoo box with just 256MB of RAM that's running web server (apache + php), mail server (postfix + dovecot), and database (mariadb), and it works fine if i disable swap. I do normally have swap enabled on it, though, because emerging sometimes uses a lot of RAM. Indeed, I have a server with 256MB of memory and it couldn't emerge gcc when I had swap disabled. However, it typically runs fine with no swap. It's only running bind basically.
Re: [gentoo-user] mysqld invoked oom-killer
Thanks Paul. I'm leaning toward leaving swap disabled. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? Yep. It sounds like adding physical RAM is better than enabling swap in every way. I'll stay in the anti-swap camp. - Grant
Re: [gentoo-user] mysqld invoked oom-killer
[..] I think if you have 4GB of RAM you shouldn't need any swap under normal circumstances. I have a gentoo box with just 256MB of RAM that's running web server (apache + php), mail server (postfix + dovecot), and database (mariadb), and it works fine if i disable swap. I do normally have swap enabled on it, though, because emerging sometimes uses a lot of RAM. Indeed, I have a server with 256MB of memory and it couldn't emerge gcc when I had swap disabled. I can't compile openoffice with -j1 on 3GB RAM and no swap. - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On 7/21/2011 9:53 AM, Grant wrote: Next I'd look at tuning your Mysql config. If you've never touched my.cnf, by default it's set to use 64MB IIRC. You may need to raise this to get better performance. key_buffer and innodb_buffer_pool_size are the only two I'd modify without knowing more. I use the default MyISAM tables and it looks like there are three key_buffer definitions in my.cnf. One under [mysqld] is 16M, one under [isamchk] is 20M, and one under [myisamchk] is 20M. All defaults. Should I increase them all to 64M? You can, but [mysqld] is the only one that matters for normal production. Depends on the size of your data and tables, but 64M is fine to start. If you've got a few GB in your databases I'd go with 256-512M or as high as you think you can get away with. Any reason you're still using MyISAM tables? Innodb is almost as fast or much much faster than MyISAM in nearly every way these days. kashani
Re: [gentoo-user] mysqld invoked oom-killer
On 7/21/2011 10:22 AM, Grant wrote: I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. ... If you're running any other servers that utilize MySQL like Apache or something, check its access logs to see if you had an abnormal number of connections. Bruteforce hacking or some kind of flooding/DOS attack might cause it to use more memory than it ordinarily would. I don't know why I didn't check the apache2 error log before, but I got the following entry 2 seconds before the server became unresponsive: [error] server reached MaxClients setting, consider raising the MaxClients setting I use the default 256 for MaxClients. This confirms the server was brought down by too many child processes consuming too much memory. Looking back at the access_log, it's clear this condition was caused by the single IP which requested one of my pages about 300 times over the course of 1 minute. This caused my entire server to lock up for hours until I rebooted it. I hesitate to reduce MaxClients from 256. I think my server should be able to handle it since it's the default. So I need to prevent my apache2 child processes from consuming so much memory? apache2 was restarted about an hour before the lockup so it had a pretty fresh start. I do use mod_perl which is a memory hog from what I understand. Do I just need more RAM? Most people do not think about this correctly. Can your server run 1 Apache processes? No, not enough resources. 1000? No, same problem. 256? I'd say no based on this thread. If you're not going to set it at 1 why try to keep it at 256? Next image a grocery store with 256 checkout lanes, but only four cashiers. Four cashiers trying to run that many lanes is actually slower than having only four lanes. However 32 lanes could faster than 4. People can have their groceries setup, baggers aren't getting in the way, etc. The analogy breaks down a bit, but you get the point. There is no performance gain in configuring for concurrency your hardware and software can not support. kashani
Re: [gentoo-user] mysqld invoked oom-killer
On Thu, Jul 21, 2011 at 11:56 AM, kashani kashani-l...@badapple.net wrote: On 7/20/2011 6:29 PM, Michael Mol wrote: Also, run a caching proxy if at all possible. That made the single biggest difference for my server. Other useful things: * Set the MaxRequestsPerChild to something like 450. That's pretty low. You'd barely get your application parsed, cached, and load some data before you'd have to recycle the child process. Most people set it around 1. Large enough to be useful, but still deal with any minor memory leaks. Depends on your application. I had to set it low because the application wouldn't fit in a 540MB VPS, otherwise. I've since bumped up to a 2GB VPS, so I can probably afford Really, a caching proxy is the first, best thing, if it's not already in use. Let the thread carry on... -- :wq
Re: [gentoo-user] mysqld invoked oom-killer
On 7/21/2011 11:55 AM, Michael Mol wrote: On Thu, Jul 21, 2011 at 11:56 AM, kashanikashani-l...@badapple.net wrote: On 7/20/2011 6:29 PM, Michael Mol wrote: Also, run a caching proxy if at all possible. That made the single biggest difference for my server. Other useful things: * Set the MaxRequestsPerChild to something like 450. That's pretty low. You'd barely get your application parsed, cached, and load some data before you'd have to recycle the child process. Most people set it around 1. Large enough to be useful, but still deal with any minor memory leaks. Depends on your application. I had to set it low because the application wouldn't fit in a 540MB VPS, otherwise. I've since bumped up to a 2GB VPS, so I can probably afford Really, a caching proxy is the first, best thing, if it's not already in use. Let the thread carry on... Hey if it worked, but I think the thrash would be expensive in a normal system where you've got a sensible amount of RAM. I do like the reverse proxy idea. Turn Apache into an application server on localhost and let the reverse proxy deal with the Internet. If you picked the right proxy multiple requests could be collapsed, static files could be served directly, etc. kashani
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday 21 July 2011 09:39:52 Grant did opine thusly: My personal rule of thumb: if you hit swap, the bad thing has already gone very very south, usually to the point where you can't do much about it and it's already too late. Besides, that bastard deomon spawn of satan called the oom-killer is likely about to kick in and REALLY make your day. Anyone else notice how oom-killer seems to be hard coded to zap the most inconvenient process of all?. What you need to be doing is monitor your memory usage during normal conditions and deal with issues before they become problems. Hi Alan, I think it was your advice I took a long time ago when I stopped installing new machines with a swap partition and disabled it on my already-installed machines. Some time later, others on this list caught wind of what I'd done and told me I was an idiot. Is there a consensus on this? If the drawbacks and advantages of using swap cancel each other out, I won't use it. I would strongly advise you to make your own measurements and heed your own counsel. I can only speak from my own experience, and I may well be speaking a whole load of codswallop. Or I may be right and the opposing view is wrong. Who's to tell? My own experience with backing swap has been almost uniformly bad, especially on machines running Apache and MySQL due to the massive performance hit it invariably causes. I see memory as a finite resource - you only have so much of it, so use it wisely and stay away from using all of it up. The oom killer is also a point of contention. The algorithm is designed to try and detect the best pid to kill in order to keep the machine up, but there is no measurement for least important process. So instead it has to infer it from time last used, time running and various other bits. These assumptions can never be 100% right. I believe the best solution to running out of memory is well-written apps that degrade or die gracefully when they hit out of memory conditions. I always thought Apache was rather good at this, it would simply kill of the child process and deliver an error. You seem to have found a way round this :-) kashani's advice seems reasonable, tune your machine to suit it's load. We've established that the problem was a client hitting your webserver 300 times in a minute. That is a DOS, so the solution would be to find a way to configure Apache to detect abuse like this and not deliver the page. But back to swap. I have two cases where it is rather useful. This notebook uses swap for image storage when hibernating, and my Sybase database servers at work use swap well, retrieving data pages from swap is faster than searching through the database indexes for where they are, simply because the kernel alredy knows exactly where the swapped data is on disk. But this is a niche circumstance and in no way representative of a typical Linux machine's behaviour. Summary: Do your own tests, make your own conclusions and vigorously defend them. Sorry for the complete lack of a definitive answer, we are victims of TheRealWorldOutThere(tm) -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday 21 July 2011 10:30:21 Grant did opine thusly: [..] I think if you have 4GB of RAM you shouldn't need any swap under normal circumstances. I have a gentoo box with just 256MB of RAM that's running web server (apache + php), mail server (postfix + dovecot), and database (mariadb), and it works fine if i disable swap. I do normally have swap enabled on it, though, because emerging sometimes uses a lot of RAM. Indeed, I have a server with 256MB of memory and it couldn't emerge gcc when I had swap disabled. I can't compile openoffice with -j1 on 3GB RAM and no swap. Sounds like a case for a swap partition that can be activated when you need it for big emerges. I hit the same thing with firefox-5 oddly enough. As for OOo, long ago I figured the pain wasn't worth the gain so now I use the -bin packages. -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday 21 July 2011 10:27:58 Grant did opine thusly: Thanks Paul. I'm leaning toward leaving swap disabled. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? Yep. It sounds like adding physical RAM is better than enabling swap in every way. I'll stay in the anti-swap camp. To throw a spanner in my own works: Some kernels *really* want at least some swap, even if it's just a little bit. IIRC it fits the role of a bit of wiggle room for when RAM is full. It came up the last time we all discussed this topic, but I no longer have those mails. If you can determine the appropriate amount, I'd recommend a small swap *file* if you decide to go this route. -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] mysqld invoked oom-killer
Next I'd look at tuning your Mysql config. If you've never touched my.cnf, by default it's set to use 64MB IIRC. You may need to raise this to get better performance. key_buffer and innodb_buffer_pool_size are the only two I'd modify without knowing more. I use the default MyISAM tables and it looks like there are three key_buffer definitions in my.cnf. One under [mysqld] is 16M, one under [isamchk] is 20M, and one under [myisamchk] is 20M. All defaults. Should I increase them all to 64M? You can, but [mysqld] is the only one that matters for normal production. Depends on the size of your data and tables, but 64M is fine to start. If you've got a few GB in your databases I'd go with 256-512M or as high as you think you can get away with. Any reason you're still using MyISAM tables? Innodb is almost as fast or much much faster than MyISAM in nearly every way these days. Just because it's the default. I can imagine there's more to switching than flipping a bit and I haven't gotten around to it yet. I've been meaning to do it. - Grant
Re: [gentoo-user] mysqld invoked oom-killer
I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. ... If you're running any other servers that utilize MySQL like Apache or something, check its access logs to see if you had an abnormal number of connections. Bruteforce hacking or some kind of flooding/DOS attack might cause it to use more memory than it ordinarily would. I don't know why I didn't check the apache2 error log before, but I got the following entry 2 seconds before the server became unresponsive: [error] server reached MaxClients setting, consider raising the MaxClients setting I use the default 256 for MaxClients. This confirms the server was brought down by too many child processes consuming too much memory. Looking back at the access_log, it's clear this condition was caused by the single IP which requested one of my pages about 300 times over the course of 1 minute. This caused my entire server to lock up for hours until I rebooted it. I hesitate to reduce MaxClients from 256. I think my server should be able to handle it since it's the default. So I need to prevent my apache2 child processes from consuming so much memory? apache2 was restarted about an hour before the lockup so it had a pretty fresh start. I do use mod_perl which is a memory hog from what I understand. Do I just need more RAM? Most people do not think about this correctly. Can your server run 1 Apache processes? No, not enough resources. 1000? No, same problem. 256? I'd say no based on this thread. If you're not going to set it at 1 why try to keep it at 256? This is illuminating, thank you. I wonder why the default is set at 256 though. Do my apache2 processes use an unusual amount of memory at 25M-67M? Even at 25M each, you would need 6.4GB of memory for 256 processes. Is an apache2 restart the only thing that prevents apache2 processes from growing in memory usage indefinitely? - Grant Next image a grocery store with 256 checkout lanes, but only four cashiers. Four cashiers trying to run that many lanes is actually slower than having only four lanes. However 32 lanes could faster than 4. People can have their groceries setup, baggers aren't getting in the way, etc. The analogy breaks down a bit, but you get the point. There is no performance gain in configuring for concurrency your hardware and software can not support. kashani
Re: [gentoo-user] mysqld invoked oom-killer
Also, run a caching proxy if at all possible. That made the single biggest difference for my server. Other useful things: * Set the MaxRequestsPerChild to something like 450. That's pretty low. You'd barely get your application parsed, cached, and load some data before you'd have to recycle the child process. Most people set it around 1. Large enough to be useful, but still deal with any minor memory leaks. Depends on your application. I had to set it low because the application wouldn't fit in a 540MB VPS, otherwise. I've since bumped up to a 2GB VPS, so I can probably afford Really, a caching proxy is the first, best thing, if it's not already in use. Let the thread carry on... A cache makes sense, but I hesitate to establish a new layer. How is the setup and maintenance? I found this which is at least not super-easy: http://www.apachetutor.org/admin/reverseproxies Would I be able to make a change to an image, immediately refresh, and see the change? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
Next I'd look at tuning your Mysql config. If you've never touched my.cnf, by default it's set to use 64MB IIRC. You may need to raise this to get better performance. key_buffer and innodb_buffer_pool_size are the only two I'd modify without knowing more. I use the default MyISAM tables and it looks like there are three key_buffer definitions in my.cnf. One under [mysqld] is 16M, one under [isamchk] is 20M, and one under [myisamchk] is 20M. All defaults. Should I increase them all to 64M? You can, but [mysqld] is the only one that matters for normal production. Depends on the size of your data and tables, but 64M is fine to start. If you've got a few GB in your databases I'd go with 256-512M or as high as you think you can get away with. Any reason you're still using MyISAM tables? Innodb is almost as fast or much much faster than MyISAM in nearly every way these days. Can multiple processes be utilized for mysql like they are for apache2? Perhaps not since it's a database? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday, July 21 at 10:27 (-0700), Grant said: It sounds like adding physical RAM is better than enabling swap in every way. I'll stay in the anti-swap camp. I don't see why it has to be one way *or* the other... Yes more RAM is always going to be better than more swap, RAM is just way faster than disk, however byte-per-byte, disk is cheaper. The whole reason why we have swap.. back in the olden days, some programs needed more RAM than perhaps the system provided. Some of these program were written with this in mind, and actually handled this by manually writing some of it's data to disk, then freeing that data from RAM, doing something, then when it needed the disk data, reading it back into RAM (after having freed the previous data). This is of course cumbersome. Enter virtual memory operating systems, which basically treat fast memory (RAM) and slow memory (disk) as one flat memory pool. Then the program it all looks like memory, and the OS does the paging in and out to disk. By now you would think oh, but if I just had one system that had a more RAM than i would ever use simultaneously, then I don't need swap, right? Well, not exactly, because modern operationg systems also do something called filesystem caching. What this does is, recently, and often used (parts of) files on the filesystem are kept into fast RAM, so when a program needs that data it can be fetched from cached RAM instead of hitting slower disk. Ok, that's nice, but what does that have to do with swap? Well, not only does Linux keep track of wIt sounds like adding physical RAM is better than enabling swap in every way. I'll stay in the anti-swap camp.hat files are used often, it also keeps track of what pages of virtual memory are *not* used often. Say you started some program a long time ago, or some program launches at boot time, but that program sleeps and doesn't do anything for a long time. Now ordinarily that program would just sit there taking up RAM. Now you are running some other programs, and these programs are very actively hitting the disk. Now Linux would love to use more RAM for caching those disk hits, but that program you haven't touched in hours is taking up RAM doing nothing. That's where swap comes in. Linux would like to take that sleeping process and swap some of its pages out to disk, so it can use the freed RAM for more cache, and therefore speed up the programs that are actually being used. Then, if the barely used process ever does wake up, Linux can expire some cache and put it back into RAM. In this case swap *is* good because it's making more efficient use of RAM by swapping out seldom-used processes and using that RAM to cache often-accessed files. So a healthy combination of swap and RAM *can* be a good thing. If, however, you have so much RAM that you can run every program you'd ever run simultaneously with every file you'd ever access cached in RAM then I wouldn't worry about swap ;) -a
Re: [gentoo-user] mysqld invoked oom-killer
On 7/21/2011 2:50 PM, Grant wrote: Any reason you're still using MyISAM tables? Innodb is almost as fast or much much faster than MyISAM in nearly every way these days. Can multiple processes be utilized for mysql like they are for apache2? Perhaps not since it's a database? Mysql is multithreaded and spawns a thread for each connection. Try a ps -efL and you should see a number of Mysql threads. However that is part of the problem with MyISAM. It throws a giant table lock blocking all other threads until the SQL statement is complete. Innodb uses row locks which allows the other threads to use the table. As far as moving to Innodb tables it's actually easy, but with a number of caveats. I'd lower your Apache max clients, tweak my.cnf, and runs some load tests before getting deep into Mysql. When you're ready I'd go about this way. 1. Make backups first. 2. See if you have any full text fields. Tables with full text fields will have to remain MyISAM. 3. Dump your database out to text. If it's not a huge amount of data I'd just vi it and change the ENGINE to Innodb. Then import the whole thing as a new database. If you have a lot of data, I'd dump the schema with -d edit, import schema, then dump your data with no create statements and finally import the data into the new database. 4. Point your staging code to the new database and test 5. Plan a maintenance window to do all the above and take the site offline while you reimport the data to be Innodb 6. take the RAM you gave to key_buffer and give it to innodb. Storage engines do not share buffers in Mysql. You can alter tables in place, but it locks them for the duration. If you site is small and low traffic you could get away with it, but testing with a copy of your site database is better. kashani
Re: [gentoo-user] mysqld invoked oom-killer
On 07/21/2011 04:49 PM, Alan McKinnon wrote: On Thursday 21 July 2011 10:27:58 Grant did opine thusly: Thanks Paul. I'm leaning toward leaving swap disabled. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? Yep. It sounds like adding physical RAM is better than enabling swap in every way. I'll stay in the anti-swap camp. To throw a spanner in my own works: Some kernels *really* want at least some swap, even if it's just a little bit. IIRC it fits the role of a bit of wiggle room for when RAM is full. I was waiting for this =) Alan's previous advice (basically, everything should fit in RAM these days) is only true in a world where the VM doesn't occasionally make stupid decisions. In real life...
Re: [gentoo-user] mysqld invoked oom-killer
... I would strongly advise you to make your own measurements and heed your own counsel. I can only speak from my own experience, and I may well be speaking a whole load of codswallop. Or I may be right and the opposing view is wrong. Who's to tell? My own experience with backing swap has been almost uniformly bad, especially on machines running Apache and MySQL due to the massive performance hit it invariably causes. I see memory as a finite resource - you only have so much of it, so use it wisely and stay away from using all of it up. The oom killer is also a point of contention. The algorithm is designed to try and detect the best pid to kill in order to keep the machine up, but there is no measurement for least important process. So instead it has to infer it from time last used, time running and various other bits. These assumptions can never be 100% right. I believe the best solution to running out of memory is well-written apps that degrade or die gracefully when they hit out of memory conditions. I always thought Apache was rather good at this, it would simply kill of the child process and deliver an error. You seem to have found a way round this :-) I'd like to know more about that if anyone has info on it. apache2 is supposed to detect when the system runs out of memory and proceed to kill child processes? It failed to do so on my system which then proceeded to lock up. kashani's advice seems reasonable, tune your machine to suit it's load. We've established that the problem was a client hitting your webserver 300 times in a minute. That is a DOS, so the solution would be to find a way to configure Apache to detect abuse like this and not deliver the page. So any apache system that hasn't been configured with a special security module can be locked up by refreshing a page a suitable number of times within a suitable amount of time? But back to swap. I have two cases where it is rather useful. This notebook uses swap for image storage when hibernating, and my Sybase database servers at work use swap well, retrieving data pages from swap is faster than searching through the database indexes for where they are, simply because the kernel alredy knows exactly where the swapped data is on disk. But this is a niche circumstance and in no way representative of a typical Linux machine's behaviour. In situations like the above, would you prefer a swap file to a swap partition? It sounds better to me. Summary: Do your own tests, make your own conclusions and vigorously defend them. Sorry for the complete lack of a definitive answer, we are victims of TheRealWorldOutThere(tm) I can handle that. :) - Grant
Re: [gentoo-user] mysqld invoked oom-killer
[..] I think if you have 4GB of RAM you shouldn't need any swap under normal circumstances. I have a gentoo box with just 256MB of RAM that's running web server (apache + php), mail server (postfix + dovecot), and database (mariadb), and it works fine if i disable swap. I do normally have swap enabled on it, though, because emerging sometimes uses a lot of RAM. Indeed, I have a server with 256MB of memory and it couldn't emerge gcc when I had swap disabled. I can't compile openoffice with -j1 on 3GB RAM and no swap. Sounds like a case for a swap partition that can be activated when you need it for big emerges. I hit the same thing with firefox-5 oddly enough. Swap file just as good? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday 21 July 2011 16:27:03 Grant did opine thusly: [..] I think if you have 4GB of RAM you shouldn't need any swap under normal circumstances. I have a gentoo box with just 256MB of RAM that's running web server (apache + php), mail server (postfix + dovecot), and database (mariadb), and it works fine if i disable swap. I do normally have swap enabled on it, though, because emerging sometimes uses a lot of RAM. Indeed, I have a server with 256MB of memory and it couldn't emerge gcc when I had swap disabled. I can't compile openoffice with -j1 on 3GB RAM and no swap. Sounds like a case for a swap partition that can be activated when you need it for big emerges. I hit the same thing with firefox-5 oddly enough. Swap file just as good? Yeah. If the docs are true, then swap files and partitions give similar performance these days. -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday 21 July 2011 19:19:07 Michael Orlitzky did opine thusly: On 07/21/2011 04:49 PM, Alan McKinnon wrote: On Thursday 21 July 2011 10:27:58 Grant did opine thusly: Thanks Paul. I'm leaning toward leaving swap disabled. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? Yep. It sounds like adding physical RAM is better than enabling swap in every way. I'll stay in the anti-swap camp. To throw a spanner in my own works: Some kernels *really* want at least some swap, even if it's just a little bit. IIRC it fits the role of a bit of wiggle room for when RAM is full. I was waiting for this =) Alan's previous advice (basically, everything should fit in RAM these days) is only true in a world where the VM doesn't occasionally make stupid decisions. In real life... :-) You caught me out. As I was typing all these posts today I was having these haunting thoughts about RealLife -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] mysqld invoked oom-killer
It sounds like adding physical RAM is better than enabling swap in every way. I'll stay in the anti-swap camp. I don't see why it has to be one way *or* the other... Yes more RAM is always going to be better than more swap, RAM is just way faster than disk, however byte-per-byte, disk is cheaper. The whole reason why we have swap.. back in the olden days, some programs needed more RAM than perhaps the system provided. Some of these program were written with this in mind, and actually handled this by manually writing some of it's data to disk, then freeing that data from RAM, doing something, then when it needed the disk data, reading it back into RAM (after having freed the previous data). This is of course cumbersome. Enter virtual memory operating systems, which basically treat fast memory (RAM) and slow memory (disk) as one flat memory pool. Then the program it all looks like memory, and the OS does the paging in and out to disk. By now you would think oh, but if I just had one system that had a more RAM than i would ever use simultaneously, then I don't need swap, right? Well, not exactly, because modern operationg systems also do something called filesystem caching. What this does is, recently, and often used (parts of) files on the filesystem are kept into fast RAM, so when a program needs that data it can be fetched from cached RAM instead of hitting slower disk. Ok, that's nice, but what does that have to do with swap? Well, not only does Linux keep track of what files are used often, it also keeps track of what pages of virtual memory are *not* used often. Say you started some program a long time ago, or some program launches at boot time, but that program sleeps and doesn't do anything for a long time. Now ordinarily that program would just sit there taking up RAM. Now you are running some other programs, and these programs are very actively hitting the disk. Now Linux would love to use more RAM for caching those disk hits, but that program you haven't touched in hours is taking up RAM doing nothing. That's where swap comes in. Linux would like to take that sleeping process and swap some of its pages out to disk, so it can use the freed RAM for more cache, and therefore speed up the programs that are actually being used. Then, if the barely used process ever does wake up, Linux can expire some cache and put it back into RAM. In this case swap *is* good because it's making more efficient use of RAM by swapping out seldom-used processes and using that RAM to cache often-accessed files. So a healthy combination of swap and RAM *can* be a good thing. If, however, you have so much RAM that you can run every program you'd ever run simultaneously with every file you'd ever access cached in RAM then I wouldn't worry about swap ;) So swap isn't treated exactly like RAM. It actually has special handling in Linux which makes it beneficial to have on almost any Linux system? According to Alan, things get very bad when a Linux system hits swap. How can behavior like this be beneficial: When a linux machine hits swap, it does so very aggressively, there is nothing nice about it at all. The entire machine slows to a painstaking crawl for easily a minute at a time while the kernel writes pages out to disk, and disk is thousands of times slower than RAM. It gets so bad that you can't even run a shell properly to try and see what's going on and kill the actual memory hog. Also, aren't you likely to wear out your hard disk sooner using swap? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
Any reason you're still using MyISAM tables? Innodb is almost as fast or much much faster than MyISAM in nearly every way these days. Can multiple processes be utilized for mysql like they are for apache2? Perhaps not since it's a database? Mysql is multithreaded and spawns a thread for each connection. Try a ps -efL and you should see a number of Mysql threads. However that is part of the problem with MyISAM. It throws a giant table lock blocking all other threads until the SQL statement is complete. Innodb uses row locks which allows the other threads to use the table. As far as moving to Innodb tables it's actually easy, but with a number of caveats. I'd lower your Apache max clients, tweak my.cnf, and runs some load tests before getting deep into Mysql. When you're ready I'd go about this way. apache MaxClients has been lowered to 50 which is a shame because I have 30+ separate images on each of my pages and that number can not be reduced. This means I may not be able to serve more than 1 full page at a time. 1. Make backups first. 2. See if you have any full text fields. Tables with full text fields will have to remain MyISAM. Many of my tables have one or more fields defined as TEXT out of laziness. Should I instead come up with an appropriate char(N) declaration for each? Can N go as high as necessary? 3. Dump your database out to text. If it's not a huge amount of data I'd just vi it and change the ENGINE to Innodb. Then import the whole thing as a new database. If you have a lot of data, I'd dump the schema with -d edit, import schema, then dump your data with no create statements and finally import the data into the new database. 4. Point your staging code to the new database and test 5. Plan a maintenance window to do all the above and take the site offline while you reimport the data to be Innodb 6. take the RAM you gave to key_buffer and give it to innodb. Storage engines do not share buffers in Mysql. OK, just leave key_buffer at the default 16M? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On 7/21/2011 4:53 PM, Grant wrote: So swap isn't treated exactly like RAM. It actually has special handling in Linux which makes it beneficial to have on almost any Linux system? According to Alan, things get very bad when a Linux system hits swap. How can behavior like this be beneficial: When a linux machine hits swap, it does so very aggressively, there is nothing nice about it at all. The entire machine slows to a painstaking crawl for easily a minute at a time while the kernel writes pages out to disk, and disk is thousands of times slower than RAM. It gets so bad that you can't even run a shell properly to try and see what's going on and kill the actual memory hog. Also, aren't you likely to wear out your hard disk sooner using swap? 1. swap is good. Unless you have a good reason, leave it there. You do not have a good reason to remove it and neither does anyone else. 2. Don't use the swap that you have. It's slow. It is not a replacement for RAM. 3. If you use a little bit of swap, 100-200MB, that's fine. It's also a sign you need more RAM. 4. If you're using all your RAM and a couple of GB of swap, you're screwed. Avoid this. 5. Swap that you never write to or read from never needs to hit the drives. If you're worried about drive wear, turn off logging. kashani
Re: [gentoo-user] mysqld invoked oom-killer
On 7/21/2011 5:14 PM, Grant wrote: Any reason you're still using MyISAM tables? Innodb is almost as fast or much much faster than MyISAM in nearly every way these days. Can multiple processes be utilized for mysql like they are for apache2? Perhaps not since it's a database? Mysql is multithreaded and spawns a thread for each connection. Try a ps -efL and you should see a number of Mysql threads. However that is part of the problem with MyISAM. It throws a giant table lock blocking all other threads until the SQL statement is complete. Innodb uses row locks which allows the other threads to use the table. As far as moving to Innodb tables it's actually easy, but with a number of caveats. I'd lower your Apache max clients, tweak my.cnf, and runs some load tests before getting deep into Mysql. When you're ready I'd go about this way. apache MaxClients has been lowered to 50 which is a shame because I have 30+ separate images on each of my pages and that number can not be reduced. This means I may not be able to serve more than 1 full page at a time. This is wrong. 1. Make backups first. 2. See if you have any full text fields. Tables with full text fields will have to remain MyISAM. Many of my tables have one or more fields defined as TEXT out of laziness. Should I instead come up with an appropriate char(N) declaration for each? Can N go as high as necessary? TEXT fields don't matter, FULL TEXT indexes do. Sorry my mistake. OK, just leave key_buffer at the default 16M? No. Make key_buffer 256M and then restart Mysql or update it from the commandline. You're starving Mysql for resources. Fix this first. Then you can mess around with tables and engines. kashani
Re: [gentoo-user] mysqld invoked oom-killer
So swap isn't treated exactly like RAM. It actually has special handling in Linux which makes it beneficial to have on almost any Linux system? According to Alan, things get very bad when a Linux system hits swap. How can behavior like this be beneficial: When a linux machine hits swap, it does so very aggressively, there is nothing nice about it at all. The entire machine slows to a painstaking crawl for easily a minute at a time while the kernel writes pages out to disk, and disk is thousands of times slower than RAM. It gets so bad that you can't even run a shell properly to try and see what's going on and kill the actual memory hog. Also, aren't you likely to wear out your hard disk sooner using swap? 1. swap is good. Unless you have a good reason, leave it there. You do not have a good reason to remove it and neither does anyone else. 2. Don't use the swap that you have. It's slow. It is not a replacement for RAM. 3. If you use a little bit of swap, 100-200MB, that's fine. It's also a sign you need more RAM. 4. If you're using all your RAM and a couple of GB of swap, you're screwed. Avoid this. 5. Swap that you never write to or read from never needs to hit the drives. If you're worried about drive wear, turn off logging. kashani OK, how about I enable a 512MB swap file and keep an eye on it. As long as I'm not using more than 200MB, I'm not suffering from disk swap slowdown, right? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
apache MaxClients has been lowered to 50 which is a shame because I have 30+ separate images on each of my pages and that number can not be reduced. This means I may not be able to serve more than 1 full page at a time. This is wrong. Agreed. From TFM; The MaxClients directive sets the limit on the number of simultaneous requests that will be served and i'd say when they say requests, they're talking about TCP sessions. So in the old days of HTTP/1.0 you'd be right, and if you'd turned off pipelining (KeepAlives) you'd be right. The default for MaxKeepAliveRequests is 100, so no problems downloading the 30+ objects within a single session, assuming you have KeepAlive on.
Re: [gentoo-user] mysqld invoked oom-killer
Any reason you're still using MyISAM tables? Innodb is almost as fast or much much faster than MyISAM in nearly every way these days. Can multiple processes be utilized for mysql like they are for apache2? Perhaps not since it's a database? Mysql is multithreaded and spawns a thread for each connection. Try a ps -efL and you should see a number of Mysql threads. However that is part of the problem with MyISAM. It throws a giant table lock blocking all other threads until the SQL statement is complete. Innodb uses row locks which allows the other threads to use the table. As far as moving to Innodb tables it's actually easy, but with a number of caveats. I'd lower your Apache max clients, tweak my.cnf, and runs some load tests before getting deep into Mysql. When you're ready I'd go about this way. apache MaxClients has been lowered to 50 which is a shame because I have 30+ separate images on each of my pages and that number can not be reduced. This means I may not be able to serve more than 1 full page at a time. This is wrong. MaxClients is defined as the limit on the number of simultaneous requests that will be served. If each of my pages requires 30 requests in order to be fully served, I don't think I'll be able to fully serve more than one page at a time. 1. Make backups first. 2. See if you have any full text fields. Tables with full text fields will have to remain MyISAM. Many of my tables have one or more fields defined as TEXT out of laziness. Should I instead come up with an appropriate char(N) declaration for each? Can N go as high as necessary? TEXT fields don't matter, FULL TEXT indexes do. Sorry my mistake. I don't have any FULL TEXT. Since TEXT is alright, I will be making the switch to InnoDB very soon. OK, just leave key_buffer at the default 16M? No. Make key_buffer 256M and then restart Mysql or update it from the commandline. You're starving Mysql for resources. Fix this first. Then you can mess around with tables and engines. Yep, I increased key_buffer when you told me to previously. - Grant
Re: [gentoo-user] mysqld invoked oom-killer
OK, how about I enable a 512MB swap file and keep an eye on it. As long as I'm not using more than 200MB, I'm not suffering from disk swap slowdown, right? Its more how much i/o rather than the size. If you have a bunch of stuff swapped out, but it hardly ever needs to be swapped in, the impact will be low. Keep an eye on the use with vmstat; adam@rix ~ $ vmstat 5 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 56700 351244 79564 20784800 3 3 117 1 0 99 0 0 0 56700 351244 79564 20784800 0 8 52 27 0 0 100 0 0 0 56700 351244 79564 20784800 0 0 45 14 0 0 100 0 0 0 56700 351244 79564 20784800 0 0 47 17 0 0 100 0 from the man page; Swap si: Amount of memory swapped in from disk (/s). so: Amount of memory swapped to disk (/s).
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday, July 21 at 16:53 (-0700), Grant said: So swap isn't treated exactly like RAM. It actually has special handling in Linux which makes it beneficial to have on almost any Linux system? According to Alan, things get very bad when a Linux system hits swap. How can behavior like this be beneficial: When a linux machine hits swap, it does so very aggressively, there is nothing nice about it at all. The entire machine slows to a painstaking crawl for easily a minute at a time while the kernel writes pages out to disk, and disk is thousands of times slower than RAM. This is not entirely true. There's regular swapping and there is thrashing. Thrashing is indicative of a memory-starved system, i.e. when many processes are trying to access memory, but there just isn't enough and the system is frantically swapping in/out. I'm talking about your normal day-to-day swapping that you probably don't even notice. It gets so bad that you can't even run a shell properly to try and see what's going on and kill the actual memory hog. Again, that is thrashing. I'm talking about normal swappage. Dont throw the baby out with the bath water. Also, aren't you likely to wear out your hard disk sooner using swap? Is this coming from someone who uses Gentoo linux, which is constantly downloading/compiling/linking object files? Syslog and other loggers writing everything under the sun to a log file. Backups, journal writes, database transactions, etc. Compare how many disk transactions take place during your normal Gentoo usage versus a few megabytes here/there being swapped in/out. Again, I'm talking about regular swapping, not oh my god I has no RAM and my hard drive won't stop Even so, we're talking about modern drives here. This isn't the 1960s.
Re: [gentoo-user] mysqld invoked oom-killer
On Friday, July 22 at 10:56 (+1000), Adam Carter said: Its more how much i/o rather than the size. If you have a bunch of stuff swapped out, but it hardly ever needs to be swapped in, the impact will be low. Keep an eye on the use with vmstat; adam@rix ~ $ vmstat 5 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 0 56700 351244 79564 20784800 3 3 117 1 0 99 0 0 0 56700 351244 79564 20784800 0 8 52 27 0 0 100 0 0 0 56700 351244 79564 20784800 0 0 45 14 0 0 100 0 0 0 56700 351244 79564 20784800 0 0 47 17 0 0 100 0 from the man page; Swap si: Amount of memory swapped in from disk (/s). so: Amount of memory swapped to disk (/s). Exactly! My system is the same way. Right now I've got a 4GB system that's using 708MB swap. But vmstat isn't showing any swap activity. Why? Because some processes that I'm not aware about because I'm obviously not using, got swapped out a long time ago, and Linux is using that reclaimed RAM to compile chromium ;) If/when I need part of that 708MB becomes active, Linux will swap it back in in one short burst that I doubt that I'll even notice.
Re: [gentoo-user] mysqld invoked oom-killer
apache MaxClients has been lowered to 50 which is a shame because I have 30+ separate images on each of my pages and that number can not be reduced. This means I may not be able to serve more than 1 full page at a time. This is wrong. Agreed. From TFM; The MaxClients directive sets the limit on the number of simultaneous requests that will be served and i'd say when they say requests, they're talking about TCP sessions. So in the old days of HTTP/1.0 you'd be right, and if you'd turned off pipelining (KeepAlives) you'd be right. The default for MaxKeepAliveRequests is 100, so no problems downloading the 30+ objects within a single session, assuming you have KeepAlive on. I'm trying to figure out the maximum number of apache2 processes that could run simultaneously according to my config so I don't run out of memory again. I have KeepAlive on, but I can see in the log that a different pid serves each file associated with a particular page request. Doesn't that mean a different apache2 process is serving each file and I need one process for each file served at any particular moment? How does KeepAlive relate to the number of running apache2 processes? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
Its more how much i/o rather than the size. If you have a bunch of stuff swapped out, but it hardly ever needs to be swapped in, the impact will be low. Keep an eye on the use with vmstat; adam@rix ~ $ vmstat 5 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 56700 351244 79564 207848 0 0 3 3 11 7 1 0 99 0 0 0 56700 351244 79564 207848 0 0 0 8 52 27 0 0 100 0 0 0 56700 351244 79564 207848 0 0 0 0 45 14 0 0 100 0 0 0 56700 351244 79564 207848 0 0 0 0 47 17 0 0 100 0 from the man page; Swap si: Amount of memory swapped in from disk (/s). so: Amount of memory swapped to disk (/s). Exactly! My system is the same way. Right now I've got a 4GB system that's using 708MB swap. But vmstat isn't showing any swap activity. Why? Because some processes that I'm not aware about because I'm obviously not using, got swapped out a long time ago, and Linux is using that reclaimed RAM to compile chromium ;) If/when I need part of that 708MB becomes active, Linux will swap it back in in one short burst that I doubt that I'll even notice. Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
I'm trying to figure out the maximum number of apache2 processes that could run simultaneously according to my config so I don't run out of memory again. I have KeepAlive on, but I can see in the log that a different pid serves each file associated with a particular page request. Ok, I didnt not expect that, and am now questioning my own understanding. Perhaps each process can only handle a single get at once, so if you're pipelining you have to use multiple processes to support the request. How did you determine each file was served from a different PID? What browser did you use to test? FWIW Firefox only sends 6 gets per persistent session. Doesn't that mean a different apache2 process is serving each file and I need one process for each file served at any particular moment? Certainly sounds like it. How does KeepAlive relate to the number of running apache2 processes? Not directly related AFAIK.
Re: [gentoo-user] mysqld invoked oom-killer
So swap isn't treated exactly like RAM. It actually has special handling in Linux which makes it beneficial to have on almost any Linux system? According to Alan, things get very bad when a Linux system hits swap. How can behavior like this be beneficial: When a linux machine hits swap, it does so very aggressively, there is nothing nice about it at all. The entire machine slows to a painstaking crawl for easily a minute at a time while the kernel writes pages out to disk, and disk is thousands of times slower than RAM. This is not entirely true. There's regular swapping and there is thrashing. Thrashing is indicative of a memory-starved system, i.e. when many processes are trying to access memory, but there just isn't enough and the system is frantically swapping in/out. I'm talking about your normal day-to-day swapping that you probably don't even notice. It gets so bad that you can't even run a shell properly to try and see what's going on and kill the actual memory hog. Again, that is thrashing. I'm talking about normal swappage. Dont throw the baby out with the bath water. Also, aren't you likely to wear out your hard disk sooner using swap? Is this coming from someone who uses Gentoo linux, which is constantly downloading/compiling/linking object files? Syslog and other loggers writing everything under the sun to a log file. Backups, journal writes, database transactions, etc. Compare how many disk transactions take place during your normal Gentoo usage versus a few megabytes here/there being swapped in/out. Again, I'm talking about regular swapping, not oh my god I has no RAM and my hard drive won't stop Even so, we're talking about modern drives here. This isn't the 1960s. If I understand correctly, an out-of-memory condition that would lock up a system without swap, will cause it to thrash with swap. A remote system of mine was locked up for many hours due to running out of memory without swap. If I had enabled swap, the system would have thrashed for those hours? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
I'm trying to figure out the maximum number of apache2 processes that could run simultaneously according to my config so I don't run out of memory again. I have KeepAlive on, but I can see in the log that a different pid serves each file associated with a particular page request. Ok, I didnt not expect that, and am now questioning my own understanding. Perhaps each process can only handle a single get at once, so if you're pipelining you have to use multiple processes to support the request. How did you determine each file was served from a different PID? What browser did you use to test? FWIW Firefox only sends 6 gets per persistent session. My apologies. I assumed the seemingly random numbers that appear at the end of my access_log entries were PIDs but they are in fact the Size of response in bytes. I'm sure your understanding of KeepAlive is correct. So with KeepAlive on, the same apache2 process serves the page itself and all associated files? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday, July 21 at 18:29 (-0700), Grant said: Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? You've not understood what I said, I think. Swap is not useful as filesystem cache. Swap is as efficient (probably a little less) than the files on the disk. It's RAM that's efficient as filesystem cache. Where swap comes in is the kernel can swap out pages from stale processes, and reclaim the RAM as filesystem cache. Think of it this way: You have a house with an attic. Now the attic is not as efficient as say, the middle of your living room. You have a Christmas tree, but you only use that Christmas tree maybe once a year. Now it's much more efficient to keep that Christmas tree in the attic for 11 months of the year and use that reclaimed space in your living room for.. say a coffee table. Then, when you need that Christmas tree in December, you pull it out of the attic and maybe put the coffee table up in the attic for a month. The Christmas tree represents a process that's just sitting out there doing not much half the time, but taking up space. The space in your living room is RAM, and the space in your attic is swap. The coffee table is filesystem cache.
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday, July 21 at 18:43 (-0700), Grant said: If I understand correctly, an out-of-memory condition that would lock up a system without swap, will cause it to thrash with swap. A remote system of mine was locked up for many hours due to running out of memory without swap. If I had enabled swap, the system would have thrashed for those hours? It really depends. The rule of thumb is, if your disk is thrashing or you are getting ooms/locks then you need more RAM (or less memory-hungry processes).
Re: [gentoo-user] mysqld invoked oom-killer
So with KeepAlive on, the same apache2 process serves the page itself and all associated files? That's my understanding, but i'm not sure if its what i've read over the years or just assumed. The way I think it worked is; - one apache process running as root, listening on port 80; - once a connection is made to port 80, the root apache process hands the connection off to a user apache process that handles all the action associated with that TCP connection. - so when using persistence, the same user apache process handles all the gets until it hits a client or user imposed limit, when the TCP connection is torn down and re-established. The tear down and re-establishment has a minor cpu cost both ends, and a latency cost, which is exacerbated by TCP slow start.
Re: [gentoo-user] mysqld invoked oom-killer
- so when using persistence, the same user apache process handles all the gets until it hits a client or user imposed limit, That should have been client or server imposed limit
Re: [gentoo-user] mysqld invoked oom-killer
So with KeepAlive on, the same apache2 process serves the page itself and all associated files? That's my understanding, but i'm not sure if its what i've read over the years or just assumed. The way I think it worked is; - one apache process running as root, listening on port 80; - once a connection is made to port 80, the root apache process hands the connection off to a user apache process that handles all the action associated with that TCP connection. - so when using persistence, the same user apache process handles all the gets until it hits a client or user imposed limit, when the TCP connection is torn down and re-established. The tear down and re-establishment has a minor cpu cost both ends, and a latency cost, which is exacerbated by TCP slow start. Assuming you have the concept right, if I have 'MaxClients 50' and 'MaxSpareServers 10', there should never be more than 60 apache2 processes running and I should be able to serve up to 50 simultaneous TCP sessions? Can anyone explain why I have 20 apache2 processes running moments after an apache2 restart with 'MaxSpareServers 10' and without more than 1 or 2 simultaneous TCP sessions? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? You've not understood what I said, I think. Swap is not useful as filesystem cache. Swap is as efficient (probably a little less) than the files on the disk. It's RAM that's efficient as filesystem cache. Where swap comes in is the kernel can swap out pages from stale processes, and reclaim the RAM as filesystem cache. That all makes perfect sense, but if a small swap is good and a large swap is not any better, I'm missing something. Maybe the pages from stale processes never total more than a small amount? I don't see how that could be. - Grant Think of it this way: You have a house with an attic. Now the attic is not as efficient as say, the middle of your living room. You have a Christmas tree, but you only use that Christmas tree maybe once a year. Now it's much more efficient to keep that Christmas tree in the attic for 11 months of the year and use that reclaimed space in your living room for.. say a coffee table. Then, when you need that Christmas tree in December, you pull it out of the attic and maybe put the coffee table up in the attic for a month. The Christmas tree represents a process that's just sitting out there doing not much half the time, but taking up space. The space in your living room is RAM, and the space in your attic is swap. The coffee table is filesystem cache.
Re: [gentoo-user] mysqld invoked oom-killer
Assuming you have the concept right, if I have 'MaxClients 50' and 'MaxSpareServers 10', there should never be more than 60 apache2 processes running and I should be able to serve up to 50 simultaneous TCP sessions? I'd guess it wouldnt go past 50. Can anyone explain why I have 20 apache2 processes running moments after an apache2 restart with 'MaxSpareServers 10' and without more than 1 or 2 simultaneous TCP sessions? Do you have StartServer set to 10 (default is 2 or 3) Have any TCP sessions recently closed? Maybe the Server isn't re-cycled until the *_WAIT TCP states have timed out.
Re: [gentoo-user] mysqld invoked oom-killer
On Thursday, July 21 at 20:07 (-0700), Grant said: Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? You've not understood what I said, I think. Swap is not useful as filesystem cache. Swap is as efficient (probably a little less) than the files on the disk. It's RAM that's efficient as filesystem cache. Where swap comes in is the kernel can swap out pages from stale processes, and reclaim the RAM as filesystem cache. That all makes perfect sense, but if a small swap is good and a large swap is not any better, I'm missing something. Maybe the pages from stale processes never total more than a small amount? I don't see how that could be Because you're (likely) never going to be using 100GB of memory at one time for all your processes, let alone idle processes, so what's the point of allocating all that swap? Continuing the analogy, it's like getting a stadium-sized attic that's 100x bigger than the house your building it on just to store a Christmas tree and a few other items. Here's another way of looking at it. The kernel wants to use *all* your RAM. RAM is fast (compared to disk). But it wants to use the RAM for stuff that's actually needed most at the present time. So say you have 4G RAM. You're only using maybe 1.5G memory for applications. So the kernel is going to try to use the remaining 2.5G for cache when/if it needs to. But let's say you're hitting the disk a lot because you're compiling something, then the kernel might decide it would like to cache more files than the 2.5G. So it sees you have 300M of paged in process memory that hasn't been used in a long while. A better use of RAM may be to swap out those 300M and use it for more filesystem cache, causing your compilation to run faster. But if you have a 100G swap file and only 300M of idle pages then all that extra swap isn't going to be of any use. Similarly, you don't want to swap out all of the 1.5G RAM because some of it is actually being actively used (e.g. by the compiler).
Re: [gentoo-user] mysqld invoked oom-killer
Grant wrote: Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? You've not understood what I said, I think. Swap is not useful as filesystem cache. Swap is as efficient (probably a little less) than the files on the disk. It's RAM that's efficient as filesystem cache. Where swap comes in is the kernel can swap out pages from stale processes, and reclaim the RAM as filesystem cache. That all makes perfect sense, but if a small swap is good and a large swap is not any better, I'm missing something. Maybe the pages from stale processes never total more than a small amount? I don't see how that could be. - Grant To confuse you even more, there is a swappiness setting as well. On my old x86 rig, I have 2Gbs of ram. My hard drive is really slow since it is IDE. I set swappiness to 20. That tells the kernel that I have swap space but don't use it unless you must. For what I use the rig for, 2Gbs is plenty of ram. The lower the swappiness setting, the less the kernel will try to use ram. The higher the setting, the more it will try to use swap. I have a new rig that is amd64 and has SATA drives which are pretty fast. I still have swappiness set to 20. Why do I have it set to 20 when the drives are faster you ask? I have it set to 20 because I have 16Gbs of ram here. Even if I have portage's work directory on tmpfs and am compiling OOo, it should not need swap then either. By the way, my swap partition is 1Gb on both systems. Why have it this way since one machine has 2Gbs and one has 16Gbs? As it has been said, you want a little swap and even using a little swap is OK. You just don't want it to be using swap and actually swapping data all the time. On my old rig, it started out with 512Mbs. I use KDE and it got to the point where it was using enough ram that it was not just using swap and letting things sit, it was actively swapping data from swap and doing so a lot. It would only be using a 100Mbs sometimes 200Mbs. The point is, it was slowing the system down because of the swapping process. I bought a stick of ram and all was well again. It would still use a 100Mbs of swap at times but it would not be actively swapping the data back and forth so it wasn't a big deal. I think the point is this, it is good to have a little swap. It is even OK for it to use a little swap when it is mostly sitting there. When you notice it using swap and it is actively swapping and moving things back and forth, you need more memory. Having the swap may can save you from a crash but is can also give you a time to add more ram hint too. If Linux starts using swap a good bit, you need more ram. I do like that attic analogy tho. You may not mind going up in the attic and dragging the tree down once a year but you may not want to go to the attic to get a glass of water. That would put a lot of wear on the stairs and it would also get old after a while to. Dale :-) :-)
Re: [gentoo-user] mysqld invoked oom-killer
Dale wrote: Grant wrote: Then why not have a really big swap file? If swap is useful as a second layer of caching behind RAM, why doesn't everyone with some extra hard drive space have a 100GB swap file? You've not understood what I said, I think. Swap is not useful as filesystem cache. Swap is as efficient (probably a little less) than the files on the disk. It's RAM that's efficient as filesystem cache. Where swap comes in is the kernel can swap out pages from stale processes, and reclaim the RAM as filesystem cache. That all makes perfect sense, but if a small swap is good and a large swap is not any better, I'm missing something. Maybe the pages from stale processes never total more than a small amount? I don't see how that could be. - Grant To confuse you even more, there is a swappiness setting as well. On my old x86 rig, I have 2Gbs of ram. My hard drive is really slow since it is IDE. I set swappiness to 20. That tells the kernel that I have swap space but don't use it unless you must. For what I use the rig for, 2Gbs is plenty of ram. The lower the swappiness setting, the less the kernel will try to use *SWAP* . The higher the setting, the more it will try to use swap. I have a new rig that is amd64 and has SATA drives which are pretty fast. I still have swappiness set to 20. Why do I have it set to 20 when the drives are faster you ask? I have it set to 20 because I have 16Gbs of ram here. Even if I have portage's work directory on tmpfs and am compiling OOo, it should not need swap then either. By the way, my swap partition is 1Gb on both systems. Why have it this way since one machine has 2Gbs and one has 16Gbs? As it has been said, you want a little swap and even using a little swap is OK. You just don't want it to be using swap and actually swapping data all the time. On my old rig, it started out with 512Mbs. I use KDE and it got to the point where it was using enough ram that it was not just using swap and letting things sit, it was actively swapping data from swap and doing so a lot. It would only be using a 100Mbs sometimes 200Mbs. The point is, it was slowing the system down because of the swapping process. I bought a stick of ram and all was well again. It would still use a 100Mbs of swap at times but it would not be actively swapping the data back and forth so it wasn't a big deal. I think the point is this, it is good to have a little swap. It is even OK for it to use a little swap when it is mostly sitting there. When you notice it using swap and it is actively swapping and moving things back and forth, you need more memory. Having the swap may can save you from a crash but is can also give you a time to add more ram hint too. If Linux starts using swap a good bit, you need more ram. I do like that attic analogy tho. You may not mind going up in the attic and dragging the tree down once a year but you may not want to go to the attic to get a glass of water. That would put a lot of wear on the stairs and it would also get old after a while to. Dale :-) :-) P. S. Corrected a disconnect between brain and keyboard. It is in *bold* in the first paragraph.
[gentoo-user] mysqld invoked oom-killer
I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. Can anyone tell how much swap this is: /dev/sda2 80325 1140614 530145 82 Linux swap / Solaris If it's something like 512MB, that may not have prevented me from running out of memory since I have 4GB RAM. Is there any way to find out if there was a memory leak or other problem that should be investigated? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
On Wed, Jul 20, 2011 at 3:30 PM, Grant emailgr...@gmail.com wrote: I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. Yes, just uncomment it and should be automatic. (you can use swapon to enable it without rebooting) Can anyone tell how much swap this is: /dev/sda2 80325 1140614 530145 82 Linux swap / Solaris If it's something like 512MB, that may not have prevented me from running out of memory since I have 4GB RAM. Is there any way to find out if there was a memory leak or other problem that should be investigated? That's 512MB. You can also create a swap file to supplement the swap partition if you don't want to or aren't able to repartition. I'd check the MySQL logs to see if it shows anything. Maybe check the settings with regard to memory upper limits (Google it, there's a lot of info about MySQL RAM management). If you're running any other servers that utilize MySQL like Apache or something, check its access logs to see if you had an abnormal number of connections. Bruteforce hacking or some kind of flooding/DOS attack might cause it to use more memory than it ordinarily would. A Basic what's using up my memory? technique is to log the output of top by using the -b command. Something like top -b toplog.txt. Then you can go back to the time when the OOM occurred and see what was using a lot of RAM at that time.
Re: [gentoo-user] mysqld invoked oom-killer
On Wednesday 20 July 2011 13:30:05 Grant did opine thusly: I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. Can anyone tell how much swap this is: /dev/sda2 80325 1140614 530145 82 Linux swap / Solaris If it's something like 512MB, that may not have prevented me from running out of memory since I have 4GB RAM. Is there any way to find out if there was a memory leak or other problem that should be investigated? To activate swap, put a line in fstab like so: /dev/sda2 noneswapsw 0 0 However, you do not want to use it. it is not the life-saver some howto authors on the internet claim it to be. When a linux machine hits swap, it does so very aggressively, there is nothing nice about it at all. The entire machine slows to a painstaking crawl for easily a minute at a time while the kernel writes pages out to disk, and disk is thousands of times slower than RAM. It gets so bad that you can't even run a shell properly to try and see what's going on and kill the actual memory hog. My personal rule of thumb: if you hit swap, the bad thing has already gone very very south, usually to the point where you can't do much about it and it's already too late. Besides, that bastard deomon spawn of satan called the oom-killer is likely about to kick in and REALLY make your day. Anyone else notice how oom-killer seems to be hard coded to zap the most inconvenient process of all?. What you need to be doing is monitor your memory usage during normal conditions and deal with issues before they become problems. -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] mysqld invoked oom-killer
I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. Yes, just uncomment it and should be automatic. (you can use swapon to enable it without rebooting) Got it. Can anyone tell how much swap this is: /dev/sda2 80325 1140614 530145 82 Linux swap / Solaris If it's something like 512MB, that may not have prevented me from running out of memory since I have 4GB RAM. Is there any way to find out if there was a memory leak or other problem that should be investigated? That's 512MB. You can also create a swap file to supplement the swap partition if you don't want to or aren't able to repartition. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? I'd check the MySQL logs to see if it shows anything. Maybe check the settings with regard to memory upper limits (Google it, there's a lot of info about MySQL RAM management). Nothing in the log and from what I read online, an error should be logged if I reach mysql's memory limit. If you're running any other servers that utilize MySQL like Apache or something, check its access logs to see if you had an abnormal number of connections. Bruteforce hacking or some kind of flooding/DOS attack might cause it to use more memory than it ordinarily would. It runs apache and I found some info there. A Basic what's using up my memory? technique is to log the output of top by using the -b command. Something like top -b toplog.txt. Then you can go back to the time when the OOM occurred and see what was using a lot of RAM at that time. The kernel actually logged some top-like output and it looks like I had a large number of apache2 processes running, likely 256 processes which is the default MaxClients. The specified total_vm for each process was about 67000 which means 256 x 67MB = 17GB??? I looked over my apache2 log and I was hit severely by a single IP right as the server went down. However, that IP looks to be a residential customer in the US and they engaged in normal browsing behavior both before and after the disruption. I think that IP may have done the refresh-100-times thing out of frustration as the server started to go down. Does it sound like apache2 was using up all the memory? If so, should I look further for a catalyst or did this likely happen slowly? What can I do to prevent it from happening again? Should I switch apache2 from prefork to threads? - Grant
Re: [gentoo-user] mysqld invoked oom-killer
Does it sound like apache2 was using up all the memory? If so, should I look further for a catalyst or did this likely happen slowly? What can I do to prevent it from happening again? Should I switch apache2 from prefork to threads? Do you need the full 256 instances? How many simultaneous connections do you have? Are you running keep alive (connection persistence, and i *think* it also implies connection pipelining). Its the default these days but if you're been upgrading through older versions the config might have it disabled. Also, WRT swap you can set vm.swappiness in /etc/sysctl.conf from the default of 60 to something lower, to reduce the propensity to use swap.
Re: [gentoo-user] mysqld invoked oom-killer
On 7/20/2011 4:08 PM, Grant wrote: I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. Yes, just uncomment it and should be automatic. (you can use swapon to enable it without rebooting) Got it. Can anyone tell how much swap this is: /dev/sda2 80325 1140614 530145 82 Linux swap / Solaris If it's something like 512MB, that may not have prevented me from running out of memory since I have 4GB RAM. Is there any way to find out if there was a memory leak or other problem that should be investigated? That's 512MB. You can also create a swap file to supplement the swap partition if you don't want to or aren't able to repartition. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? I'd check the MySQL logs to see if it shows anything. Maybe check the settings with regard to memory upper limits (Google it, there's a lot of info about MySQL RAM management). Nothing in the log and from what I read online, an error should be logged if I reach mysql's memory limit. If you're running any other servers that utilize MySQL like Apache or something, check its access logs to see if you had an abnormal number of connections. Bruteforce hacking or some kind of flooding/DOS attack might cause it to use more memory than it ordinarily would. It runs apache and I found some info there. A Basic what's using up my memory? technique is to log the output of top by using the -b command. Something like top -b toplog.txt. Then you can go back to the time when the OOM occurred and see what was using a lot of RAM at that time. The kernel actually logged some top-like output and it looks like I had a large number of apache2 processes running, likely 256 processes which is the default MaxClients. The specified total_vm for each process was about 67000 which means 256 x 67MB = 17GB??? I looked over my apache2 log and I was hit severely by a single IP right as the server went down. However, that IP looks to be a residential customer in the US and they engaged in normal browsing behavior both before and after the disruption. I think that IP may have done the refresh-100-times thing out of frustration as the server started to go down. Does it sound like apache2 was using up all the memory? If so, should I look further for a catalyst or did this likely happen slowly? What can I do to prevent it from happening again? Should I switch apache2 from prefork to threads? Switching from prefork to threads and vice versa can be very difficult depending on which modules and libraries your site uses. It is not on the list of things you should try first. Or second. Maybe 37th. I wouldn't expect adding swap to do much in this case. Your site gets hit hard, Mysql is a bit slow, Apache processes start stacking up, the system starts swapping, disk is really slow compared to RAM, and everything grinds to a complete halt possibly locking the machine up. The easiest thing to try is to turn off keepalives so child processes aren't hanging around keeping connections up. Also lower the number of Apache children to 8 * number of processors or a minimum of 32. Test a bit. Turning off keep alive can cause problems for Flash based uploaders to your site and code that expect the connection to stay up. For most sites this shouldn't matter. Next I'd look at tuning your Mysql config. If you've never touched my.cnf, by default it's set to use 64MB IIRC. You may need to raise this to get better performance. key_buffer and innodb_buffer_pool_size are the only two I'd modify without knowing more. kashani
Re: [gentoo-user] mysqld invoked oom-killer
The easiest thing to try is to turn off keepalives so child processes aren't hanging around keeping connections up. KeepAliveTimeout defaults to 5 seconds, so that shouldn't be a significant problem, and you get the efficiency of persistence and probably pipelining too. Could be worth reducing some of the tcp timers (the *_WAIT ones) if your netstat is full of CLOSE_WAIT/TIME_WAIT etc sessions (which will be more of a problem if you turn off keepalives). But completely agree, if your mysql is sluggish, it will mean apache processes hang around for longer than they need to and therefore build up and use more memory. Use top then shift-m for quick overview of memory use.
Re: [gentoo-user] mysqld invoked oom-killer
On Wed, Jul 20, 2011 at 7:54 PM, kashani kashani-l...@badapple.net wrote: On 7/20/2011 4:08 PM, Grant wrote: I ran into an out of memory problem. The first mention of it in the kernel log is mysqld invoked oom-killer. I haven't run into this before. I do have a swap partition but I don't activate it based on something I read previously that I later found out was wrong so I suppose I should activate it. Is fstab the way to do that? I have a commented line in there for swap. Yes, just uncomment it and should be automatic. (you can use swapon to enable it without rebooting) Got it. Can anyone tell how much swap this is: /dev/sda2 80325 1140614 530145 82 Linux swap / Solaris If it's something like 512MB, that may not have prevented me from running out of memory since I have 4GB RAM. Is there any way to find out if there was a memory leak or other problem that should be investigated? That's 512MB. You can also create a swap file to supplement the swap partition if you don't want to or aren't able to repartition. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? I'd check the MySQL logs to see if it shows anything. Maybe check the settings with regard to memory upper limits (Google it, there's a lot of info about MySQL RAM management). Nothing in the log and from what I read online, an error should be logged if I reach mysql's memory limit. If you're running any other servers that utilize MySQL like Apache or something, check its access logs to see if you had an abnormal number of connections. Bruteforce hacking or some kind of flooding/DOS attack might cause it to use more memory than it ordinarily would. It runs apache and I found some info there. A Basic what's using up my memory? technique is to log the output of top by using the -b command. Something like top -b toplog.txt. Then you can go back to the time when the OOM occurred and see what was using a lot of RAM at that time. The kernel actually logged some top-like output and it looks like I had a large number of apache2 processes running, likely 256 processes which is the default MaxClients. The specified total_vm for each process was about 67000 which means 256 x 67MB = 17GB??? I looked over my apache2 log and I was hit severely by a single IP right as the server went down. However, that IP looks to be a residential customer in the US and they engaged in normal browsing behavior both before and after the disruption. I think that IP may have done the refresh-100-times thing out of frustration as the server started to go down. Does it sound like apache2 was using up all the memory? If so, should I look further for a catalyst or did this likely happen slowly? What can I do to prevent it from happening again? Should I switch apache2 from prefork to threads? Switching from prefork to threads and vice versa can be very difficult depending on which modules and libraries your site uses. It is not on the list of things you should try first. Or second. Maybe 37th. I wouldn't expect adding swap to do much in this case. Your site gets hit hard, Mysql is a bit slow, Apache processes start stacking up, the system starts swapping, disk is really slow compared to RAM, and everything grinds to a complete halt possibly locking the machine up. The easiest thing to try is to turn off keepalives so child processes aren't hanging around keeping connections up. Also lower the number of Apache children to 8 * number of processors or a minimum of 32. Test a bit. Turning off keep alive can cause problems for Flash based uploaders to your site and code that expect the connection to stay up. For most sites this shouldn't matter. Next I'd look at tuning your Mysql config. If you've never touched my.cnf, by default it's set to use 64MB IIRC. You may need to raise this to get better performance. key_buffer and innodb_buffer_pool_size are the only two I'd modify without knowing more. Also, run a caching proxy if at all possible. That made the single biggest difference for my server. Other useful things: * Set the MaxRequestsPerChild to something like 450. As part of their caching, things like mod_php will grow the process size a bit as the apache process gets old in the tooth. Setting MaxRequestsPerChild lower causes the process to expire and be replaced sooner. On my server, I see apache processes consume about 60MB towards the end, and then cycle back and consume about 22MB. * On my server, I have MinSpareServers at 10, and MaxSpareServers at 12. I handle spikes pretty well, and free the memory quickly. * If you're using PHP, set memory_limit in php.ini as low as your applications can survive. I'm assuming you're running on a VPS or similar. At 512MB of RAM with a web server and database server, you need to keep
