>> I ran into an out of memory problem. The first mention of it in the >> kernel log is "mysqld invoked oom-killer". I haven't run into this >> before. I do have a swap partition but I don't activate it based on >> something I read previously that I later found out was wrong so I >> suppose I should activate it. Is fstab the way to do that? I have a >> commented line in there for swap. > > Yes, just uncomment it and should be automatic. (you can use "swapon" > to enable it without rebooting)
Got it. >> Can anyone tell how much swap this is: >> >> /dev/sda2 80325 1140614 530145 82 Linux swap / Solaris >> >> If it's something like 512MB, that may not have prevented me from >> running out of memory since I have 4GB RAM. Is there any way to find >> out if there was a memory leak or other problem that should be >> investigated? > > That's 512MB. You can also create a swap file to supplement the swap > partition if you don't want to or aren't able to repartition. So I'm sure I have the concept right, is adding a 1GB swap partition functionally identical to adding 1GB RAM with regard to the potential for out-of-memory conditions? > I'd check the MySQL logs to see if it shows anything. Maybe check the > settings with regard to memory upper limits (Google it, there's a lot > of info about MySQL RAM management). Nothing in the log and from what I read online, an error should be logged if I reach mysql's memory limit. > If you're running any other servers that utilize MySQL like Apache or > something, check its access logs to see if you had an abnormal number > of connections. Bruteforce hacking or some kind of flooding/DOS attack > might cause it to use more memory than it ordinarily would. It runs apache and I found some info there. > A Basic "what's using up my memory?" technique is to log the output of > "top" by using the -b command. Something like "top -b > toplog.txt". > Then you can go back to the time when the OOM occurred and see what > was using a lot of RAM at that time. The kernel actually logged some top-like output and it looks like I had a large number of apache2 processes running, likely 256 processes which is the default MaxClients. The specified total_vm for each process was about 67000 which means 256 x 67MB = 17GB??? I looked over my apache2 log and I was hit severely by a single IP right as the server went down. However, that IP looks to be a residential customer in the US and they engaged in normal browsing behavior both before and after the disruption. I think that IP may have done the refresh-100-times thing out of frustration as the server started to go down. Does it sound like apache2 was using up all the memory? If so, should I look further for a catalyst or did this likely happen slowly? What can I do to prevent it from happening again? Should I switch apache2 from prefork to threads? - Grant
