[CentOS] x86_64 server OOM problems after update to 5.3
Hello, Over the weekend, I upgraded one of my servers that runs mysql and pdns to 5.3. Previous to the update I have not had an issue form this server. But since, I have had mysql die multiple times from oom-killer. $ uname -a Linux rack2a 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux Unfortunately, the server has been rebooted by others before I have been able to look at it while the problem is occurring. But I have found this in the logs: http://pastebin.centos.org/25553 Running sar -A: http://pastebin.centos.org/25556 Since the server was rebooted, the amount of swap is used is 0. How do I determine what process is/was chewing up the memory on this server? What should I be looking for to narrow this down? Thanks, Rick ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] x86_64 server OOM problems after update to 5.3
Rick Barnes wrote: How do I determine what process is/was chewing up the memory on this server? What should I be looking for to narrow this down? Setup a script to monitor memory usage and alert you when swap usage starts getting high. Or setup a monitor that just runs ps auxw or something and sends output to a file so the next time it happens you know how much memory everything was using at the time. nate ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] x86_64 server OOM problems after update to 5.3
On Mon, 2009-04-13 at 10:31 -0400, Rick Barnes wrote: Hello, Over the weekend, I upgraded one of my servers that runs mysql and pdns to 5.3. Previous to the update I have not had an issue form this server. But since, I have had mysql die multiple times from oom-killer. $ uname -a Linux rack2a 2.6.18-128.1.6.el5 #1 SMP Wed Apr 1 09:10:25 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux Unfortunately, the server has been rebooted by others before I have been able to look at it while the problem is occurring. But I have found this in the logs: http://pastebin.centos.org/25553 Running sar -A: http://pastebin.centos.org/25556 Since the server was rebooted, the amount of swap is used is 0. How do I determine what process is/was chewing up the memory on this server? What should I be looking for to narrow this down? Thanks, Rick --- You found your problem already. MYSQL. There could be something helping it also. When SQL server begins running out of RAM there could be many things to blame on it doing so. What actually does mysql do? As in what do you use it for. Try ps –auxf | grep mysqld. Something else you can use is mytop. You can use the below as a script and it works in an infinite loop or just use the command only. What you need to see is the running queries. ## #!/bin/bash while [ 1 ] do mysql -N -u root -ppassword -e 'show processlist' |grep -v 'show processlist' sleep 2 done ## JohnStanley ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] x86_64 server OOM problems after update to 5.3
On Mon, 2009-04-13 at 10:31 -0400, Rick Barnes wrote: Hello, I have had mysql die multiple times from oom-killer. Thanks, Rick --- In seeing what you do I would be looking at the queries being made. SPROCs, Triggers, Views. Also very import is if that one particular data base is doing graffing running predictions as you all do GIS and Mapping. Predictions can be huge memory burners. Last thing does it run as a virtual machine or on real hardware? JohnStanley ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] x86_64 server OOM problems after update to 5.3
JohnS wrote: On Mon, 2009-04-13 at 10:31 -0400, Rick Barnes wrote: Hello, I have had mysql die multiple times from oom-killer. Thanks, Rick --- In seeing what you do I would be looking at the queries being made. SPROCs, Triggers, Views. Also very import is if that one particular data base is doing graffing running predictions as you all do GIS and Mapping. Predictions can be huge memory burners. Last thing does it run as a virtual machine or on real hardware? This is a real hardware server. It is not doing GIS or mapping of the like, most of the DBs on this are for web sites, mostly form data storage, some CMS DBs for joomla, WP, etc. also some others for cacti and powerdns backends. I have setup some scripts to watch the swap usage to see if I can track this down. Thanks for the help Rick ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] x86_64 server OOM problems after update to 5.3
JohnS wrote: Seeing as you said you upgraded from 5.2 - 5.3 I would be looking at the kernel release notes and the mysql release notes for known problems since you did not have prior problems. I would check out the Cacti and DNS Databases because there more realtime in nature to running on the server than the content ones. Using the script I posted will catch the offending query. I myself would take a hard look @ MYSQL itself. There is a huge debate about it not being Production Ready. Last option would be to do a yum --allow-downgrade until it's sorted out on a test machine. It appears as though apache is to blame: http://pastebin.centos.org/25568 By stopping and starting apache, %swpused went from 92.84% to 6.41% and has remained for about an hour now. Thanks, Rick ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] x86_64 server OOM problems after update to 5.3
On Mon, 2009-04-13 at 15:25 -0400, Rick Barnes wrote: JohnS wrote: Seeing as you said you upgraded from 5.2 - 5.3 I would be looking at the kernel release notes and the mysql release notes for known problems since you did not have prior problems. I would check out the Cacti and DNS Databases because there more realtime in nature to running on the server than the content ones. Using the script I posted will catch the offending query. I myself would take a hard look @ MYSQL itself. There is a huge debate about it not being Production Ready. Last option would be to do a yum --allow-downgrade until it's sorted out on a test machine. It appears as though apache is to blame: http://pastebin.centos.org/25568 By stopping and starting apache, %swpused went from 92.84% to 6.41% and has remained for about an hour now. Thanks, Rick --- Now you get to nail down the offending site/application. By the way that's a lot of ram for apache to eat up. I may be wrong but didn't MYSQL show eating all that RAM also? Keep in mind that bad sql queries will also make a web server eat ram. Totally dependent on your situation. JohnStanley ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] x86_64 server OOM problems after update to 5.3
On Mon, 2009-04-13 at 14:18 -0400, Rick Barnes wrote: This is a real hardware server. It is not doing GIS or mapping of the like, most of the DBs on this are for web sites, mostly form data storage, some CMS DBs for joomla, WP, etc. also some others for cacti and powerdns backends. I have setup some scripts to watch the swap usage to see if I can track this down. Thanks for the help Rick --- Seeing as you said you upgraded from 5.2 - 5.3 I would be looking at the kernel release notes and the mysql release notes for known problems since you did not have prior problems. I would check out the Cacti and DNS Databases because there more realtime in nature to running on the server than the content ones. Using the script I posted will catch the offending query. I myself would take a hard look @ MYSQL itself. There is a huge debate about it not being Production Ready. Last option would be to do a yum --allow-downgrade until it's sorted out on a test machine. JohnStanley ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
