Re: [gentoo-user] Web Server Memory Issues
Am 13.01.2011 18:59, schrieb Kaddeh: I have a standard 2x RAM swap size of 4gb. The problem that I am seeing though is that the applications (MySQL and apache) are segfaulting -before- the system starts to swap, almost where they have an aversion to using swap. What does cat /proc/sys/vm/swappiness say? Can you use swapspace at all aka is the needed kernel switch on?
Re: [gentoo-user] Web Server Memory Issues
On Wed, Jan 12, 2011 at 1:59 PM, Kaddeh kad...@gmail.com wrote: So, I have run into an interesting problem while building out a web server for a client which I haven't come across before and I was hoping that the list would be a good way for me to find the answer. A little beckground on the systems: P4 @ 3.0Ghz 2GB PC2 4200 2x 250GB drives in RAID1 The system configurations are default for the most part with the server running MySQL and Apache. The problem that I am running into at this point, however is that the machine seems to run out of memory and will segfault either apache or mysql when does so, when apache segfaults, it is a recoverable error, when mysql does it, mysql can't recover short of restarting it. At this point, I have found a soft fix by running a cron job every 6 hours or so to clear the cached memory, which seems to be the problem, however, I would like to find a more permanent fix to this issue. Anything that would help at this point would be much appreciated. Cheers Kad I've seen a similar problem before: a chrooted webhost running Apache, MySQL, and a very old version of phpnuke. MySQL ran a muck using 50%+ of CPU time, eventually. I had a cron job set to restart it once an hour, but even that became too much. We eventually moved the site to another server on a temporary basis, then migrated to vBulletin.
Re: [gentoo-user] Web Server Memory Issues
On 12/1/2011, at 10:47pm, Kaddeh wrote: ... First, addressing the SQL issue and why I think that that could be one of the causes. The entire site, for the most part is all in one giant DB (~9GB) a significant part of that is a 3gb table full of raw image data (yes, I know that this is a REALLY bad idea to do, but I didn't design the site, I just did a migration to off-site) that being said, there could be a problem with that. I think I may have heard database guys argue for keeping image data in the DB. I'm not sure that it's always a bad idea. However: how much swap do you have? If you have a 12GB swap file, perhaps you will no longer see this problem? Stroller.
Re: [gentoo-user] Web Server Memory Issues
I have a standard 2x RAM swap size of 4gb. The problem that I am seeing though is that the applications (MySQL and apache) are segfaulting -before- the system starts to swap, almost where they have an aversion to using swap. Cheers Kad On Thu, Jan 13, 2011 at 9:34 AM, Stroller strol...@stellar.eclipse.co.ukwrote: On 12/1/2011, at 10:47pm, Kaddeh wrote: ... First, addressing the SQL issue and why I think that that could be one of the causes. The entire site, for the most part is all in one giant DB (~9GB) a significant part of that is a 3gb table full of raw image data (yes, I know that this is a REALLY bad idea to do, but I didn't design the site, I just did a migration to off-site) that being said, there could be a problem with that. I think I may have heard database guys argue for keeping image data in the DB. I'm not sure that it's always a bad idea. However: how much swap do you have? If you have a 12GB swap file, perhaps you will no longer see this problem? Stroller.
Re: [gentoo-user] Web Server Memory Issues
On 01/13/2011 09:59 AM, Kaddeh wrote: I have a standard 2x RAM swap size of 4gb. The problem that I am seeing though is that the applications (MySQL and apache) are segfaulting -before- the system starts to swap, almost where they have an aversion to using swap. Are you running 32 bits?
Re: [gentoo-user] Web Server Memory Issues
On 1/12/2011 10:59 AM, Kaddeh wrote: So, I have run into an interesting problem while building out a web server for a client which I haven't come across before and I was hoping that the list would be a good way for me to find the answer. A little beckground on the systems: P4 @ 3.0Ghz 2GB PC2 4200 2x 250GB drives in RAID1 The system configurations are default for the most part with the server running MySQL and Apache. The problem that I am running into at this point, however is that the machine seems to run out of memory and will segfault either apache or mysql when does so, when apache segfaults, it is a recoverable error, when mysql does it, mysql can't recover short of restarting it. At this point, I have found a soft fix by running a cron job every 6 hours or so to clear the cached memory, which seems to be the problem, however, I would like to find a more permanent fix to this issue. Anything that would help at this point would be much appreciated. Cheers Kad Overall I'd expect your Mysql is running slow, which causes Apache to back up, which create more Apache children while your code blocks on the db, which then uses all the RAM. 1. Assuming you're running prefork, Turn KeepAlives Off if you haven't already. That'll reduce the number of Apache threads sitting around doing nothing but using your RAM. 2. The default my.conf in Gentoo (and nearly all distros) is configured to use 64MB. You should bump this up to 512MB total. The two settings I would touch are the following and THEY ARE SEPARATE POOLS that do not share configured memory with each other. Configure accordingly. innodb_buffer_pool_size = 16M key_buffer = 16M Both variables are dynamic and can be set from with Mysql use set variables key_buffer='1024'; syntax. Assuming you use Innodb tables I'd try 256MB for that setting and 128MB for the key_buffer and see how it goes. 3. Mysql slow query log. Turn it on and look at it. Your db design sounds sketchy at best and I'd be surprised if your weren't seeing a ton of slow queries especially with no db tuning. 4. /tmp is how big? Make sure it's a couple of gigs so that Mysql can build tmp tables in it. Again your db design is strange enough that you might be generating large tmp tables that file /tmp (and / if you haven't separated them) and causes Mysql problems. This is a fairly common problem in my experience. The simplest solution is: sudo mkdid -p /home/mysql sudo chown -R mysql: /home/mysql vi /etc/mysql/my.cnf and change to tmpdir = /home/mysql/ sudo /etc/init.d/mysql restart Yes, tmpdir is *not* a dynamic variable so you will have to restart Mysql to make this change. kashani
Re: [gentoo-user] Web Server Memory Issues
yes, but that should have an effect on swap space. Cheers Kad On Thu, Jan 13, 2011 at 11:56 AM, Bill Longman bill.long...@gmail.comwrote: On 01/13/2011 09:59 AM, Kaddeh wrote: I have a standard 2x RAM swap size of 4gb. The problem that I am seeing though is that the applications (MySQL and apache) are segfaulting -before- the system starts to swap, almost where they have an aversion to using swap. Are you running 32 bits?
Re: [gentoo-user] Web Server Memory Issues
On 12. 1. 2011 19:59, Kaddeh wrote: P4 @ 3.0Ghz 2GB PC2 4200 2x 250GB drives in RAID1 The system configurations are default for the most part with the server running MySQL and Apache. The problem that I am running into at this point, however is that the machine seems to run out of memory and will segfault either apache or mysql when does so, when apache segfaults, it is a recoverable error, when mysql does it, mysql can't recover short of restarting it. At this point, I have found a soft fix by running a cron job every 6 hours or so to clear the cached memory, which seems to be the problem, however, I would like to find a more permanent fix to this issue. First of all, find what is causing that excessive memory usage. I think 2GB should be enough for moderate web with apache+mysql. Second, use some monitoring software. Personally I'm using monit and I am very satisfied with it. It can monitor processes (if it is running, answering requests, etc), resources (disk, memory, swap, cpu, i/o), files (content, permissions, checksums), remote hosts (with some basic protocol checks i.e. http, ssh, smtp, ftp, mysql, ntp, dns...), it can inform you about problems (mail, log) and you can define rules what to do in case of anomalies (i.e. if mysql is using to much memory, it will be restarted). It can start/restart processes if they die (happened to me once with sshd on server which was ~50 miles away from me). You can put monit in inittab, so in case monit itself dies it is restarted automatically. Etc, etc. Jarry -- ___ This mailbox accepts e-mails only from selected mailing-lists! Everything else is considered to be spam and therefore deleted.
Re: [gentoo-user] Web Server Memory Issues
Jarry, Thanks for the monitoring advice, I am checking out monit right now. In terms of what is the root cause of the issue, I have narrowed it down to either write caching of a SQL cache issue. First, addressing the SQL issue and why I think that that could be one of the causes. The entire site, for the most part is all in one giant DB (~9GB) a significant part of that is a 3gb table full of raw image data (yes, I know that this is a REALLY bad idea to do, but I didn't design the site, I just did a migration to off-site) that being said, there could be a problem with that. The write caching hteroy just comes up because I can clear the cached memory down to 14mb cached using 'sync echo 3 /proc/sys/vm/dump_cache' Cheers Kad On Wed, Jan 12, 2011 at 1:37 PM, Jarry mr.ja...@gmail.com wrote: On 12. 1. 2011 19:59, Kaddeh wrote: P4 @ 3.0Ghz 2GB PC2 4200 2x 250GB drives in RAID1 The system configurations are default for the most part with the server running MySQL and Apache. The problem that I am running into at this point, however is that the machine seems to run out of memory and will segfault either apache or mysql when does so, when apache segfaults, it is a recoverable error, when mysql does it, mysql can't recover short of restarting it. At this point, I have found a soft fix by running a cron job every 6 hours or so to clear the cached memory, which seems to be the problem, however, I would like to find a more permanent fix to this issue. First of all, find what is causing that excessive memory usage. I think 2GB should be enough for moderate web with apache+mysql. Second, use some monitoring software. Personally I'm using monit and I am very satisfied with it. It can monitor processes (if it is running, answering requests, etc), resources (disk, memory, swap, cpu, i/o), files (content, permissions, checksums), remote hosts (with some basic protocol checks i.e. http, ssh, smtp, ftp, mysql, ntp, dns...), it can inform you about problems (mail, log) and you can define rules what to do in case of anomalies (i.e. if mysql is using to much memory, it will be restarted). It can start/restart processes if they die (happened to me once with sshd on server which was ~50 miles away from me). You can put monit in inittab, so in case monit itself dies it is restarted automatically. Etc, etc. Jarry -- ___ This mailbox accepts e-mails only from selected mailing-lists! Everything else is considered to be spam and therefore deleted.
Re: [gentoo-user] Web Server Memory Issues
On Wed, Jan 12, 2011 at 4:47 PM, Kaddeh kad...@gmail.com wrote: Jarry, Thanks for the monitoring advice, I am checking out monit right now. In terms of what is the root cause of the issue, I have narrowed it down to either write caching of a SQL cache issue. First, addressing the SQL issue and why I think that that could be one of the causes. The entire site, for the most part is all in one giant DB (~9GB) a significant part of that is a 3gb table full of raw image data (yes, I know that this is a REALLY bad idea to do, but I didn't design the site, I just did a migration to off-site) that being said, there could be a problem with that. The write caching hteroy just comes up because I can clear the cached memory down to 14mb cached using 'sync echo 3 /proc/sys/vm/dump_cache' Cheers Kad On Wed, Jan 12, 2011 at 1:37 PM, Jarry mr.ja...@gmail.com wrote: On 12. 1. 2011 19:59, Kaddeh wrote: P4 @ 3.0Ghz 2GB PC2 4200 2x 250GB drives in RAID1 The system configurations are default for the most part with the server running MySQL and Apache. The problem that I am running into at this point, however is that the machine seems to run out of memory and will segfault either apache or mysql when does so, when apache segfaults, it is a recoverable error, when mysql does it, mysql can't recover short of restarting it. At this point, I have found a soft fix by running a cron job every 6 hours or so to clear the cached memory, which seems to be the problem, however, I would like to find a more permanent fix to this issue. First of all, find what is causing that excessive memory usage. I think 2GB should be enough for moderate web with apache+mysql. Second, use some monitoring software. Personally I'm using monit and I am very satisfied with it. It can monitor processes (if it is running, answering requests, etc), resources (disk, memory, swap, cpu, i/o), files (content, permissions, checksums), remote hosts (with some basic protocol checks i.e. http, ssh, smtp, ftp, mysql, ntp, dns...), it can inform you about problems (mail, log) and you can define rules what to do in case of anomalies (i.e. if mysql is using to much memory, it will be restarted). It can start/restart processes if they die (happened to me once with sshd on server which was ~50 miles away from me). You can put monit in inittab, so in case monit itself dies it is restarted automatically. Etc, etc. Jarry -- ___ This mailbox accepts e-mails only from selected mailing-lists! Everything else is considered to be spam and therefore deleted. So, a few questions: What apache MPM are you using? You can control the number of processes or threads in that file. The default is something like 200 processes or threads (depending on MPM), so that could cause issues. What does your my.cnf look like? MySQL makes it pretty easy to regulate memory usage in my.cnf. What sort of webapp is this, PHP, python, perl, ...? That should be a good start. Cheers -- Matthew W. Summers
Re: [gentoo-user] Web Server Memory Issues
Matthew, Default settings for both my.cnf and httpd.conf are defaults, however, I would assume that a restart of a service would clear up the memory that was used by child processes. The only things that are really different in my.cnf is the base stuff like bin-log and such for doing DB replication. As for the webapp itself, it is PHP, but that is literally to make the MySQL connections to pull down the pages in the database (literally, entire pages of html in columns). Cheers Kad On Wed, Jan 12, 2011 at 4:08 PM, Matthew Summers quantumsumm...@gentoo.orgwrote: On Wed, Jan 12, 2011 at 4:47 PM, Kaddeh kad...@gmail.com wrote: Jarry, Thanks for the monitoring advice, I am checking out monit right now. In terms of what is the root cause of the issue, I have narrowed it down to either write caching of a SQL cache issue. First, addressing the SQL issue and why I think that that could be one of the causes. The entire site, for the most part is all in one giant DB (~9GB) a significant part of that is a 3gb table full of raw image data (yes, I know that this is a REALLY bad idea to do, but I didn't design the site, I just did a migration to off-site) that being said, there could be a problem with that. The write caching hteroy just comes up because I can clear the cached memory down to 14mb cached using 'sync echo 3 /proc/sys/vm/dump_cache' Cheers Kad On Wed, Jan 12, 2011 at 1:37 PM, Jarry mr.ja...@gmail.com wrote: On 12. 1. 2011 19:59, Kaddeh wrote: P4 @ 3.0Ghz 2GB PC2 4200 2x 250GB drives in RAID1 The system configurations are default for the most part with the server running MySQL and Apache. The problem that I am running into at this point, however is that the machine seems to run out of memory and will segfault either apache or mysql when does so, when apache segfaults, it is a recoverable error, when mysql does it, mysql can't recover short of restarting it. At this point, I have found a soft fix by running a cron job every 6 hours or so to clear the cached memory, which seems to be the problem, however, I would like to find a more permanent fix to this issue. First of all, find what is causing that excessive memory usage. I think 2GB should be enough for moderate web with apache+mysql. Second, use some monitoring software. Personally I'm using monit and I am very satisfied with it. It can monitor processes (if it is running, answering requests, etc), resources (disk, memory, swap, cpu, i/o), files (content, permissions, checksums), remote hosts (with some basic protocol checks i.e. http, ssh, smtp, ftp, mysql, ntp, dns...), it can inform you about problems (mail, log) and you can define rules what to do in case of anomalies (i.e. if mysql is using to much memory, it will be restarted). It can start/restart processes if they die (happened to me once with sshd on server which was ~50 miles away from me). You can put monit in inittab, so in case monit itself dies it is restarted automatically. Etc, etc. Jarry -- ___ This mailbox accepts e-mails only from selected mailing-lists! Everything else is considered to be spam and therefore deleted. So, a few questions: What apache MPM are you using? You can control the number of processes or threads in that file. The default is something like 200 processes or threads (depending on MPM), so that could cause issues. What does your my.cnf look like? MySQL makes it pretty easy to regulate memory usage in my.cnf. What sort of webapp is this, PHP, python, perl, ...? That should be a good start. Cheers -- Matthew W. Summers
