Re: ReviewBoard server tips over about twice a week
Just to close the loop on this issue - we doubled the RAM on our server to 32GB and allocated 8GB to memcached. So far we've gone 2 weeks without any problems so it appears it was memcached that was thrashing. Thanks for all the input. ---Steve On Thursday, September 12, 2013 5:27:39 AM UTC-7, Paul Fee wrote: MaxRequestsPerChild of 4000 should be fine. I happen to have mine configured to 3000. I didn't find the root cause of my server's lock ups, at the time I noted increased memory use by Apache. A few months of evidence shows the change to be effective. BTW I'm running RB 1.5 on an old Ubuntu release, so my problems may well have been fixed in more recent versions. I expect you've got server issues different from mine, but MaxRequestsPerChild shouldn't do any harm. On Wednesday, September 11, 2013 4:19:17 PM UTC+1, Steve wrote: Thanks Paul, that's helpful. We're using the prefork MPM. Do you think 4000 is too large a number for MaxRequetsPerChild? --Steve On Wednesday, September 11, 2013 2:15:35 AM UTC-7, Paul Fee wrote: Your httpd.conf suggests you've got mod_status enabled. Therefore visit the /server-status URL on your web server. This should cause Apache httpd to provide a page that summarises the status of all its workers. That may help you understand why you've got more httpd processes than expected. Also, I've found the MaxRequestsPerChild directive to be useful on an installation of mine that would lock up occasionally. Your config shows the directive set to 4000 for the prefork MPM, but 0 (infinity) for the worker MPM. Check which MPM you're using (execute httpd -V) and ensure that MaxRequestsPerChild is taking effect. The config file contains a set of MPM tuning directives, duplicated for prefork and worker, make sure the required configuration is actually active. -- Paul On Tuesday, September 10, 2013 11:36:59 PM UTC+1, Steve wrote: I see the 'server cache' information in the admin page. It shows the memory usage at around 150MB, so clearly 64MB was not enough. Unfortunately, the load average once again climbed over 100 and I had close to 200 apache processes running. It's baffling. Can you guide me on how to limit the number of apache processes and threads? I've attached the apache httpd.conf file. Thanks Christian. --steve -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups reviewboard group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: ReviewBoard server tips over about twice a week
MaxRequestsPerChild of 4000 should be fine. I happen to have mine configured to 3000. I didn't find the root cause of my server's lock ups, at the time I noted increased memory use by Apache. A few months of evidence shows the change to be effective. BTW I'm running RB 1.5 on an old Ubuntu release, so my problems may well have been fixed in more recent versions. I expect you've got server issues different from mine, but MaxRequestsPerChild shouldn't do any harm. On Wednesday, September 11, 2013 4:19:17 PM UTC+1, Steve wrote: Thanks Paul, that's helpful. We're using the prefork MPM. Do you think 4000 is too large a number for MaxRequetsPerChild? --Steve On Wednesday, September 11, 2013 2:15:35 AM UTC-7, Paul Fee wrote: Your httpd.conf suggests you've got mod_status enabled. Therefore visit the /server-status URL on your web server. This should cause Apache httpd to provide a page that summarises the status of all its workers. That may help you understand why you've got more httpd processes than expected. Also, I've found the MaxRequestsPerChild directive to be useful on an installation of mine that would lock up occasionally. Your config shows the directive set to 4000 for the prefork MPM, but 0 (infinity) for the worker MPM. Check which MPM you're using (execute httpd -V) and ensure that MaxRequestsPerChild is taking effect. The config file contains a set of MPM tuning directives, duplicated for prefork and worker, make sure the required configuration is actually active. -- Paul On Tuesday, September 10, 2013 11:36:59 PM UTC+1, Steve wrote: I see the 'server cache' information in the admin page. It shows the memory usage at around 150MB, so clearly 64MB was not enough. Unfortunately, the load average once again climbed over 100 and I had close to 200 apache processes running. It's baffling. Can you guide me on how to limit the number of apache processes and threads? I've attached the apache httpd.conf file. Thanks Christian. --steve -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups reviewboard group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: ReviewBoard server tips over about twice a week
Your httpd.conf suggests you've got mod_status enabled. Therefore visit the /server-status URL on your web server. This should cause Apache httpd to provide a page that summarises the status of all its workers. That may help you understand why you've got more httpd processes than expected. Also, I've found the MaxRequestsPerChild directive to be useful on an installation of mine that would lock up occasionally. Your config shows the directive set to 4000 for the prefork MPM, but 0 (infinity) for the worker MPM. Check which MPM you're using (execute httpd -V) and ensure that MaxRequestsPerChild is taking effect. The config file contains a set of MPM tuning directives, duplicated for prefork and worker, make sure the required configuration is actually active. -- Paul On Tuesday, September 10, 2013 11:36:59 PM UTC+1, Steve wrote: I see the 'server cache' information in the admin page. It shows the memory usage at around 150MB, so clearly 64MB was not enough. Unfortunately, the load average once again climbed over 100 and I had close to 200 apache processes running. It's baffling. Can you guide me on how to limit the number of apache processes and threads? I've attached the apache httpd.conf file. Thanks Christian. --steve -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups reviewboard group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: ReviewBoard server tips over about twice a week
Thanks Paul, that's helpful. We're using the prefork MPM. Do you think 4000 is too large a number for MaxRequetsPerChild? --Steve On Wednesday, September 11, 2013 2:15:35 AM UTC-7, Paul Fee wrote: Your httpd.conf suggests you've got mod_status enabled. Therefore visit the /server-status URL on your web server. This should cause Apache httpd to provide a page that summarises the status of all its workers. That may help you understand why you've got more httpd processes than expected. Also, I've found the MaxRequestsPerChild directive to be useful on an installation of mine that would lock up occasionally. Your config shows the directive set to 4000 for the prefork MPM, but 0 (infinity) for the worker MPM. Check which MPM you're using (execute httpd -V) and ensure that MaxRequestsPerChild is taking effect. The config file contains a set of MPM tuning directives, duplicated for prefork and worker, make sure the required configuration is actually active. -- Paul On Tuesday, September 10, 2013 11:36:59 PM UTC+1, Steve wrote: I see the 'server cache' information in the admin page. It shows the memory usage at around 150MB, so clearly 64MB was not enough. Unfortunately, the load average once again climbed over 100 and I had close to 200 apache processes running. It's baffling. Can you guide me on how to limit the number of apache processes and threads? I've attached the apache httpd.conf file. Thanks Christian. --steve -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups reviewboard group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: ReviewBoard server tips over about twice a week
It just happened again. I had to restart apache twice. At one point I saw 135 apache processes running. And heavy traffic that I think is coming from memcache, but I'm not sure. Maybe I need to clear the cache? --Steve On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote: I know this is a shot in the dark, but I was hoping people with more apache experience and better knowledge of RB might be able to give me some pointers. We're running RB 1.7.12 on a Centos6 box on a very beefy machine which is dedicated to RB. We're using MySQL and perforce and have close to 900 users. About twice a week (sometimes more) the load average on the server spikes into the hundreds and the server becomes unresponsive. The only recourse is to stop apache, wait for the load average to come down and then restart. Sometimes when we restart, it spikes again immediately so we typically pause for a bit and let the load average get down under 1 before restarting. When this happens, it looks like there a way more apache processes running than there should be. We're using the default settings for apache on linux, which I believe limits the number of connections. It also looks like there's a lot of traffic going from apache to memcache, which is running on the same machine. I know this is not much to go on, but I was hoping someone who has experience with fine-tuning RB on a linux server might give me some pointers as to where to begin. Thanks --Steve -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups reviewboard group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: ReviewBoard server tips over about twice a week
Hi Steve,
You shouldn't need to clear the cache. Something is wrong, though. We know
of servers reliably handling thousands if users every day.
How much memory have you given memcached?
What are the specs of the server?
What kind of repositories are you using?
Are there any gigantic, multi-megabyte files it's working with (diffs or in
the repository)?
Can you show the Apache limits in your config?
Any other info you can provide about your setup would help.
Thanks,
Christian
On Tuesday, September 10, 2013, Steve wrote:
It just happened again. I had to restart apache twice. At one point I
saw 135 apache processes running. And heavy traffic that I think is coming
from memcache, but I'm not sure. Maybe I need to clear the cache?
--Steve
On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote:
I know this is a shot in the dark, but I was hoping people with more
apache experience and better knowledge of RB might be able to give me some
pointers. We're running RB 1.7.12 on a Centos6 box on a very beefy machine
which is dedicated to RB. We're using MySQL and perforce and have close to
900 users. About twice a week (sometimes more) the load average on the
server spikes into the hundreds and the server becomes unresponsive. The
only recourse is to stop apache, wait for the load average to come down and
then restart. Sometimes when we restart, it spikes again immediately so we
typically pause for a bit and let the load average get down under 1 before
restarting. When this happens, it looks like there a way more apache
processes running than there should be. We're using the default settings
for apache on linux, which I believe limits the number of connections. It
also looks like there's a lot of traffic going from apache to memcache,
which is running on the same machine.
I know this is not much to go on, but I was hoping someone who has
experience with fine-tuning RB on a linux server might give me some
pointers as to where to begin.
Thanks
--Steve
--
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups
reviewboard group.
To unsubscribe from this group and stop receiving emails from it, send an
email to reviewboard+unsubscr...@googlegroups.com javascript:_e({},
'cvml', 'reviewboard%2bunsubscr...@googlegroups.com');.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Christian Hammond - chip...@chipx86.com
Review Board - http://www.reviewboard.org
Beanbag, Inc. - http://www.beanbaginc.com
--
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups
reviewboard group.
To unsubscribe from this group and stop receiving emails from it, send an email
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Re: ReviewBoard server tips over about twice a week
On Tuesday, September 10, 2013 1:51:56 PM UTC-7, Christian Hammond wrote: Hi Steve, You shouldn't need to clear the cache. Something is wrong, though. We know of servers reliably handling thousands if users every day. How much memory have you given memcached? We're using the default settings, which allow 1024 connections and allocates 64MB of memory. I didn't realize until just now that it was only using 64MB of memory. I'm going to bump that to 2GB What are the specs of the server? Intel(R) Xeon(R) CPU 5160 @ 3.00GHz with 16 GB RAM What kind of repositories are you using? Perfore exclusively. Are there any gigantic, multi-megabyte files it's working with (diffs or in the repository)? I can't say off hand - I'd have to check that. We do sometimes get people trying to post reviews with large numbers of files. We've discouraged that but haven't stopped it completely. Can you show the Apache limits in your config? I will need your help finding that information. Are you just asking for the main apache conf file? Or the rb conf file? Hopefully, bumping the memory on memcached will resolve this. --Steve Any other info you can provide about your setup would help. Thanks, Christian On Tuesday, September 10, 2013, Steve wrote: It just happened again. I had to restart apache twice. At one point I saw 135 apache processes running. And heavy traffic that I think is coming from memcache, but I'm not sure. Maybe I need to clear the cache? --Steve On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote: I know this is a shot in the dark, but I was hoping people with more apache experience and better knowledge of RB might be able to give me some pointers. We're running RB 1.7.12 on a Centos6 box on a very beefy machine which is dedicated to RB. We're using MySQL and perforce and have close to 900 users. About twice a week (sometimes more) the load average on the server spikes into the hundreds and the server becomes unresponsive. The only recourse is to stop apache, wait for the load average to come down and then restart. Sometimes when we restart, it spikes again immediately so we typically pause for a bit and let the load average get down under 1 before restarting. When this happens, it looks like there a way more apache processes running than there should be. We're using the default settings for apache on linux, which I believe limits the number of connections. It also looks like there's a lot of traffic going from apache to memcache, which is running on the same machine. I know this is not much to go on, but I was hoping someone who has experience with fine-tuning RB on a linux server might give me some pointers as to where to begin. Thanks --Steve -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups reviewboard group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- -- Christian Hammond - chi...@chipx86.com javascript: Review Board - http://www.reviewboard.org Beanbag, Inc. - http://www.beanbaginc.com -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups reviewboard group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: ReviewBoard server tips over about twice a week
Hi Steve,
You shouldn't need to clear the cache. Something is wrong, though. We know
of servers reliably handling thousands if users every day.
How much memory have you given memcached?
What are the specs of the server?
What kind of repositories are you using?
Are there any gigantic, multi-megabyte files it's working with (diffs or in
the repository)?
Can you show the Apache limits in your config?
Any other info you can provide about your setup would help.
Thanks,
Christian
On Tuesday, September 10, 2013, Steve wrote:
It just happened again. I had to restart apache twice. At one point I
saw 135 apache processes running. And heavy traffic that I think is coming
from memcache, but I'm not sure. Maybe I need to clear the cache?
--Steve
On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote:
I know this is a shot in the dark, but I was hoping people with more
apache experience and better knowledge of RB might be able to give me some
pointers. We're running RB 1.7.12 on a Centos6 box on a very beefy machine
which is dedicated to RB. We're using MySQL and perforce and have close to
900 users. About twice a week (sometimes more) the load average on the
server spikes into the hundreds and the server becomes unresponsive. The
only recourse is to stop apache, wait for the load average to come down and
then restart. Sometimes when we restart, it spikes again immediately so we
typically pause for a bit and let the load average get down under 1 before
restarting. When this happens, it looks like there a way more apache
processes running than there should be. We're using the default settings
for apache on linux, which I believe limits the number of connections. It
also looks like there's a lot of traffic going from apache to memcache,
which is running on the same machine.
I know this is not much to go on, but I was hoping someone who has
experience with fine-tuning RB on a linux server might give me some
pointers as to where to begin.
Thanks
--Steve
--
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups
reviewboard group.
To unsubscribe from this group and stop receiving emails from it, send an
email to reviewboard+unsubscr...@googlegroups.com javascript:_e({},
'cvml', 'reviewboard%2bunsubscr...@googlegroups.com');.
For more options, visit https://groups.google.com/groups/opt_out.
--
--
Christian Hammond - chip...@chipx86.com
Review Board - http://www.reviewboard.org
Beanbag, Inc. - http://www.beanbaginc.com
--
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups
reviewboard group.
To unsubscribe from this group and stop receiving emails from it, send an email
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
ReviewBoard server tips over about twice a week
I know this is a shot in the dark, but I was hoping people with more apache experience and better knowledge of RB might be able to give me some pointers. We're running RB 1.7.12 on a Centos6 box on a very beefy machine which is dedicated to RB. We're using MySQL and perforce and have close to 900 users. About twice a week (sometimes more) the load average on the server spikes into the hundreds and the server becomes unresponsive. The only recourse is to stop apache, wait for the load average to come down and then restart. Sometimes when we restart, it spikes again immediately so we typically pause for a bit and let the load average get down under 1 before restarting. When this happens, it looks like there a way more apache processes running than there should be. We're using the default settings for apache on linux, which I believe limits the number of connections. It also looks like there's a lot of traffic going from apache to memcache, which is running on the same machine. I know this is not much to go on, but I was hoping someone who has experience with fine-tuning RB on a linux server might give me some pointers as to where to begin. Thanks --Steve -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups reviewboard group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: ReviewBoard server tips over about twice a week
On Tue, Sep 10, 2013 at 2:26 PM, Steve seide.al...@gmail.com wrote: On Tuesday, September 10, 2013 1:51:56 PM UTC-7, Christian Hammond wrote: Hi Steve, You shouldn't need to clear the cache. Something is wrong, though. We know of servers reliably handling thousands if users every day. How much memory have you given memcached? We're using the default settings, which allow 1024 connections and allocates 64MB of memory. I didn't realize until just now that it was only using 64MB of memory. I'm going to bump that to 2GB That alone is going to do wonders. We make heavy use of the cache, and diff viewing is expensive. 64MB will only hold so much of that (let alone the other things we cache) at once. The more memory you can give it, the better. Also, can you confirm in the Admin UI - General Settings page that it is indeed using memcached? The Admin UI dashboard should also provide some stats on memcached usage, if things are configured correctly. If you don't see that, it's possible it's fallen back on local memory, which would be bad. What are the specs of the server? Intel(R) Xeon(R) CPU 5160 @ 3.00GHz with 16 GB RAM Yeah, that should certainly be sufficient :) What kind of repositories are you using? Perfore exclusively. That'd be fine. Are there any gigantic, multi-megabyte files it's working with (diffs or in the repository)? I can't say off hand - I'd have to check that. We do sometimes get people trying to post reviews with large numbers of files. We've discouraged that but haven't stopped it completely. Okay. The version of RB you're running should do a pretty good job of handling larger files, and disabling expensive functionality for large ones, so that may not be the cause. Still something to consider. Of course, large changes will use more memory in memcached, so that could be part of it. Can you show the Apache limits in your config? I will need your help finding that information. Are you just asking for the main apache conf file? Or the rb conf file? Just the main Apache config file. One option, if memcached was not the culprit, is to limit the number of servers/threads running at any particular time. Hopefully, bumping the memory on memcached will resolve this. I expect it will. If it does not, do let me know, and we'll look into more. If you find you need additional scalability as you grow, we have a beta of our new product, the Review Board Power Pack ( http://www.reviewboard.org/powerpack/), which, amongst other features, makes it easier to scale out Review Board across servers, in order to share the load. Christian --Steve Any other info you can provide about your setup would help. Thanks, Christian On Tuesday, September 10, 2013, Steve wrote: It just happened again. I had to restart apache twice. At one point I saw 135 apache processes running. And heavy traffic that I think is coming from memcache, but I'm not sure. Maybe I need to clear the cache? --Steve On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote: I know this is a shot in the dark, but I was hoping people with more apache experience and better knowledge of RB might be able to give me some pointers. We're running RB 1.7.12 on a Centos6 box on a very beefy machine which is dedicated to RB. We're using MySQL and perforce and have close to 900 users. About twice a week (sometimes more) the load average on the server spikes into the hundreds and the server becomes unresponsive. The only recourse is to stop apache, wait for the load average to come down and then restart. Sometimes when we restart, it spikes again immediately so we typically pause for a bit and let the load average get down under 1 before restarting. When this happens, it looks like there a way more apache processes running than there should be. We're using the default settings for apache on linux, which I believe limits the number of connections. It also looks like there's a lot of traffic going from apache to memcache, which is running on the same machine. I know this is not much to go on, but I was hoping someone who has experience with fine-tuning RB on a linux server might give me some pointers as to where to begin. Thanks --Steve -- Get the Review Board Power Pack at http://www.reviewboard.org/** powerpack/ http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/**users/http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups reviewboard group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@**googlegroups.com. For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out . -- -- Christian Hammond -
