Re: ReviewBoard server tips over about twice a week

2013-10-07 Thread Steve 
Just to close the loop on this issue - we doubled the RAM on our server to 
32GB and allocated 8GB to memcached.  So far we've gone 2 weeks without any 
problems so it appears it was memcached that was thrashing.

Thanks for all the input.

---Steve


On Thursday, September 12, 2013 5:27:39 AM UTC-7, Paul Fee wrote:
>
> MaxRequestsPerChild of 4000 should be fine.  I happen to have mine 
> configured to 3000.  I didn't find the root cause of my server's lock ups, 
> at the time I noted increased memory use by Apache.  A few months of 
> evidence shows the change to be effective.  BTW I'm running RB 1.5 on an 
> old Ubuntu release, so my problems may well have been fixed in more recent 
> versions.  I expect you've got server issues different from mine, but 
> MaxRequestsPerChild shouldn't do any harm.
>
> On Wednesday, September 11, 2013 4:19:17 PM UTC+1, Steve wrote:
>>
>> Thanks Paul, that's helpful. We're using the prefork MPM.  Do you think 
>> 4000 is too large a number for MaxRequetsPerChild?
>>
>> --Steve
>>
>> On Wednesday, September 11, 2013 2:15:35 AM UTC-7, Paul Fee wrote:
>>>
>>> Your httpd.conf suggests you've got mod_status enabled.  Therefore visit 
>>> the /server-status URL on your web server.  This should cause Apache httpd 
>>> to provide a page that summarises the status of all its workers.  That may 
>>> help you understand why you've got more httpd processes than expected.
>>>
>>> Also, I've found the MaxRequestsPerChild directive to be useful on an 
>>> installation of mine that would lock up occasionally.  Your config shows 
>>> the directive set to 4000 for the prefork MPM, but 0 (infinity) for the 
>>> worker MPM.  Check which MPM you're using (execute "httpd -V") and ensure 
>>> that MaxRequestsPerChild is taking effect.  The config file contains a set 
>>> of MPM tuning directives, duplicated for prefork and worker, make sure the 
>>> required configuration is actually active.
>>>
>>> -- 
>>> Paul
>>>
>>> On Tuesday, September 10, 2013 11:36:59 PM UTC+1, Steve wrote:

 I see the 'server cache' information in the admin page.  It shows the 
 memory usage at around 150MB, so clearly 64MB was not enough. 
  Unfortunately, the load average once again climbed over 100 and I had 
 close to 200 apache processes running.  It's baffling.  Can you guide me 
 on 
 how to limit the number of apache processes and threads?  I've attached 
 the 
 apache httpd.conf file.

 Thanks Christian.

 --steve

>>>

-- 
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ReviewBoard server tips over about twice a week

2013-09-12 Thread Paul Fee 
MaxRequestsPerChild of 4000 should be fine.  I happen to have mine 
configured to 3000.  I didn't find the root cause of my server's lock ups, 
at the time I noted increased memory use by Apache.  A few months of 
evidence shows the change to be effective.  BTW I'm running RB 1.5 on an 
old Ubuntu release, so my problems may well have been fixed in more recent 
versions.  I expect you've got server issues different from mine, but 
MaxRequestsPerChild shouldn't do any harm.

On Wednesday, September 11, 2013 4:19:17 PM UTC+1, Steve wrote:
>
> Thanks Paul, that's helpful. We're using the prefork MPM.  Do you think 
> 4000 is too large a number for MaxRequetsPerChild?
>
> --Steve
>
> On Wednesday, September 11, 2013 2:15:35 AM UTC-7, Paul Fee wrote:
>>
>> Your httpd.conf suggests you've got mod_status enabled.  Therefore visit 
>> the /server-status URL on your web server.  This should cause Apache httpd 
>> to provide a page that summarises the status of all its workers.  That may 
>> help you understand why you've got more httpd processes than expected.
>>
>> Also, I've found the MaxRequestsPerChild directive to be useful on an 
>> installation of mine that would lock up occasionally.  Your config shows 
>> the directive set to 4000 for the prefork MPM, but 0 (infinity) for the 
>> worker MPM.  Check which MPM you're using (execute "httpd -V") and ensure 
>> that MaxRequestsPerChild is taking effect.  The config file contains a set 
>> of MPM tuning directives, duplicated for prefork and worker, make sure the 
>> required configuration is actually active.
>>
>> -- 
>> Paul
>>
>> On Tuesday, September 10, 2013 11:36:59 PM UTC+1, Steve wrote:
>>>
>>> I see the 'server cache' information in the admin page.  It shows the 
>>> memory usage at around 150MB, so clearly 64MB was not enough. 
>>>  Unfortunately, the load average once again climbed over 100 and I had 
>>> close to 200 apache processes running.  It's baffling.  Can you guide me on 
>>> how to limit the number of apache processes and threads?  I've attached the 
>>> apache httpd.conf file.
>>>
>>> Thanks Christian.
>>>
>>> --steve
>>>
>>

-- 
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ReviewBoard server tips over about twice a week

2013-09-11 Thread Steve 
Thanks Paul, that's helpful. We're using the prefork MPM.  Do you think 
4000 is too large a number for MaxRequetsPerChild?

--Steve

On Wednesday, September 11, 2013 2:15:35 AM UTC-7, Paul Fee wrote:
>
> Your httpd.conf suggests you've got mod_status enabled.  Therefore visit 
> the /server-status URL on your web server.  This should cause Apache httpd 
> to provide a page that summarises the status of all its workers.  That may 
> help you understand why you've got more httpd processes than expected.
>
> Also, I've found the MaxRequestsPerChild directive to be useful on an 
> installation of mine that would lock up occasionally.  Your config shows 
> the directive set to 4000 for the prefork MPM, but 0 (infinity) for the 
> worker MPM.  Check which MPM you're using (execute "httpd -V") and ensure 
> that MaxRequestsPerChild is taking effect.  The config file contains a set 
> of MPM tuning directives, duplicated for prefork and worker, make sure the 
> required configuration is actually active.
>
> -- 
> Paul
>
> On Tuesday, September 10, 2013 11:36:59 PM UTC+1, Steve wrote:
>>
>> I see the 'server cache' information in the admin page.  It shows the 
>> memory usage at around 150MB, so clearly 64MB was not enough. 
>>  Unfortunately, the load average once again climbed over 100 and I had 
>> close to 200 apache processes running.  It's baffling.  Can you guide me on 
>> how to limit the number of apache processes and threads?  I've attached the 
>> apache httpd.conf file.
>>
>> Thanks Christian.
>>
>> --steve
>>
>

-- 
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ReviewBoard server tips over about twice a week

2013-09-11 Thread Paul Fee 
Your httpd.conf suggests you've got mod_status enabled.  Therefore visit 
the /server-status URL on your web server.  This should cause Apache httpd 
to provide a page that summarises the status of all its workers.  That may 
help you understand why you've got more httpd processes than expected.

Also, I've found the MaxRequestsPerChild directive to be useful on an 
installation of mine that would lock up occasionally.  Your config shows 
the directive set to 4000 for the prefork MPM, but 0 (infinity) for the 
worker MPM.  Check which MPM you're using (execute "httpd -V") and ensure 
that MaxRequestsPerChild is taking effect.  The config file contains a set 
of MPM tuning directives, duplicated for prefork and worker, make sure the 
required configuration is actually active.

-- 
Paul

On Tuesday, September 10, 2013 11:36:59 PM UTC+1, Steve wrote:
>
> I see the 'server cache' information in the admin page.  It shows the 
> memory usage at around 150MB, so clearly 64MB was not enough. 
>  Unfortunately, the load average once again climbed over 100 and I had 
> close to 200 apache processes running.  It's baffling.  Can you guide me on 
> how to limit the number of apache processes and threads?  I've attached the 
> apache httpd.conf file.
>
> Thanks Christian.
>
> --steve
>

-- 
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ReviewBoard server tips over about twice a week

2013-09-10 Thread Christian Hammond 
On Tue, Sep 10, 2013 at 2:26 PM, Steve  wrote:

>
>
> On Tuesday, September 10, 2013 1:51:56 PM UTC-7, Christian Hammond wrote:
>>
>> Hi Steve,
>>
>> You shouldn't need to clear the cache. Something is wrong, though. We
>> know of servers reliably handling thousands if users every day.
>>
>> How much memory have you given memcached?
>>
>
>
> We're using the default settings, which allow 1024 connections and
> allocates 64MB of memory.  I didn't realize until just now that it was only
> using 64MB of memory.  I'm going to bump that to 2GB
>

That alone is going to do wonders. We make heavy use of the cache, and diff
viewing is expensive. 64MB will only hold so much of that (let alone the
other things we cache) at once.

The more memory you can give it, the better.

Also, can you confirm in the Admin UI -> General Settings page that it is
indeed using memcached? The Admin UI dashboard should also provide some
stats on memcached usage, if things are configured correctly. If you don't
see that, it's possible it's fallen back on local memory, which would be
bad.



>
>
>
>>
>> What are the specs of the server?
>>
>
>
> Intel(R) Xeon(R) CPU 5160 @ 3.00GHz with 16 GB RAM
>

Yeah, that should certainly be sufficient :)



>
>
>
>> What kind of repositories are you using?
>
>
>
> Perfore exclusively.
>

That'd be fine.


>
>
>
>>
>> Are there any gigantic, multi-megabyte files it's working with (diffs or
>> in the repository)?
>>
>
>
> I can't say off hand - I'd have to check that.  We do sometimes get people
> trying to post reviews with large numbers of files.  We've discouraged that
> but haven't stopped it completely.
>

Okay. The version of RB you're running should do a pretty good job of
handling larger files, and disabling expensive functionality for large
ones, so that may not be the cause. Still something to consider. Of course,
large changes will use more memory in memcached, so that could be part of
it.


>
>
>
>> Can you show the Apache limits in your config?
>>
>
>
> I will need your help finding that information.  Are you just asking for
> the main apache conf file? Or the rb conf file?
>

Just the main Apache config file.

One option, if memcached was not the culprit, is to limit the number of
servers/threads running at any particular time.



> Hopefully, bumping the memory on memcached will resolve this.
>

I expect it will. If it does not, do let me know, and we'll look into more.

If you find you need additional scalability as you grow, we have a beta of
our new product, the Review Board Power Pack (
http://www.reviewboard.org/powerpack/), which, amongst other features,
makes it easier to scale out Review Board across servers, in order to share
the load.

Christian



>
> --Steve
>
>
>
>>
>> Any other info you can provide about your setup would help.
>>
>> Thanks,
>>
>> Christian
>>
>>
>> On Tuesday, September 10, 2013, Steve wrote:
>>
>>> It just happened again.  I had to restart apache twice.  At one point I
>>> saw 135 apache processes running.  And heavy traffic that I think is coming
>>> from memcache, but I'm not sure.  Maybe I need to clear the cache?
>>>
>>> --Steve
>>>
>>>
>>> On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote:

 I know this is a shot in the dark, but I was hoping people with more
 apache experience and better knowledge of RB might be able to give me some
 pointers.  We're running RB 1.7.12 on a Centos6 box on a very beefy machine
 which is dedicated to RB.  We're using MySQL and perforce and have close to
 900 users.  About twice a week (sometimes more) the load average on the
 server spikes into the hundreds and the server becomes unresponsive.  The
 only recourse is to stop apache, wait for the load average to come down and
 then restart.  Sometimes when we restart, it spikes again immediately so we
 typically pause for a bit and let the load average get down under 1 before
 restarting.  When this happens, it looks like there a way more apache
 processes running than there should be.  We're using the default settings
 for apache on linux, which I believe limits the number of connections.  It
 also looks like there's a lot of traffic going from apache to memcache,
 which is running on the same machine.

 I know this is not much to go on, but I was hoping someone who has
 experience with fine-tuning RB on a linux server might give me some
 pointers as to where to begin.

 Thanks

 --Steve

  --
>>> Get the Review Board Power Pack at http://www.reviewboard.org/**
>>> powerpack/ 
>>> ---
>>> Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
>>> ---
>>> Happy user? Let us know at 
>>> http://www.reviewboard.org/**users/
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "reviewboard" group.
>>> To unsubscribe from this group and stop

Re: ReviewBoard server tips over about twice a week

2013-09-10 Thread Steve 
I see the 'server cache' information in the admin page.  It shows the 
memory usage at around 150MB, so clearly 64MB was not enough. 
 Unfortunately, the load average once again climbed over 100 and I had 
close to 200 apache processes running.  It's baffling.  Can you guide me on 
how to limit the number of apache processes and threads?  I've attached the 
apache httpd.conf file.

Thanks Christian.

--steve


On Tuesday, September 10, 2013 2:55:38 PM UTC-7, Christian Hammond wrote:
>
>
> On Tue, Sep 10, 2013 at 2:26 PM, Steve  >wrote:
>
>>
>>
>> On Tuesday, September 10, 2013 1:51:56 PM UTC-7, Christian Hammond wrote:
>>>
>>> Hi Steve,
>>>
>>> You shouldn't need to clear the cache. Something is wrong, though. We 
>>> know of servers reliably handling thousands if users every day.
>>>
>>> How much memory have you given memcached?
>>>
>>
>>
>> We're using the default settings, which allow 1024 connections and 
>> allocates 64MB of memory.  I didn't realize until just now that it was only 
>> using 64MB of memory.  I'm going to bump that to 2GB
>>
>
> That alone is going to do wonders. We make heavy use of the cache, and 
> diff viewing is expensive. 64MB will only hold so much of that (let alone 
> the other things we cache) at once.
>
> The more memory you can give it, the better.
>
> Also, can you confirm in the Admin UI -> General Settings page that it is 
> indeed using memcached? The Admin UI dashboard should also provide some 
> stats on memcached usage, if things are configured correctly. If you don't 
> see that, it's possible it's fallen back on local memory, which would be 
> bad.
>
>  
>
>>
>>  
>>
>>>
>>> What are the specs of the server?
>>>
>>
>>
>> Intel(R) Xeon(R) CPU 5160 @ 3.00GHz with 16 GB RAM
>>
>
> Yeah, that should certainly be sufficient :)
>
>  
>
>>
>>
>>
>>> What kind of repositories are you using?
>>
>>
>>
>> Perfore exclusively.
>>
>
> That'd be fine.
>  
>
>>
>>  
>>
>>>
>>> Are there any gigantic, multi-megabyte files it's working with (diffs or 
>>> in the repository)?
>>>
>>
>>
>> I can't say off hand - I'd have to check that.  We do sometimes get 
>> people trying to post reviews with large numbers of files.  We've 
>> discouraged that but haven't stopped it completely.
>>
>
> Okay. The version of RB you're running should do a pretty good job of 
> handling larger files, and disabling expensive functionality for large 
> ones, so that may not be the cause. Still something to consider. Of course, 
> large changes will use more memory in memcached, so that could be part of 
> it.
>  
>
>>
>>
>>
>>> Can you show the Apache limits in your config?
>>>
>>
>>
>> I will need your help finding that information.  Are you just asking for 
>> the main apache conf file? Or the rb conf file?
>>
>
> Just the main Apache config file.
>
> One option, if memcached was not the culprit, is to limit the number of 
> servers/threads running at any particular time.
>
>
>
>> Hopefully, bumping the memory on memcached will resolve this.
>>
>
> I expect it will. If it does not, do let me know, and we'll look into more.
>  
> If you find you need additional scalability as you grow, we have a beta of 
> our new product, the Review Board Power Pack (
> http://www.reviewboard.org/powerpack/), which, amongst other features, 
> makes it easier to scale out Review Board across servers, in order to share 
> the load.
>
> Christian
>
>
>
>>
>> --Steve
>>
>>  
>>
>>>
>>> Any other info you can provide about your setup would help.
>>>
>>> Thanks,
>>>
>>> Christian
>>>
>>>
>>> On Tuesday, September 10, 2013, Steve wrote:
>>>
 It just happened again.  I had to restart apache twice.  At one point I 
 saw 135 apache processes running.  And heavy traffic that I think is 
 coming 
 from memcache, but I'm not sure.  Maybe I need to clear the cache?

 --Steve


 On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote:
>
> I know this is a shot in the dark, but I was hoping people with more 
> apache experience and better knowledge of RB might be able to give me 
> some 
> pointers.  We're running RB 1.7.12 on a Centos6 box on a very beefy 
> machine 
> which is dedicated to RB.  We're using MySQL and perforce and have close 
> to 
> 900 users.  About twice a week (sometimes more) the load average on the 
> server spikes into the hundreds and the server becomes unresponsive.  The 
> only recourse is to stop apache, wait for the load average to come down 
> and 
> then restart.  Sometimes when we restart, it spikes again immediately so 
> we 
> typically pause for a bit and let the load average get down under 1 
> before 
> restarting.  When this happens, it looks like there a way more apache 
> processes running than there should be.  We're using the default settings 
> for apache on linux, which I believe limits the number of connections.  
> It 
> also looks like there's a lot of traffic going

Re: ReviewBoard server tips over about twice a week

2013-09-10 Thread Christian Hammond 
Hi Steve,

You shouldn't need to clear the cache. Something is wrong, though. We know
of servers reliably handling thousands if users every day.

How much memory have you given memcached?

What are the specs of the server?

What kind of repositories are you using?

Are there any gigantic, multi-megabyte files it's working with (diffs or in
the repository)?

Can you show the Apache limits in your config?

Any other info you can provide about your setup would help.

Thanks,

Christian


On Tuesday, September 10, 2013, Steve wrote:

> It just happened again.  I had to restart apache twice.  At one point I
> saw 135 apache processes running.  And heavy traffic that I think is coming
> from memcache, but I'm not sure.  Maybe I need to clear the cache?
>
> --Steve
>
>
> On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote:
>>
>> I know this is a shot in the dark, but I was hoping people with more
>> apache experience and better knowledge of RB might be able to give me some
>> pointers.  We're running RB 1.7.12 on a Centos6 box on a very beefy machine
>> which is dedicated to RB.  We're using MySQL and perforce and have close to
>> 900 users.  About twice a week (sometimes more) the load average on the
>> server spikes into the hundreds and the server becomes unresponsive.  The
>> only recourse is to stop apache, wait for the load average to come down and
>> then restart.  Sometimes when we restart, it spikes again immediately so we
>> typically pause for a bit and let the load average get down under 1 before
>> restarting.  When this happens, it looks like there a way more apache
>> processes running than there should be.  We're using the default settings
>> for apache on linux, which I believe limits the number of connections.  It
>> also looks like there's a lot of traffic going from apache to memcache,
>> which is running on the same machine.
>>
>> I know this is not much to go on, but I was hoping someone who has
>> experience with fine-tuning RB on a linux server might give me some
>> pointers as to where to begin.
>>
>> Thanks
>>
>> --Steve
>>
>>  --
> Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
> ---
> Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
> ---
> Happy user? Let us know at http://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "reviewboard" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com  'cvml', 'reviewboard%2bunsubscr...@googlegroups.com');>.
> For more options, visit https://groups.google.com/groups/opt_out.
>


-- 
-- 
Christian Hammond - chip...@chipx86.com
Review Board - http://www.reviewboard.org
Beanbag, Inc. - http://www.beanbaginc.com

-- 
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ReviewBoard server tips over about twice a week

2013-09-10 Thread Steve 


On Tuesday, September 10, 2013 1:51:56 PM UTC-7, Christian Hammond wrote:
>
> Hi Steve,
>
> You shouldn't need to clear the cache. Something is wrong, though. We know 
> of servers reliably handling thousands if users every day.
>
> How much memory have you given memcached?
>


We're using the default settings, which allow 1024 connections and 
allocates 64MB of memory.  I didn't realize until just now that it was only 
using 64MB of memory.  I'm going to bump that to 2GB

 

>
> What are the specs of the server?
>


Intel(R) Xeon(R) CPU 5160 @ 3.00GHz with 16 GB RAM



> What kind of repositories are you using?



Perfore exclusively.

 

>
> Are there any gigantic, multi-megabyte files it's working with (diffs or 
> in the repository)?
>


I can't say off hand - I'd have to check that.  We do sometimes get people 
trying to post reviews with large numbers of files.  We've discouraged that 
but haven't stopped it completely.



> Can you show the Apache limits in your config?
>


I will need your help finding that information.  Are you just asking for 
the main apache conf file? Or the rb conf file?

Hopefully, bumping the memory on memcached will resolve this.


--Steve

 

>
> Any other info you can provide about your setup would help.
>
> Thanks,
>
> Christian
>
>
> On Tuesday, September 10, 2013, Steve wrote:
>
>> It just happened again.  I had to restart apache twice.  At one point I 
>> saw 135 apache processes running.  And heavy traffic that I think is coming 
>> from memcache, but I'm not sure.  Maybe I need to clear the cache?
>>
>> --Steve
>>
>>
>> On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote:
>>>
>>> I know this is a shot in the dark, but I was hoping people with more 
>>> apache experience and better knowledge of RB might be able to give me some 
>>> pointers.  We're running RB 1.7.12 on a Centos6 box on a very beefy machine 
>>> which is dedicated to RB.  We're using MySQL and perforce and have close to 
>>> 900 users.  About twice a week (sometimes more) the load average on the 
>>> server spikes into the hundreds and the server becomes unresponsive.  The 
>>> only recourse is to stop apache, wait for the load average to come down and 
>>> then restart.  Sometimes when we restart, it spikes again immediately so we 
>>> typically pause for a bit and let the load average get down under 1 before 
>>> restarting.  When this happens, it looks like there a way more apache 
>>> processes running than there should be.  We're using the default settings 
>>> for apache on linux, which I believe limits the number of connections.  It 
>>> also looks like there's a lot of traffic going from apache to memcache, 
>>> which is running on the same machine.  
>>>
>>> I know this is not much to go on, but I was hoping someone who has 
>>> experience with fine-tuning RB on a linux server might give me some 
>>> pointers as to where to begin.
>>>
>>> Thanks
>>>
>>> --Steve
>>>
>>>  -- 
>> Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
>> ---
>> Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
>> ---
>> Happy user? Let us know at http://www.reviewboard.org/users/
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "reviewboard" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to reviewboard+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
> -- 
> -- 
> Christian Hammond - chi...@chipx86.com 
> Review Board - http://www.reviewboard.org
> Beanbag, Inc. - http://www.beanbaginc.com
>
>

-- 
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ReviewBoard server tips over about twice a week

2013-09-10 Thread Christian Hammond 
Hi Steve,

You shouldn't need to clear the cache. Something is wrong, though. We know
of servers reliably handling thousands if users every day.

How much memory have you given memcached?

What are the specs of the server?

What kind of repositories are you using?

Are there any gigantic, multi-megabyte files it's working with (diffs or in
the repository)?

Can you show the Apache limits in your config?

Any other info you can provide about your setup would help.

Thanks,

Christian


On Tuesday, September 10, 2013, Steve wrote:

> It just happened again.  I had to restart apache twice.  At one point I
> saw 135 apache processes running.  And heavy traffic that I think is coming
> from memcache, but I'm not sure.  Maybe I need to clear the cache?
>
> --Steve
>
>
> On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote:
>>
>> I know this is a shot in the dark, but I was hoping people with more
>> apache experience and better knowledge of RB might be able to give me some
>> pointers.  We're running RB 1.7.12 on a Centos6 box on a very beefy machine
>> which is dedicated to RB.  We're using MySQL and perforce and have close to
>> 900 users.  About twice a week (sometimes more) the load average on the
>> server spikes into the hundreds and the server becomes unresponsive.  The
>> only recourse is to stop apache, wait for the load average to come down and
>> then restart.  Sometimes when we restart, it spikes again immediately so we
>> typically pause for a bit and let the load average get down under 1 before
>> restarting.  When this happens, it looks like there a way more apache
>> processes running than there should be.  We're using the default settings
>> for apache on linux, which I believe limits the number of connections.  It
>> also looks like there's a lot of traffic going from apache to memcache,
>> which is running on the same machine.
>>
>> I know this is not much to go on, but I was hoping someone who has
>> experience with fine-tuning RB on a linux server might give me some
>> pointers as to where to begin.
>>
>> Thanks
>>
>> --Steve
>>
>>  --
> Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
> ---
> Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
> ---
> Happy user? Let us know at http://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups
> "reviewboard" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to reviewboard+unsubscr...@googlegroups.com  'cvml', 'reviewboard%2bunsubscr...@googlegroups.com');>.
> For more options, visit https://groups.google.com/groups/opt_out.
>


-- 
-- 
Christian Hammond - chip...@chipx86.com
Review Board - http://www.reviewboard.org
Beanbag, Inc. - http://www.beanbaginc.com

-- 
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: ReviewBoard server tips over about twice a week

2013-09-10 Thread Steve 
It just happened again.  I had to restart apache twice.  At one point I saw 
135 apache processes running.  And heavy traffic that I think is coming 
from memcache, but I'm not sure.  Maybe I need to clear the cache?

--Steve


On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote:
>
> I know this is a shot in the dark, but I was hoping people with more 
> apache experience and better knowledge of RB might be able to give me some 
> pointers.  We're running RB 1.7.12 on a Centos6 box on a very beefy machine 
> which is dedicated to RB.  We're using MySQL and perforce and have close to 
> 900 users.  About twice a week (sometimes more) the load average on the 
> server spikes into the hundreds and the server becomes unresponsive.  The 
> only recourse is to stop apache, wait for the load average to come down and 
> then restart.  Sometimes when we restart, it spikes again immediately so we 
> typically pause for a bit and let the load average get down under 1 before 
> restarting.  When this happens, it looks like there a way more apache 
> processes running than there should be.  We're using the default settings 
> for apache on linux, which I believe limits the number of connections.  It 
> also looks like there's a lot of traffic going from apache to memcache, 
> which is running on the same machine.  
>
> I know this is not much to go on, but I was hoping someone who has 
> experience with fine-tuning RB on a linux server might give me some 
> pointers as to where to begin.
>
> Thanks
>
> --Steve
>
>

-- 
Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
---
Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
---
Happy user? Let us know at http://www.reviewboard.org/users/
--- 
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to reviewboard+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.