On Tue, Sep 10, 2013 at 2:26 PM, Steve <seide.al...@gmail.com> wrote:
> > > On Tuesday, September 10, 2013 1:51:56 PM UTC-7, Christian Hammond wrote: >> >> Hi Steve, >> >> You shouldn't need to clear the cache. Something is wrong, though. We >> know of servers reliably handling thousands if users every day. >> >> How much memory have you given memcached? >> > > > We're using the default settings, which allow 1024 connections and > allocates 64MB of memory. I didn't realize until just now that it was only > using 64MB of memory. I'm going to bump that to 2GB > That alone is going to do wonders. We make heavy use of the cache, and diff viewing is expensive. 64MB will only hold so much of that (let alone the other things we cache) at once. The more memory you can give it, the better. Also, can you confirm in the Admin UI -> General Settings page that it is indeed using memcached? The Admin UI dashboard should also provide some stats on memcached usage, if things are configured correctly. If you don't see that, it's possible it's fallen back on local memory, which would be bad. > > > >> >> What are the specs of the server? >> > > > Intel(R) Xeon(R) CPU 5160 @ 3.00GHz with 16 GB RAM > Yeah, that should certainly be sufficient :) > > > >> What kind of repositories are you using? > > > > Perfore exclusively. > That'd be fine. > > > >> >> Are there any gigantic, multi-megabyte files it's working with (diffs or >> in the repository)? >> > > > I can't say off hand - I'd have to check that. We do sometimes get people > trying to post reviews with large numbers of files. We've discouraged that > but haven't stopped it completely. > Okay. The version of RB you're running should do a pretty good job of handling larger files, and disabling expensive functionality for large ones, so that may not be the cause. Still something to consider. Of course, large changes will use more memory in memcached, so that could be part of it. > > > >> Can you show the Apache limits in your config? >> > > > I will need your help finding that information. Are you just asking for > the main apache conf file? Or the rb conf file? > Just the main Apache config file. One option, if memcached was not the culprit, is to limit the number of servers/threads running at any particular time. > Hopefully, bumping the memory on memcached will resolve this. > I expect it will. If it does not, do let me know, and we'll look into more. If you find you need additional scalability as you grow, we have a beta of our new product, the Review Board Power Pack ( http://www.reviewboard.org/powerpack/), which, amongst other features, makes it easier to scale out Review Board across servers, in order to share the load. Christian > > --Steve > > > >> >> Any other info you can provide about your setup would help. >> >> Thanks, >> >> Christian >> >> >> On Tuesday, September 10, 2013, Steve wrote: >> >>> It just happened again. I had to restart apache twice. At one point I >>> saw 135 apache processes running. And heavy traffic that I think is coming >>> from memcache, but I'm not sure. Maybe I need to clear the cache? >>> >>> --Steve >>> >>> >>> On Tuesday, September 10, 2013 8:09:39 AM UTC-7, Steve wrote: >>>> >>>> I know this is a shot in the dark, but I was hoping people with more >>>> apache experience and better knowledge of RB might be able to give me some >>>> pointers. We're running RB 1.7.12 on a Centos6 box on a very beefy machine >>>> which is dedicated to RB. We're using MySQL and perforce and have close to >>>> 900 users. About twice a week (sometimes more) the load average on the >>>> server spikes into the hundreds and the server becomes unresponsive. The >>>> only recourse is to stop apache, wait for the load average to come down and >>>> then restart. Sometimes when we restart, it spikes again immediately so we >>>> typically pause for a bit and let the load average get down under 1 before >>>> restarting. When this happens, it looks like there a way more apache >>>> processes running than there should be. We're using the default settings >>>> for apache on linux, which I believe limits the number of connections. It >>>> also looks like there's a lot of traffic going from apache to memcache, >>>> which is running on the same machine. >>>> >>>> I know this is not much to go on, but I was hoping someone who has >>>> experience with fine-tuning RB on a linux server might give me some >>>> pointers as to where to begin. >>>> >>>> Thanks >>>> >>>> --Steve >>>> >>>> -- >>> Get the Review Board Power Pack at http://www.reviewboard.org/** >>> powerpack/ <http://www.reviewboard.org/powerpack/> >>> --- >>> Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ >>> --- >>> Happy user? Let us know at >>> http://www.reviewboard.org/**users/<http://www.reviewboard.org/users/> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "reviewboard" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to reviewboard+unsubscribe@**googlegroups.com. >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >> >> >> -- >> -- >> Christian Hammond - chi...@chipx86.com >> >> Review Board - http://www.reviewboard.org >> Beanbag, Inc. - http://www.beanbaginc.com >> >> -- > Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ > --- > Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ > --- > Happy user? Let us know at http://www.reviewboard.org/users/ > --- > You received this message because you are subscribed to the Google Groups > "reviewboard" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to reviewboard+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/ --- Sign up for Review Board hosting at RBCommons: https://rbcommons.com/ --- Happy user? Let us know at http://www.reviewboard.org/users/ --- You received this message because you are subscribed to the Google Groups "reviewboard" group. To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
