Thanks again. I'm really not that familiar with Apache 2 w/ regards to performance issues. At this time, I just have a more or less out-of- the-box Apache2 setup on CentOS 5. Any recommendations here would be hugely appreciated. Further answers inline below...
On Mar 19, 12:09 pm, Christian Hammond <chip...@chipx86.com> wrote: > We should figure out first why your backend server is being hit so hard. We > have over a thousand people at VMware submitting good sized diffs and aren't > hitting any of these issues. So a few more questions. Apologies if you've > answered these before in other threads. > > 1) What MPM is your Apache using? > I'm using the default for httpd 2.2.3 CentOS. I believe this is 'prefork'. > 2) And is this fastcgi or mod_python? > mod_python (mod_python-3.2.8-3.1) > 3) Are you absolutely sure Review Board is using your memcached server? Go > in the admin dashboard and click Server Cache, then copy and paste the info > and paste it here. Server Cache Cache backend: django.core.cache.backends.memcached Statistics 127.0.0.1:11211 Memory usage: 45.7 MB Keys in cache: 1767 of 1777 Cache hits: 3996 of 7464: 53% Cache misses: 3468 of 7464: 46% Cache evictions: 0 Cache traffic: 46.1 MB in, 103.8 MB out Uptime: 53009 > > We might be able to do something smarter with deleted diffs, but that would > require a good bit of work, and we'd have to think carefully about it. > > We talked about splitting up the reviews page, but decided it wouldn't make > much sense due to how it's ordered. We came up with a method for collapsing, > but I'm probably holding off until after 1.0 to finish this, because it > needs a lot more thought and testing. > > Christian > > -- > Christian Hammond - chip...@chipx86.com > Review Board -http://www.review-board.org > VMware, Inc. -http://www.vmware.com > > > > On Thu, Mar 19, 2009 at 11:39 AM, ciaomary <ciaom...@gmail.com> wrote: > > > Thanks for your response. Please see comments inline below... > > > On Mar 18, 11:35 am, Christian Hammond <chip...@chipx86.com> wrote: > > > Not immediately, no. > > > > I can't even determine right now whether we can do anything about this. > > > You're seeing crazy CPU loads that we're definitely not seeing, and most > > of > > > this has to do with the browser itself. We're loading the files > > > asynchronously, but it appears that for whatever reason your browser is > > > blocking until it's all loaded (if I'm understanding right). > > > Just for clarity, the page can sometimes load partially, but since the > > CPU is at 99% it's of little value. I don't get the CPU back until the > > page is fully loaded. > > > We now understand that our backend server is also getting maxed out to > > the point of failure. One request of this page demands 800 M of > > virtual memory from Apache! Once the request is fully loaded, 400 M of > > that memory is freed again - but since we have 200 reviewboard users > > this isn't very scalable. > > > > I'm wondering > > > if there's something with the configuration in your browsers where you > > work > > > that is causing some of this. > > > I really don't think this is a browser configuration issue because > > users across many different OS and browsers are seeing this. I see it > > on default FireFox 3.0.7 and IE 7.0. Users have reported that Safari > > works the best, but I have not confirmed. FireFox 2 and IE 6 users > > have reported the same problem. > > > > You say this is happening on several different browsers? In the case of > > > Firefox, what extensions are loaded? > > > Just the default FireFox 3.0.7 install. > > > > We definitely would like to fix this. But we just don't know enough about > > > what the problem is right now to determine if it's in our control. > > > Apache must load up a huge amount of data in VM for these large diffs > > (I believe this is especially true when diffs contain deleted files). > > This isn't scalable across many users. Can the amount of data/ > > filediffs/etc needed to load the page be scaled back further? Changing > > the 'Paginate by' value from 20 to 3 has helped, but not enough and it > > hasn't helped the review page obviously. > > > Instead of showing all lines for deleted files can this be loaded > > later only on request? > > Can the 'View Reviews' page be broken up or paginated as well? > > I think these two suggestions would help us. > > > > Christian > > > > -- > > > Christian Hammond - chip...@chipx86.com > > > Review Board -http://www.review-board.org > > > VMware, Inc. -http://www.vmware.com > > > > On Wed, Mar 18, 2009 at 9:54 AM, ciaomary <ciaom...@gmail.com> wrote: > > > > > Can anything be done to fix the pegged CPU issue I've reported in my > > > > prior email? > > > > > On Mar 17, 9:38 am, ciaomary <ciaom...@gmail.com> wrote: > > > > > The downloadable diff for the review is 1.6MB. > > > > > > The review has 138 different files (7 pages). > > > > > > Page 1 loads in ~1 minute in FireFox. The twenty files are on average > > > > > 38 K. (max=118K, min=2K). Here all files have edit changes (no > > deleted > > > > > files.) > > > > > > Page 3 loads in ~3 minutes in FireFox. The twenty files are on > > average > > > > > 25K . No major deviations from the average. Here 12 of the files have > > > > > been deleted rather than just edited. > > > > > > There are hundreds of review comments on this review across many > > users > > > > > and files. > > > > > > During page load, the CPU is pegged and the browser is unresponsive > > > > > for a good 2-3 minutes. This is the cause every time, very > > > > > reproducable. If you are patient the page finally loads and the > > system > > > > > CPU returns to normal. > > > > > > FireBug shows an HTTPS GET request for each of the twenty files > > during > > > > > the slow down/cpu pegging. Each HTTPS GETS are taking longer on page > > > > > 3, with some taking up to 2.5 seconds (others < 50ms). Since the is > > > > > cpu is pegged the browser processing seems to be the cause of the > > > > > delay however. > > > > > > On Mar 16, 6:31 pm, Christian Hammond <chip...@chipx86.com> wrote: > > > > > > > So it's really going to depend on the sizes of the individual files > > > > > > themselves. If a file is by itself large enough to cause a couple > > meg > > > > diff > > > > > > to be generated, there's nothing we can do really do, since that > > boils > > > > down > > > > > > to too large a file to show in the browser. > > > > > > > In your page 3 example, how big are the files and the diffs in > > > > particular? > > > > > > > The page itself loads quick enough, right? It's just the diff > > > > fragments? > > > > > > Which ones are you finding takes a long time and how big are those > > > > changes? > > > > > > > Christian > > > > > > > -- > > > > > > Christian Hammond - chip...@chipx86.com > > > > > > Review Board -http://www.review-board.org > > > > > > VMware, Inc. -http://www.vmware.com > > > > > > > On Mon, Mar 16, 2009 at 5:32 PM, mary <ciaom...@gmail.com> wrote: > > > > > > > > On really large diffs, performance is very slow and can crash the > > > > > > > browser. > > > > > > > > We've picked up the recent 3/13/09 nightly which has helped > > > > enormously > > > > > > > (A BIG THANKS!) but that said, we're still finding pages that > > take 3+ > > > > > > > minutes to load. > > > > > > > > For example.... we have one review in particular that has File > > > > Changes > > > > > > > spread over 6 ReviewBoard pages. Page 3 takes on average 3+ > > minutes > > > > to > > > > > > > load. It also readily crashes my browser - pegging my CPU and I > > have > > > > > > > to kill the session. (URL= > >https://reviewboard/r/2808/diff/?page=3). > > > > > > > This is in FireFox 3.0.7. Under IE the page doesn't load without > > > > > > > running into script errors. Unfortunately, these large > > ReviewBoard > > > > > > > diffs are a typical use case at our company (typically when large > > > > > > > merges are done) and breaking things across more than 10 reviews > > jsut > > > > > > > isn't plausible. I should mention that not only are the diff > > > > fragments > > > > > > > themselves large for these use cases but the number of reviewers > > and > > > > > > > comments gets to be very large too. We're maxing out what > > ReviewBoard > > > > > > > can handle on all fronts. > > > > > > > > Can further investigation be done into these performance issues > > for > > > > > > > us? I will supply further information as I collect it. > > > > > > > > Thanks, > > > > > > > Mary- Hide quoted text - > > > > > > > - Show quoted text -- Hide quoted text - > > > > > > - Show quoted text -- Hide quoted text - > > > > - Show quoted text -- Hide quoted text - > > - Show quoted text - --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "reviewboard" group. To post to this group, send email to reviewboard@googlegroups.com To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~----------~----~----~----~------~----~------~--~---
