We should figure out first why your backend server is being hit so hard. We have over a thousand people at VMware submitting good sized diffs and aren't hitting any of these issues. So a few more questions. Apologies if you've answered these before in other threads.
1) What MPM is your Apache using? 2) And is this fastcgi or mod_python? 3) Are you absolutely sure Review Board is using your memcached server? Go in the admin dashboard and click Server Cache, then copy and paste the info and paste it here. We might be able to do something smarter with deleted diffs, but that would require a good bit of work, and we'd have to think carefully about it. We talked about splitting up the reviews page, but decided it wouldn't make much sense due to how it's ordered. We came up with a method for collapsing, but I'm probably holding off until after 1.0 to finish this, because it needs a lot more thought and testing. Christian -- Christian Hammond - chip...@chipx86.com Review Board - http://www.review-board.org VMware, Inc. - http://www.vmware.com On Thu, Mar 19, 2009 at 11:39 AM, ciaomary <ciaom...@gmail.com> wrote: > > Thanks for your response. Please see comments inline below... > > On Mar 18, 11:35 am, Christian Hammond <chip...@chipx86.com> wrote: > > Not immediately, no. > > > > I can't even determine right now whether we can do anything about this. > > You're seeing crazy CPU loads that we're definitely not seeing, and most > of > > this has to do with the browser itself. We're loading the files > > asynchronously, but it appears that for whatever reason your browser is > > blocking until it's all loaded (if I'm understanding right). > > Just for clarity, the page can sometimes load partially, but since the > CPU is at 99% it's of little value. I don't get the CPU back until the > page is fully loaded. > > We now understand that our backend server is also getting maxed out to > the point of failure. One request of this page demands 800 M of > virtual memory from Apache! Once the request is fully loaded, 400 M of > that memory is freed again - but since we have 200 reviewboard users > this isn't very scalable. > > > I'm wondering > > if there's something with the configuration in your browsers where you > work > > that is causing some of this. > > > > I really don't think this is a browser configuration issue because > users across many different OS and browsers are seeing this. I see it > on default FireFox 3.0.7 and IE 7.0. Users have reported that Safari > works the best, but I have not confirmed. FireFox 2 and IE 6 users > have reported the same problem. > > > You say this is happening on several different browsers? In the case of > > Firefox, what extensions are loaded? > > > > Just the default FireFox 3.0.7 install. > > > We definitely would like to fix this. But we just don't know enough about > > what the problem is right now to determine if it's in our control. > > > > Apache must load up a huge amount of data in VM for these large diffs > (I believe this is especially true when diffs contain deleted files). > This isn't scalable across many users. Can the amount of data/ > filediffs/etc needed to load the page be scaled back further? Changing > the 'Paginate by' value from 20 to 3 has helped, but not enough and it > hasn't helped the review page obviously. > > Instead of showing all lines for deleted files can this be loaded > later only on request? > Can the 'View Reviews' page be broken up or paginated as well? > I think these two suggestions would help us. > > > > Christian > > > > -- > > Christian Hammond - chip...@chipx86.com > > Review Board -http://www.review-board.org > > VMware, Inc. -http://www.vmware.com > > > > > > > > On Wed, Mar 18, 2009 at 9:54 AM, ciaomary <ciaom...@gmail.com> wrote: > > > > > Can anything be done to fix the pegged CPU issue I've reported in my > > > prior email? > > > > > On Mar 17, 9:38 am, ciaomary <ciaom...@gmail.com> wrote: > > > > The downloadable diff for the review is 1.6MB. > > > > > > The review has 138 different files (7 pages). > > > > > > Page 1 loads in ~1 minute in FireFox. The twenty files are on average > > > > 38 K. (max=118K, min=2K). Here all files have edit changes (no > deleted > > > > files.) > > > > > > Page 3 loads in ~3 minutes in FireFox. The twenty files are on > average > > > > 25K . No major deviations from the average. Here 12 of the files have > > > > been deleted rather than just edited. > > > > > > There are hundreds of review comments on this review across many > users > > > > and files. > > > > > > During page load, the CPU is pegged and the browser is unresponsive > > > > for a good 2-3 minutes. This is the cause every time, very > > > > reproducable. If you are patient the page finally loads and the > system > > > > CPU returns to normal. > > > > > > FireBug shows an HTTPS GET request for each of the twenty files > during > > > > the slow down/cpu pegging. Each HTTPS GETS are taking longer on page > > > > 3, with some taking up to 2.5 seconds (others < 50ms). Since the is > > > > cpu is pegged the browser processing seems to be the cause of the > > > > delay however. > > > > > > On Mar 16, 6:31 pm, Christian Hammond <chip...@chipx86.com> wrote: > > > > > > > So it's really going to depend on the sizes of the individual files > > > > > themselves. If a file is by itself large enough to cause a couple > meg > > > diff > > > > > to be generated, there's nothing we can do really do, since that > boils > > > down > > > > > to too large a file to show in the browser. > > > > > > > In your page 3 example, how big are the files and the diffs in > > > particular? > > > > > > > The page itself loads quick enough, right? It's just the diff > > > fragments? > > > > > Which ones are you finding takes a long time and how big are those > > > changes? > > > > > > > Christian > > > > > > > -- > > > > > Christian Hammond - chip...@chipx86.com > > > > > Review Board -http://www.review-board.org > > > > > VMware, Inc. -http://www.vmware.com > > > > > > > On Mon, Mar 16, 2009 at 5:32 PM, mary <ciaom...@gmail.com> wrote: > > > > > > > > On really large diffs, performance is very slow and can crash the > > > > > > browser. > > > > > > > > We've picked up the recent 3/13/09 nightly which has helped > > > enormously > > > > > > (A BIG THANKS!) but that said, we're still finding pages that > take 3+ > > > > > > minutes to load. > > > > > > > > For example.... we have one review in particular that has File > > > Changes > > > > > > spread over 6 ReviewBoard pages. Page 3 takes on average 3+ > minutes > > > to > > > > > > load. It also readily crashes my browser - pegging my CPU and I > have > > > > > > to kill the session. (URL= > https://reviewboard/r/2808/diff/?page=3). > > > > > > This is in FireFox 3.0.7. Under IE the page doesn't load without > > > > > > running into script errors. Unfortunately, these large > ReviewBoard > > > > > > diffs are a typical use case at our company (typically when large > > > > > > merges are done) and breaking things across more than 10 reviews > jsut > > > > > > isn't plausible. I should mention that not only are the diff > > > fragments > > > > > > themselves large for these use cases but the number of reviewers > and > > > > > > comments gets to be very large too. We're maxing out what > ReviewBoard > > > > > > can handle on all fronts. > > > > > > > > Can further investigation be done into these performance issues > for > > > > > > us? I will supply further information as I collect it. > > > > > > > > Thanks, > > > > > > Mary- Hide quoted text - > > > > > > > - Show quoted text -- Hide quoted text - > > > > > > - Show quoted text -- Hide quoted text - > > > > - Show quoted text - > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "reviewboard" group. To post to this group, send email to reviewboard@googlegroups.com To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~----------~----~----~----~------~----~------~--~---
