So I just looked at a diff on our server that spans 55 pages, and as far as
loading diffs goes, it's not at all slow. How many pages did you say yours
were on average?

When looking at diffs, do you have it set to expand the entire file, or show
the collapsed version?

Are you seeing slowdown on large diffs that don't have any reviews, or is it
just when there's a lot of reviews for that revision of the diff?

Christian

-- 
Christian Hammond - chip...@chipx86.com
Review Board - http://www.review-board.org
VMware, Inc. - http://www.vmware.com


On Mon, Mar 23, 2009 at 5:02 PM, Christian Hammond <chip...@chipx86.com>wrote:

> I'm still looking into it. So far I have not seen the slowdown you have
> described, but will continue to try to figure out what's going on. I know
> this is inconvenient. We just haven't seen usage like this yet.
>
> Christian
>
> --
> Christian Hammond - chip...@chipx86.com
> Review Board - http://www.review-board.org
> VMware, Inc. - http://www.vmware.com
>
>
> On Mon, Mar 23, 2009 at 1:19 PM, ciaomary <ciaom...@gmail.com> wrote:
>
>>
>> Any follow up on this? Any configuration guidance?
>>
>> We've moved to a 16 G + 4 cpu system and still have unacceptable
>> response times (~30 seconds) for really large diffs with hundreds of
>> review comments.
>>
>> If you can't reproduce this, I recommend you set the number of diff
>> files per page from 20 to something really high (like 100) and then
>> post a review with the same number of files deleted. For us, this puts
>> a huge load on the local browser and the server VM during load.
>>
>> On Mar 19, 12:41 pm, ciaomary <ciaom...@gmail.com> wrote:
>> > Thanks again. I'm really not that familiar with Apache 2 w/ regards to
>> > performance issues. At this time, I just have a more or less out-of-
>> > the-box Apache2 setup on CentOS 5. Any recommendations here would be
>> > hugely appreciated. Further answers inline below...
>> >
>> > On Mar 19, 12:09 pm, Christian Hammond <chip...@chipx86.com> wrote:
>> >
>> > > We should figure out first why your backend server is being hit so
>> hard. We
>> > > have over a thousand people at VMware submitting good sized diffs and
>> aren't
>> > > hitting any of these issues. So a few more questions. Apologies if
>> you've
>> > > answered these before in other threads.
>> >
>> > > 1) What MPM is your Apache using?
>> >
>> > I'm using the default for httpd 2.2.3 CentOS. I believe this is
>> > 'prefork'.
>> >
>> > > 2) And is this fastcgi or mod_python?
>> >
>> > mod_python (mod_python-3.2.8-3.1)
>> >
>> > > 3) Are you absolutely sure Review Board is using your memcached
>> server? Go
>> > > in the admin dashboard and click Server Cache, then copy and paste the
>> info
>> > > and paste it here.
>> >
>> > Server Cache
>> > Cache backend: django.core.cache.backends.memcached
>> > Statistics
>> > 127.0.0.1:11211 Memory usage:   45.7 MB
>> > Keys in cache:  1767 of 1777
>> > Cache hits:     3996 of 7464: 53%
>> > Cache misses:   3468 of 7464: 46%
>> > Cache evictions:        0
>> > Cache traffic:  46.1 MB in, 103.8 MB out
>> > Uptime:         53009
>> >
>> >
>> >
>> >
>> >
>> > > We might be able to do something smarter with deleted diffs, but that
>> would
>> > > require a good bit of work, and we'd have to think carefully about it.
>> >
>> > > We talked about splitting up the reviews page, but decided it wouldn't
>> make
>> > > much sense due to how it's ordered. We came up with a method for
>> collapsing,
>> > > but  I'm probably holding off until after 1.0 to finish this, because
>> it
>> > > needs a lot more thought and testing.
>> >
>> > > Christian
>> >
>> > > --
>> > > Christian Hammond - chip...@chipx86.com
>> > > Review Board -http://www.review-board.org
>> > > VMware, Inc. -http://www.vmware.com
>> >
>> > > On Thu, Mar 19, 2009 at 11:39 AM, ciaomary <ciaom...@gmail.com>
>> wrote:
>> >
>> > > > Thanks for your response. Please see comments inline below...
>> >
>> > > > On Mar 18, 11:35 am, Christian Hammond <chip...@chipx86.com> wrote:
>> > > > > Not immediately, no.
>> >
>> > > > > I can't even determine right now whether we can do anything about
>> this.
>> > > > > You're seeing crazy CPU loads that we're definitely not seeing,
>> and most
>> > > > of
>> > > > > this has to do with the browser itself. We're loading the files
>> > > > > asynchronously, but it appears that for whatever reason your
>> browser is
>> > > > > blocking until it's all loaded (if I'm understanding right).
>> >
>> > > > Just for clarity, the page can sometimes load partially, but since
>> the
>> > > > CPU is at 99% it's of little value. I don't get the CPU back until
>> the
>> > > > page is fully loaded.
>> >
>> > > > We now understand that our backend server is also getting maxed out
>> to
>> > > > the point of failure. One request of this page demands 800 M of
>> > > > virtual memory from Apache! Once the request is fully loaded, 400 M
>> of
>> > > > that memory is freed again - but since we have 200 reviewboard users
>> > > > this isn't very scalable.
>> >
>> > > > > I'm wondering
>> > > > > if there's something with the configuration in your browsers where
>> you
>> > > > work
>> > > > > that is causing some of this.
>> >
>> > > > I really don't think this is a browser configuration issue because
>> > > > users across many different OS and browsers are seeing this. I see
>> it
>> > > > on default FireFox 3.0.7 and IE 7.0. Users have reported that Safari
>> > > > works the best, but I have not confirmed. FireFox 2 and IE 6 users
>> > > > have reported the same problem.
>> >
>> > > > > You say this is happening on several different browsers? In the
>> case of
>> > > > > Firefox, what extensions are loaded?
>> >
>> > > > Just the default FireFox 3.0.7 install.
>> >
>> > > > > We definitely would like to fix this. But we just don't know
>> enough about
>> > > > > what the problem is right now to determine if it's in our control.
>> >
>> > > > Apache must load up a huge amount of data in VM for these large
>> diffs
>> > > > (I believe this is especially true when diffs contain deleted
>> files).
>> > > > This isn't scalable across many users. Can the amount of data/
>> > > > filediffs/etc needed to load the page be scaled back further?
>> Changing
>> > > > the 'Paginate by' value from 20 to 3 has helped, but not enough and
>> it
>> > > > hasn't helped the review page obviously.
>> >
>> > > > Instead of showing all lines for deleted files can this be loaded
>> > > > later only on request?
>> > > > Can the 'View Reviews' page be broken up or paginated as well?
>> > > > I think these two suggestions would help us.
>> >
>> > > > > Christian
>> >
>> > > > > --
>> > > > > Christian Hammond - chip...@chipx86.com
>> > > > > Review Board -http://www.review-board.org
>> > > > > VMware, Inc. -http://www.vmware.com
>> >
>> > > > > On Wed, Mar 18, 2009 at 9:54 AM, ciaomary <ciaom...@gmail.com>
>> wrote:
>> >
>> > > > > > Can anything be done to fix the pegged CPU issue I've reported
>> in my
>> > > > > > prior email?
>> >
>> > > > > > On Mar 17, 9:38 am, ciaomary <ciaom...@gmail.com> wrote:
>> > > > > > > The downloadable diff for the review is 1.6MB.
>> >
>> > > > > > > The review has 138 different files (7 pages).
>> >
>> > > > > > > Page 1 loads in ~1 minute in FireFox. The twenty files are on
>> average
>> > > > > > > 38 K. (max=118K, min=2K). Here all files have edit changes (no
>> > > > deleted
>> > > > > > > files.)
>> >
>> > > > > > > Page 3 loads in ~3 minutes in FireFox. The twenty files are on
>> > > > average
>> > > > > > > 25K . No major deviations from the average. Here 12 of the
>> files have
>> > > > > > > been deleted rather than just edited.
>> >
>> > > > > > > There are hundreds of review comments on this review across
>> many
>> > > > users
>> > > > > > > and files.
>> >
>> > > > > > > During page load, the CPU is pegged and the browser is
>> unresponsive
>> > > > > > > for a good 2-3 minutes. This is the cause every time, very
>> > > > > > > reproducable. If you are patient the page finally loads and
>> the
>> > > > system
>> > > > > > > CPU returns to normal.
>> >
>> > > > > > > FireBug shows an HTTPS GET request for each of the twenty
>> files
>> > > > during
>> > > > > > > the slow down/cpu pegging. Each HTTPS GETS are taking longer
>> on page
>> > > > > > > 3, with some taking up to 2.5 seconds (others < 50ms). Since
>> the is
>> > > > > > > cpu is pegged the browser processing seems to be the cause of
>> the
>> > > > > > > delay however.
>> >
>> > > > > > > On Mar 16, 6:31 pm, Christian Hammond <chip...@chipx86.com>
>> wrote:
>> >
>> > > > > > > > So it's really going to depend on the sizes of the
>> individual files
>> > > > > > > > themselves. If a file is by itself large enough to cause a
>> couple
>> > > > meg
>> > > > > > diff
>> > > > > > > > to be generated, there's nothing we can do really do, since
>> that
>> > > > boils
>> > > > > > down
>> > > > > > > > to too large a file to show in the browser.
>> >
>> > > > > > > > In your page 3 example, how big are the files and the diffs
>> in
>> > > > > > particular?
>> >
>> > > > > > > > The page itself loads quick enough, right? It's just the
>> diff
>> > > > > > fragments?
>> > > > > > > > Which ones are you finding takes a long time and how big are
>> those
>> > > > > > changes?
>> >
>> > > > > > > > Christian
>> >
>> > > > > > > > --
>> > > > > > > > Christian Hammond - chip...@chipx86.com
>> > > > > > > > Review Board -http://www.review-board.org
>> > > > > > > > VMware, Inc. -http://www.vmware.com
>> >
>> > > > > > > > On Mon, Mar 16, 2009 at 5:32 PM, mary <ciaom...@gmail.com>
>> wrote:
>> >
>> > > > > > > > > On really large diffs, performance is very slow and can
>> crash the
>> > > > > > > > > browser.
>> >
>> > > > > > > > > We've picked up the recent 3/13/09 nightly which has
>> helped
>> > > > > > enormously
>> > > > > > > > > (A BIG THANKS!) but that said, we're still finding pages
>> that
>> > > > take 3+
>> > > > > > > > > minutes to load.
>> >
>> > > > > > > > > For example.... we have one review in particular that has
>> File
>> > > > > > Changes
>> > > > > > > > > spread over 6 ReviewBoard pages. Page 3 takes on average
>> 3+
>> > > > minutes
>> > > > > > to
>> > > > > > > > > load. It also readily crashes my browser - pegging my CPU
>> and I
>> > > > have
>> > > > > > > > > to kill the session. (URL=
>> > > >https://reviewboard/r/2808/diff/?page=3).
>> > > > > > > > > This is in FireFox 3.0.7. Under IE the page doesn't load
>> without
>> > > > > > > > > running into script errors. Unfortunately, these large
>> > > > ReviewBoard
>> > > > > > > > > diffs are a typical use case at our company (typically
>> when large
>> > > > > > > > > merges are done) and breaking things across more than 10
>> reviews
>> > > > jsut
>> > > > > > > > > isn't plausible. I should mention that not only are the
>> diff
>> > > > > > fragments
>> > > > > > > > > themselves large for these use cases but the number of
>> reviewers
>> > > > and
>> > > > > > > > > comments gets to be very large too. We're maxing out what
>> > > > ReviewBoard
>> > > > > > > > > can handle on all fronts.
>> >
>> > > > > > > > > Can further investigation be done into these performance
>> issues
>> > > > for
>> > > > > > > > > us? I will supply further information as I collect it.
>> >
>> > > > > > > > > Thanks,
>> > > > > > > > > Mary- Hide quoted text -
>> >
>> > > > > > > > - Show quoted text -- Hide quoted text -
>> >
>> > > > > > > - Show quoted text -- Hide quoted text -
>> >
>> > > > > - Show quoted text -- Hide quoted text -
>> >
>> > > - Show quoted text -- Hide quoted text -
>> >
>> > - Show quoted text -
>> >>
>>
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To post to this group, send email to reviewboard@googlegroups.com
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply via email to