I've made some optimizations today that will be going into beta 1.

I created a change with some deleted files and put about 15 comment flags on
one of the files. I'm sure this isn't anywhere near the amount on your
reviews, but it was a good sample to start with.

The comment loading code took, on average, 1 second for that particular file
(which was a file deletion, so 100% changes and expanded).

After my changes, that went down to ~155 msec.

I'll be curious to see if you guys will notice a measurable impact on
performance in beta 1.

Christian

-- 
Christian Hammond - chip...@chipx86.com
Review Board - http://www.review-board.org
VMware, Inc. - http://www.vmware.com


On Fri, Mar 27, 2009 at 2:39 PM, Mary B <ciaom...@gmail.com> wrote:

> That's great to hear, thanks! I will stay tuned...
>
>
> On Fri, Mar 27, 2009 at 2:27 PM, Christian Hammond <chip...@chipx86.com>wrote:
>
>> Okay, I have a couple ideas now where the problem lies.
>>
>> I want to give you a heads up that we'll be releasing beta 1 soon, and it
>> won't include a fix for this. This is going to be a much larger problem to
>> tackle and I don't want to hold up this release (since it has many very
>> important fixes in it). I hope to get to this one for beta 2.
>>
>> Christian
>>
>> --
>> Christian Hammond - chip...@chipx86.com
>> Review Board - http://www.review-board.org
>> VMware, Inc. - http://www.vmware.com
>>
>>
>>   On Fri, Mar 27, 2009 at 10:27 AM, ciaomary <ciaom...@gmail.com> wrote:
>>
>>>
>>> We typically only see perf problems on reviews with large diffs and
>>> large number of reviews. One that has given us a lot of trouble has a
>>> downloadable diff of 1.6MB, 138 different files (with an average file
>>> size of ~30K), and hundreds+ review comments. (Bringing up diffs for
>>> deleted files give the biggest load and slow down.)
>>>
>>> Under Admin Settings I've changed the "Paginate by" value from 20 down
>>> to just 3 which has helped a lot. It is set to the default 5 "Lines of
>>> Context". Also increasing the RAM on our server from 4G to 16G has
>>> helped - RB takes a lot of server memory under this kind of load!
>>> After all this we're still seeing these big reviews taking 45-60
>>> seconds to load, pegging local cpu during that time.
>>>
>>> I suspect you could reproduce this by setting the "Paginate by" value
>>> to something very high. (Try 55 for your 55 page diff.) I believe the
>>> problem is only on reviews with lots of comments as well. Our use case
>>> has been that if there is a really large diff, there are always a lot
>>> of review comments as well.
>>>
>>> Thanks!
>>>
>>> On Mar 27, 1:38 am, Christian Hammond <chip...@chipx86.com> wrote:
>>> > So I just looked at a diff on our server that spans 55 pages, and as
>>> far as
>>> > loading diffs goes, it's not at all slow. How many pages did you say
>>> yours
>>> > were on average?
>>> >
>>> > When looking at diffs, do you have it set to expand the entire file, or
>>> show
>>> > the collapsed version?
>>> >
>>> > Are you seeing slowdown on large diffs that don't have any reviews, or
>>> is it
>>> > just when there's a lot of reviews for that revision of the diff?
>>> >
>>> > Christian
>>> >
>>> > --
>>>  > Christian Hammond - chip...@chipx86.com
>>> > Review Board -http://www.review-board.org
>>> > VMware, Inc. -http://www.vmware.com
>>> >
>>> > On Mon, Mar 23, 2009 at 5:02 PM, Christian Hammond <
>>> chip...@chipx86.com>wrote:
>>> >
>>> >
>>> >
>>> > > I'm still looking into it. So far I have not seen the slowdown you
>>> have
>>> > > described, but will continue to try to figure out what's going on. I
>>> know
>>> > > this is inconvenient. We just haven't seen usage like this yet.
>>> >
>>> > > Christian
>>> >
>>> > > --
>>> > > Christian Hammond - chip...@chipx86.com
>>> > > Review Board -http://www.review-board.org
>>> > > VMware, Inc. -http://www.vmware.com
>>> >
>>>  > > On Mon, Mar 23, 2009 at 1:19 PM, ciaomary <ciaom...@gmail.com>
>>> wrote:
>>> >
>>> > >> Any follow up on this? Any configuration guidance?
>>> >
>>> > >> We've moved to a 16 G + 4 cpu system and still have unacceptable
>>> > >> response times (~30 seconds) for really large diffs with hundreds of
>>> > >> review comments.
>>> >
>>> > >> If you can't reproduce this, I recommend you set the number of diff
>>> > >> files per page from 20 to something really high (like 100) and then
>>> > >> post a review with the same number of files deleted. For us, this
>>> puts
>>> > >> a huge load on the local browser and the server VM during load.
>>> >
>>> > >> On Mar 19, 12:41 pm, ciaomary <ciaom...@gmail.com> wrote:
>>> > >> > Thanks again. I'm really not that familiar with Apache 2 w/
>>> regards to
>>> > >> > performance issues. At this time, I just have a more or less
>>> out-of-
>>> > >> > the-box Apache2 setup on CentOS 5. Any recommendations here would
>>> be
>>> > >> > hugely appreciated. Further answers inline below...
>>> >
>>> > >> > On Mar 19, 12:09 pm, Christian Hammond <chip...@chipx86.com>
>>> wrote:
>>> >
>>> > >> > > We should figure out first why your backend server is being hit
>>> so
>>> > >> hard. We
>>> > >> > > have over a thousand people at VMware submitting good sized
>>> diffs and
>>> > >> aren't
>>> > >> > > hitting any of these issues. So a few more questions. Apologies
>>> if
>>> > >> you've
>>> > >> > > answered these before in other threads.
>>> >
>>> > >> > > 1) What MPM is your Apache using?
>>> >
>>> > >> > I'm using the default for httpd 2.2.3 CentOS. I believe this is
>>> > >> > 'prefork'.
>>> >
>>> > >> > > 2) And is this fastcgi or mod_python?
>>> >
>>> > >> > mod_python (mod_python-3.2.8-3.1)
>>> >
>>> > >> > > 3) Are you absolutely sure Review Board is using your memcached
>>> > >> server? Go
>>> > >> > > in the admin dashboard and click Server Cache, then copy and
>>> paste the
>>> > >> info
>>> > >> > > and paste it here.
>>> >
>>> > >> > Server Cache
>>> > >> > Cache backend: django.core.cache.backends.memcached
>>> > >> > Statistics
>>> > >> > 127.0.0.1:11211 Memory usage:   45.7 MB
>>> > >> > Keys in cache:  1767 of 1777
>>> > >> > Cache hits:     3996 of 7464: 53%
>>> > >> > Cache misses:   3468 of 7464: 46%
>>> > >> > Cache evictions:        0
>>> > >> > Cache traffic:  46.1 MB in, 103.8 MB out
>>> > >> > Uptime:         53009
>>> >
>>> > >> > > We might be able to do something smarter with deleted diffs, but
>>> that
>>> > >> would
>>> > >> > > require a good bit of work, and we'd have to think carefully
>>> about it.
>>> >
>>> > >> > > We talked about splitting up the reviews page, but decided it
>>> wouldn't
>>> > >> make
>>> > >> > > much sense due to how it's ordered. We came up with a method for
>>> > >> collapsing,
>>> > >> > > but  I'm probably holding off until after 1.0 to finish this,
>>> because
>>> > >> it
>>> > >> > > needs a lot more thought and testing.
>>> >
>>> > >> > > Christian
>>> >
>>> > >> > > --
>>> > >> > > Christian Hammond - chip...@chipx86.com
>>> > >> > > Review Board -http://www.review-board.org
>>> > >> > > VMware, Inc. -http://www.vmware.com
>>> >
>>> > >> > > On Thu, Mar 19, 2009 at 11:39 AM, ciaomary <ciaom...@gmail.com>
>>> > >> wrote:
>>> >
>>> > >> > > > Thanks for your response. Please see comments inline below...
>>> >
>>> > >> > > > On Mar 18, 11:35 am, Christian Hammond <chip...@chipx86.com>
>>> wrote:
>>> > >> > > > > Not immediately, no.
>>> >
>>> > >> > > > > I can't even determine right now whether we can do anything
>>> about
>>> > >> this.
>>> > >> > > > > You're seeing crazy CPU loads that we're definitely not
>>> seeing,
>>> > >> and most
>>> > >> > > > of
>>> > >> > > > > this has to do with the browser itself. We're loading the
>>> files
>>> > >> > > > > asynchronously, but it appears that for whatever reason your
>>> > >> browser is
>>> > >> > > > > blocking until it's all loaded (if I'm understanding right).
>>> >
>>> > >> > > > Just for clarity, the page can sometimes load partially, but
>>> since
>>> > >> the
>>> > >> > > > CPU is at 99% it's of little value. I don't get the CPU back
>>> until
>>> > >> the
>>> > >> > > > page is fully loaded.
>>> >
>>> > >> > > > We now understand that our backend server is also getting
>>> maxed out
>>> > >> to
>>> > >> > > > the point of failure. One request of this page demands 800 M
>>> of
>>> > >> > > > virtual memory from Apache! Once the request is fully loaded,
>>> 400 M
>>> > >> of
>>> > >> > > > that memory is freed again - but since we have 200 reviewboard
>>> users
>>> > >> > > > this isn't very scalable.
>>> >
>>> > >> > > > > I'm wondering
>>> > >> > > > > if there's something with the configuration in your browsers
>>> where
>>> > >> you
>>> > >> > > > work
>>> > >> > > > > that is causing some of this.
>>> >
>>> > >> > > > I really don't think this is a browser configuration issue
>>> because
>>> > >> > > > users across many different OS and browsers are seeing this. I
>>> see
>>> > >> it
>>> > >> > > > on default FireFox 3.0.7 and IE 7.0. Users have reported that
>>> Safari
>>> > >> > > > works the best, but I have not confirmed. FireFox 2 and IE 6
>>> users
>>> > >> > > > have reported the same problem.
>>> >
>>> > >> > > > > You say this is happening on several different browsers? In
>>> the
>>> > >> case of
>>> > >> > > > > Firefox, what extensions are loaded?
>>> >
>>> > >> > > > Just the default FireFox 3.0.7 install.
>>> >
>>> > >> > > > > We definitely would like to fix this. But we just don't know
>>> > >> enough about
>>> > >> > > > > what the problem is right now to determine if it's in our
>>> control.
>>> >
>>> > >> > > > Apache must load up a huge amount of data in VM for these
>>> large
>>> > >> diffs
>>> > >> > > > (I believe this is especially true when diffs contain deleted
>>> > >> files).
>>> > >> > > > This isn't scalable across many users. Can the amount of data/
>>> > >> > > > filediffs/etc needed to load the page be scaled back further?
>>> > >> Changing
>>> > >> > > > the 'Paginate by' value from 20 to 3 has helped, but not
>>> enough and
>>> > >> it
>>> > >> > > > hasn't helped the review page obviously.
>>> >
>>> > >> > > > Instead of showing all lines for deleted files can this be
>>> loaded
>>> > >> > > > later only on request?
>>> > >> > > > Can the 'View Reviews' page be broken up or paginated as well?
>>> > >> > > > I think these two suggestions would help us.
>>> >
>>> > >> > > > > Christian
>>> >
>>> > >> > > > > --
>>> > >> > > > > Christian Hammond - chip...@chipx86.com
>>> > >> > > > > Review Board -http://www.review-board.org
>>> > >> > > > > VMware, Inc. -http://www.vmware.com
>>> >
>>> > >> > > > > On Wed, Mar 18, 2009 at 9:54 AM, ciaomary <
>>> ciaom...@gmail.com>
>>> > >> wrote:
>>> >
>>> > >> > > > > > Can anything be done to fix the pegged CPU issue I've
>>> reported
>>> > >> in my
>>> > >> > > > > > prior email?
>>> >
>>> > >> > > > > > On Mar 17, 9:38 am, ciaomary <ciaom...@gmail.com> wrote:
>>> > >> > > > > > > The downloadable diff for the review is 1.6MB.
>>> >
>>> > >> > > > > > > The review has 138 different files (7 pages).
>>> >
>>> > >> > > > > > > Page 1 loads in ~1 minute in FireFox. The twenty files
>>> are on
>>> > >> average
>>> > >> > > > > > > 38 K. (max=118K, min=2K). Here all files have edit
>>> changes (no
>>> > >> > > > deleted
>>> > >> > > > > > > files.)
>>> >
>>> > >> > > > > > > Page 3 loads in ~3 minutes in FireFox. The twenty files
>>> are on
>>> > >> > > > average
>>> > >> > > > > > > 25K . No major deviations from the average. Here 12 of
>>> the
>>> > >> files have
>>> > >> > > > > > > been deleted rather than just edited.
>>> >
>>> > >> > > > > > > There are hundreds of review comments on this review
>>> across
>>> > >> many
>>> > >> > > > users
>>> > >> > > > > > > and files.
>>> >
>>> > >> > > > > > > During page load, the CPU is pegged and the browser is
>>> > >> unresponsive
>>> > >> > > > > > > for a good 2-3 minutes. This is the cause every time,
>>> very
>>> > >> > > > > > > reproducable. If you are patient the page finally loads
>>> and
>>> > >> the
>>> > >> > > > system
>>> > >> > > > > > > CPU returns to normal.
>>> >
>>> > >> > > > > > > FireBug shows an HTTPS GET request for each of the
>>> twenty
>>> > >> files
>>> > >> > > > during
>>> > >> > > > > > > the slow down/cpu pegging. Each HTTPS GETS are taking
>>> longer
>>> > >> on page
>>> > >> > > > > > > 3, with some taking up to 2.5 seconds (others < 50ms).
>>> Since
>>> > >> the is
>>> > >> > > > > > > cpu is pegged the browser processing seems to be the
>>> cause of
>>> > >> the
>>> > >> > > > > > > delay however.
>>> >
>>> > >> > > > > > > On Mar 16, 6:31 pm, Christian Hammond <
>>> chip...@chipx86.com>
>>> > >> wrote:
>>> >
>>> > >> > > > > > > > So it's really going to depend on the sizes of the
>>> > >> individual files
>>> > >> > > > > > > > themselves. If a file is by itself large enough to
>>> cause a
>>> > >> couple
>>> > >> > > > meg
>>> > >> > > > > > diff
>>> > >> > > > > > > > to be generated, there's nothing we can do really do,
>>> since
>>> > >> that
>>> > >> > > > boils
>>> > >> > > > > > down
>>> > >> > > > > > > > to too large a file to show in the browser.
>>> >
>>> > >> > > > > > > > In your page 3 example, how big are the files and the
>>> diffs
>>> > >> in
>>> > >> > > > > > particular?
>>> >
>>> > >> > > > > > > > The page itself loads quick enough, right? It's just
>>> the
>>> > >> diff
>>> > >> > > > > > fragments?
>>> > >> > > > > > > > Which ones are you finding takes a long time and how
>>> big are
>>> > >> those
>>> > >> > > > > > changes?
>>> >
>>> > >> > > > > > > > Christian
>>> >
>>> > >> > > > > > > > --
>>> > >> > > > > > > > Christian Hammond - chip...@chipx86.com
>>> > >> > > > > > > > Review Board -http://www.review-board.org
>>> > >> > > > > > > > VMware, Inc. -http://www.vmware.com
>>> >
>>> > >> > > > > > > > On Mon, Mar 16, 2009 at 5:32 PM, mary <
>>> ciaom...@gmail.com>
>>> > >> wrote:
>>> >
>>> > >> > > > > > > > > On really large diffs, performance is very slow and
>>> can
>>> > >> crash the
>>> >
>>> > ...
>>> >
>>> > read more ยป- Hide quoted text -
>>>  >
>>> > - Show quoted text -
>>>
>>>
>> >>
>>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To post to this group, send email to reviewboard@googlegroups.com
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply via email to