Thanks, it's great to hear this feedback :) I was really hoping those
optimizations would improve things that noticeably.

We'll continue to profile the review page and try to come up with
improvements. I'm actually hoping we're "good enough" for 1.0, since we need
to freeze things and concentrate on bug fixes. If we can find any easy fixes
for improving speed, we'll of course do that, but if we were to implement
review collapsing (as discussed earlier), then we'd need to hold off until a
1.1 or something.

Anyhow, I'll keep looking into ways of making this faster. Thanks again!

Christian

-- 
Christian Hammond - chip...@chipx86.com
Review Board - http://www.review-board.org
VMware, Inc. - http://www.vmware.com


On Mon, Mar 30, 2009 at 2:12 PM, ciaomary <ciaom...@gmail.com> wrote:

>
> Wow, great work! Beta has been a big improvment help for us! The diff
> viewer looks to be about 5 to 10 times faster in page load based on
> preliminary testing!
>
> The only remaining perf issue we have left is on the page load of the
> review page but I believe this is limited to when there are many
> hundreds of comments. Your recent changes in nightlies and beta are a
> big improvement but we still have a few huge reviews that take 30
> seconds due to the large number of reviews.
>
> Keep up the great work!
>
> On Mar 28, 4:54 pm, Christian Hammond <chip...@chipx86.com> wrote:
> > I've made some optimizations today that will be going into beta 1.
> >
> > I created a change with some deleted files and put about 15 comment flags
> on
> > one of the files. I'm sure this isn't anywhere near the amount on your
> > reviews, but it was a good sample to start with.
> >
> > The comment loading code took, on average, 1 second for that particular
> file
> > (which was a file deletion, so 100% changes and expanded).
> >
> > After my changes, that went down to ~155 msec.
> >
> > I'll be curious to see if you guys will notice a measurable impact on
> > performance in beta 1.
> >
> > Christian
> >
> > --
> > Christian Hammond - chip...@chipx86.com
> > Review Board -http://www.review-board.org
> > VMware, Inc. -http://www.vmware.com
> >
> >
> >
> > On Fri, Mar 27, 2009 at 2:39 PM, Mary B <ciaom...@gmail.com> wrote:
> > > That's great to hear, thanks! I will stay tuned...
> >
> > > On Fri, Mar 27, 2009 at 2:27 PM, Christian Hammond <
> chip...@chipx86.com>wrote:
> >
> > >> Okay, I have a couple ideas now where the problem lies.
> >
> > >> I want to give you a heads up that we'll be releasing beta 1 soon, and
> it
> > >> won't include a fix for this. This is going to be a much larger
> problem to
> > >> tackle and I don't want to hold up this release (since it has many
> very
> > >> important fixes in it). I hope to get to this one for beta 2.
> >
> > >> Christian
> >
> > >> --
> > >> Christian Hammond - chip...@chipx86.com
> > >> Review Board -http://www.review-board.org
> > >> VMware, Inc. -http://www.vmware.com
> >
> > >>   On Fri, Mar 27, 2009 at 10:27 AM, ciaomary <ciaom...@gmail.com>
> wrote:
> >
> > >>> We typically only see perf problems on reviews with large diffs and
> > >>> large number of reviews. One that has given us a lot of trouble has a
> > >>> downloadable diff of 1.6MB, 138 different files (with an average file
> > >>> size of ~30K), and hundreds+ review comments. (Bringing up diffs for
> > >>> deleted files give the biggest load and slow down.)
> >
> > >>> Under Admin Settings I've changed the "Paginate by" value from 20
> down
> > >>> to just 3 which has helped a lot. It is set to the default 5 "Lines
> of
> > >>> Context". Also increasing the RAM on our server from 4G to 16G has
> > >>> helped - RB takes a lot of server memory under this kind of load!
> > >>> After all this we're still seeing these big reviews taking 45-60
> > >>> seconds to load, pegging local cpu during that time.
> >
> > >>> I suspect you could reproduce this by setting the "Paginate by" value
> > >>> to something very high. (Try 55 for your 55 page diff.) I believe the
> > >>> problem is only on reviews with lots of comments as well. Our use
> case
> > >>> has been that if there is a really large diff, there are always a lot
> > >>> of review comments as well.
> >
> > >>> Thanks!
> >
> > >>> On Mar 27, 1:38 am, Christian Hammond <chip...@chipx86.com> wrote:
> > >>> > So I just looked at a diff on our server that spans 55 pages, and
> as
> > >>> far as
> > >>> > loading diffs goes, it's not at all slow. How many pages did you
> say
> > >>> yours
> > >>> > were on average?
> >
> > >>> > When looking at diffs, do you have it set to expand the entire
> file, or
> > >>> show
> > >>> > the collapsed version?
> >
> > >>> > Are you seeing slowdown on large diffs that don't have any reviews,
> or
> > >>> is it
> > >>> > just when there's a lot of reviews for that revision of the diff?
> >
> > >>> > Christian
> >
> > >>> > --
> > >>>  > Christian Hammond - chip...@chipx86.com
> > >>> > Review Board -http://www.review-board.org
> > >>> > VMware, Inc. -http://www.vmware.com
> >
> > >>> > On Mon, Mar 23, 2009 at 5:02 PM, Christian Hammond <
> > >>> chip...@chipx86.com>wrote:
> >
> > >>> > > I'm still looking into it. So far I have not seen the slowdown
> you
> > >>> have
> > >>> > > described, but will continue to try to figure out what's going
> on. I
> > >>> know
> > >>> > > this is inconvenient. We just haven't seen usage like this yet.
> >
> > >>> > > Christian
> >
> > >>> > > --
> > >>> > > Christian Hammond - chip...@chipx86.com
> > >>> > > Review Board -http://www.review-board.org
> > >>> > > VMware, Inc. -http://www.vmware.com
> >
> > >>>  > > On Mon, Mar 23, 2009 at 1:19 PM, ciaomary <ciaom...@gmail.com>
> > >>> wrote:
> >
> > >>> > >> Any follow up on this? Any configuration guidance?
> >
> > >>> > >> We've moved to a 16 G + 4 cpu system and still have unacceptable
> > >>> > >> response times (~30 seconds) for really large diffs with
> hundreds of
> > >>> > >> review comments.
> >
> > >>> > >> If you can't reproduce this, I recommend you set the number of
> diff
> > >>> > >> files per page from 20 to something really high (like 100) and
> then
> > >>> > >> post a review with the same number of files deleted. For us,
> this
> > >>> puts
> > >>> > >> a huge load on the local browser and the server VM during load.
> >
> > >>> > >> On Mar 19, 12:41 pm, ciaomary <ciaom...@gmail.com> wrote:
> > >>> > >> > Thanks again. I'm really not that familiar with Apache 2 w/
> > >>> regards to
> > >>> > >> > performance issues. At this time, I just have a more or less
> > >>> out-of-
> > >>> > >> > the-box Apache2 setup on CentOS 5. Any recommendations here
> would
> > >>> be
> > >>> > >> > hugely appreciated. Further answers inline below...
> >
> > >>> > >> > On Mar 19, 12:09 pm, Christian Hammond <chip...@chipx86.com>
> > >>> wrote:
> >
> > >>> > >> > > We should figure out first why your backend server is being
> hit
> > >>> so
> > >>> > >> hard. We
> > >>> > >> > > have over a thousand people at VMware submitting good sized
> > >>> diffs and
> > >>> > >> aren't
> > >>> > >> > > hitting any of these issues. So a few more questions.
> Apologies
> > >>> if
> > >>> > >> you've
> > >>> > >> > > answered these before in other threads.
> >
> > >>> > >> > > 1) What MPM is your Apache using?
> >
> > >>> > >> > I'm using the default for httpd 2.2.3 CentOS. I believe this
> is
> > >>> > >> > 'prefork'.
> >
> > >>> > >> > > 2) And is this fastcgi or mod_python?
> >
> > >>> > >> > mod_python (mod_python-3.2.8-3.1)
> >
> > >>> > >> > > 3) Are you absolutely sure Review Board is using your
> memcached
> > >>> > >> server? Go
> > >>> > >> > > in the admin dashboard and click Server Cache, then copy and
> > >>> paste the
> > >>> > >> info
> > >>> > >> > > and paste it here.
> >
> > >>> > >> > Server Cache
> > >>> > >> > Cache backend: django.core.cache.backends.memcached
> > >>> > >> > Statistics
> > >>> > >> > 127.0.0.1:11211 Memory usage:   45.7 MB
> > >>> > >> > Keys in cache:  1767 of 1777
> > >>> > >> > Cache hits:     3996 of 7464: 53%
> > >>> > >> > Cache misses:   3468 of 7464: 46%
> > >>> > >> > Cache evictions:        0
> > >>> > >> > Cache traffic:  46.1 MB in, 103.8 MB out
> > >>> > >> > Uptime:         53009
> >
> > >>> > >> > > We might be able to do something smarter with deleted diffs,
> but
> > >>> that
> > >>> > >> would
> > >>> > >> > > require a good bit of work, and we'd have to think carefully
> > >>> about it.
> >
> > >>> > >> > > We talked about splitting up the reviews page, but decided
> it
> > >>> wouldn't
> > >>> > >> make
> > >>> > >> > > much sense due to how it's ordered. We came up with a method
> for
> > >>> > >> collapsing,
> > >>> > >> > > but  I'm probably holding off until after 1.0 to finish
> this,
> > >>> because
> > >>> > >> it
> > >>> > >> > > needs a lot more thought and testing.
> >
> > >>> > >> > > Christian
> >
> > >>> > >> > > --
> > >>> > >> > > Christian Hammond - chip...@chipx86.com
> > >>> > >> > > Review Board -http://www.review-board.org
> > >>> > >> > > VMware, Inc. -http://www.vmware.com
> >
> > >>> > >> > > On Thu, Mar 19, 2009 at 11:39 AM, ciaomary <
> ciaom...@gmail.com>
> > >>> > >> wrote:
> >
> > >>> > >> > > > Thanks for your response. Please see comments inline
> below...
> >
> > >>> > >> > > > On Mar 18, 11:35 am, Christian Hammond <
> chip...@chipx86.com>
> > >>> wrote:
> > >>> > >> > > > > Not immediately, no.
> >
> > >>> > >> > > > > I can't even determine right now whether we can do
> anything
> > >>> about
> > >>> > >> this.
> > >>> > >> > > > > You're seeing crazy CPU loads that we're definitely not
> > >>> seeing,
> > >>> > >> and most
> > >>> > >> > > > of
> > >>> > >> > > > > this has to do with the browser itself. We're loading
> the
> > >>> files
> > >>> > >> > > > > asynchronously, but it appears that for whatever reason
> your
> > >>> > >> browser is
> > >>> > >> > > > > blocking until it's all loaded (if I'm understanding
> right).
> >
> > >>> > >> > > > Just for clarity, the page can sometimes load partially,
> but
> > >>> since
> > >>> > >> the
> > >>> > >> > > > CPU is at 99% it's of little value. I don't get the CPU
> back
> > >>> until
> > >>> > >> the
> > >>> > >> > > > page is fully loaded.
> >
> > >>> > >> > > > We now understand that our backend server is also getting
> > >>> maxed out
> > >>> > >> to
> > >>> > >> > > > the point of failure. One request of this page demands 800
> M
> > >>> of
> > >>> > >> > > > virtual memory from Apache! Once the request is fully
> loaded,
> > >>> 400 M
> > >>> > >> of
> > >>> > >> > > > that memory is freed again - but since we have 200
> reviewboard
> > >>> users
> > >>> > >> > > > this isn't very scalable.
> >
> > >>> > >> > > > > I'm wondering
> > >>> > >> > > > > if there's something with the configuration in your
> browsers
> > >>> where
> > >>> > >> you
> > >>> > >> > > > work
> > >>> > >> > > > > that is causing some of this.
> >
> > >>> > >> > > > I really don't think this is a browser configuration issue
> > >>> because
> > >>> > >> > > > users across many different OS and browsers are seeing
> this. I
> > >>> see
> > >>> > >> it
> > >>> > >> > > > on default FireFox 3.0.7 and IE 7.0. Users have reported
> that
> > >>> Safari
> > >>> > >> > > > works the best, but I have not confirmed. FireFox 2 and IE
> 6
> > >>> users
> > >>> > >> > > > have reported the same problem.
> >
> > >>> > >> > > > > You say this is happening on several different browsers?
> In
> > >>> the
> > >>> > >> case of
> > >>> > >> > > > > Firefox, what extensions are loaded?
> >
> > >>> > >> > > > Just the default FireFox 3.0.7 install.
> >
> > >>> > >> > > > > We definitely would like to fix this. But we just don't
> know
> > >>> > >> enough about
> > >>> > >> > > > > what the problem is right now to determine if it's in
> our
> > >>> control.
> >
> > >>> > >> > > > Apache must load up a huge amount of data in VM for these
> > >>> large
> > >>> > >> diffs
> > >>> > >> > > > (I believe this is especially true when diffs contain
> >
> > ...
> >
> > read more ยป- Hide quoted text -
> >
> > - Show quoted text -
> >
>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To post to this group, send email to reviewboard@googlegroups.com
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply via email to