for us it takes approximately a minute to view a diff of a file
containing 24,260 lines of C code.
Is 1 minute reasonable for a file that size? (thankfully, files of
that size are an exception).
After turning off syntax highlighting, it now takes only 15 seconds.

I turned on profiling and I saw the following log (prefixed with line
numbers)

1  2009-04-01 13:32:19,874 - DEBUG - Begin: Patching file devel/foms/
programs/source/stdb3090.c
2  2009-04-01 13:32:19,974 - DEBUG - End: Patching file devel/foms/
programs/source/stdb3090.c
3  2009-04-01 13:32:19,976 - DEBUG - Patching file devel/foms/programs/
source/stdb3090.c took 0.99753 seconds
4  2009-04-01 13:32:23,129 - DEBUG - Generating diff chunks for
filediff id 31114
5  2009-04-01 13:32:28,008 - DEBUG - Done generating diff chunks for
filediff id 31114
6  2009-04-01 13:32:34,837 - DEBUG - End: Generating diff file info
for diffset id 2094, filediff 31114
7  2009-04-01 13:32:34,838 - CRITICAL - Generating diff file info for
diffset id 2094, filediff 31114 took 15.94912 seconds

My first question is, what does it do in the 3 seconds between line 3
and 4?
I suppose generating the diff chunks is the expensive part, taking 5
seconds.
Then, what is it doing in the 6 seconds between line 5 and 6?

Thanks,
Martin

On Mar 30, 6:26 pm, Christian Hammond <chip...@chipx86.com> wrote:
> Thanks, it's great to hear this feedback :) I was really hoping those
> optimizations would improve things that noticeably.
>
> We'll continue to profile the review page and try to come up with
> improvements. I'm actually hoping we're "good enough" for 1.0, since we need
> to freeze things and concentrate on bug fixes. If we can find any easy fixes
> for improving speed, we'll of course do that, but if we were to implement
> review collapsing (as discussed earlier), then we'd need to hold off until a
> 1.1 or something.
>
> Anyhow, I'll keep looking into ways of making this faster. Thanks again!
>
> Christian
>
> --
> Christian Hammond - chip...@chipx86.com
> Review Board -http://www.review-board.org
> VMware, Inc. -http://www.vmware.com
>
> On Mon, Mar 30, 2009 at 2:12 PM, ciaomary <ciaom...@gmail.com> wrote:
>
> > Wow, great work! Beta has been a big improvment help for us! The diff
> > viewer looks to be about 5 to 10 times faster in page load based on
> > preliminary testing!
>
> > The only remaining perf issue we have left is on the page load of the
> > review page but I believe this is limited to when there are many
> > hundreds of comments. Your recent changes in nightlies and beta are a
> > big improvement but we still have a few huge reviews that take 30
> > seconds due to the large number of reviews.
>
> > Keep up the great work!
>
> > On Mar 28, 4:54 pm, Christian Hammond <chip...@chipx86.com> wrote:
> > > I've made some optimizations today that will be going into beta 1.
>
> > > I created a change with some deleted files and put about 15 comment flags
> > on
> > > one of the files. I'm sure this isn't anywhere near the amount on your
> > > reviews, but it was a good sample to start with.
>
> > > The comment loading code took, on average, 1 second for that particular
> > file
> > > (which was a file deletion, so 100% changes and expanded).
>
> > > After my changes, that went down to ~155 msec.
>
> > > I'll be curious to see if you guys will notice a measurable impact on
> > > performance in beta 1.
>
> > > Christian
>
> > > --
> > > Christian Hammond - chip...@chipx86.com
> > > Review Board -http://www.review-board.org
> > > VMware, Inc. -http://www.vmware.com
>
> > > On Fri, Mar 27, 2009 at 2:39 PM, Mary B <ciaom...@gmail.com> wrote:
> > > > That's great to hear, thanks! I will stay tuned...
>
> > > > On Fri, Mar 27, 2009 at 2:27 PM, Christian Hammond <
> > chip...@chipx86.com>wrote:
>
> > > >> Okay, I have a couple ideas now where the problem lies.
>
> > > >> I want to give you a heads up that we'll be releasing beta 1 soon, and
> > it
> > > >> won't include a fix for this. This is going to be a much larger
> > problem to
> > > >> tackle and I don't want to hold up this release (since it has many
> > very
> > > >> important fixes in it). I hope to get to this one for beta 2.
>
> > > >> Christian
>
> > > >> --
> > > >> Christian Hammond - chip...@chipx86.com
> > > >> Review Board -http://www.review-board.org
> > > >> VMware, Inc. -http://www.vmware.com
>
> > > >>   On Fri, Mar 27, 2009 at 10:27 AM, ciaomary <ciaom...@gmail.com>
> > wrote:
>
> > > >>> We typically only see perf problems on reviews with large diffs and
> > > >>> large number of reviews. One that has given us a lot of trouble has a
> > > >>> downloadable diff of 1.6MB, 138 different files (with an average file
> > > >>> size of ~30K), and hundreds+ review comments. (Bringing up diffs for
> > > >>> deleted files give the biggest load and slow down.)
>
> > > >>> Under Admin Settings I've changed the "Paginate by" value from 20
> > down
> > > >>> to just 3 which has helped a lot. It is set to the default 5 "Lines
> > of
> > > >>> Context". Also increasing the RAM on our server from 4G to 16G has
> > > >>> helped - RB takes a lot of server memory under this kind of load!
> > > >>> After all this we're still seeing these big reviews taking 45-60
> > > >>> seconds to load, pegging local cpu during that time.
>
> > > >>> I suspect you could reproduce this by setting the "Paginate by" value
> > > >>> to something very high. (Try 55 for your 55 page diff.) I believe the
> > > >>> problem is only on reviews with lots of comments as well. Our use
> > case
> > > >>> has been that if there is a really large diff, there are always a lot
> > > >>> of review comments as well.
>
> > > >>> Thanks!
>
> > > >>> On Mar 27, 1:38 am, Christian Hammond <chip...@chipx86.com> wrote:
> > > >>> > So I just looked at a diff on our server that spans 55 pages, and
> > as
> > > >>> far as
> > > >>> > loading diffs goes, it's not at all slow. How many pages did you
> > say
> > > >>> yours
> > > >>> > were on average?
>
> > > >>> > When looking at diffs, do you have it set to expand the entire
> > file, or
> > > >>> show
> > > >>> > the collapsed version?
>
> > > >>> > Are you seeing slowdown on large diffs that don't have any reviews,
> > or
> > > >>> is it
> > > >>> > just when there's a lot of reviews for that revision of the diff?
>
> > > >>> > Christian
>
> > > >>> > --
> > > >>>  > Christian Hammond - chip...@chipx86.com
> > > >>> > Review Board -http://www.review-board.org
> > > >>> > VMware, Inc. -http://www.vmware.com
>
> > > >>> > On Mon, Mar 23, 2009 at 5:02 PM, Christian Hammond <
> > > >>> chip...@chipx86.com>wrote:
>
> > > >>> > > I'm still looking into it. So far I have not seen the slowdown
> > you
> > > >>> have
> > > >>> > > described, but will continue to try to figure out what's going
> > on. I
> > > >>> know
> > > >>> > > this is inconvenient. We just haven't seen usage like this yet.
>
> > > >>> > > Christian
>
> > > >>> > > --
> > > >>> > > Christian Hammond - chip...@chipx86.com
> > > >>> > > Review Board -http://www.review-board.org
> > > >>> > > VMware, Inc. -http://www.vmware.com
>
> > > >>>  > > On Mon, Mar 23, 2009 at 1:19 PM, ciaomary <ciaom...@gmail.com>
> > > >>> wrote:
>
> > > >>> > >> Any follow up on this? Any configuration guidance?
>
> > > >>> > >> We've moved to a 16 G + 4 cpu system and still have unacceptable
> > > >>> > >> response times (~30 seconds) for really large diffs with
> > hundreds of
> > > >>> > >> review comments.
>
> > > >>> > >> If you can't reproduce this, I recommend you set the number of
> > diff
> > > >>> > >> files per page from 20 to something really high (like 100) and
> > then
> > > >>> > >> post a review with the same number of files deleted. For us,
> > this
> > > >>> puts
> > > >>> > >> a huge load on the local browser and the server VM during load.
>
> > > >>> > >> On Mar 19, 12:41 pm, ciaomary <ciaom...@gmail.com> wrote:
> > > >>> > >> > Thanks again. I'm really not that familiar with Apache 2 w/
> > > >>> regards to
> > > >>> > >> > performance issues. At this time, I just have a more or less
> > > >>> out-of-
> > > >>> > >> > the-box Apache2 setup on CentOS 5. Any recommendations here
> > would
> > > >>> be
> > > >>> > >> > hugely appreciated. Further answers inline below...
>
> > > >>> > >> > On Mar 19, 12:09 pm, Christian Hammond <chip...@chipx86.com>
> > > >>> wrote:
>
> > > >>> > >> > > We should figure out first why your backend server is being
> > hit
> > > >>> so
> > > >>> > >> hard. We
> > > >>> > >> > > have over a thousand people at VMware submitting good sized
> > > >>> diffs and
> > > >>> > >> aren't
> > > >>> > >> > > hitting any of these issues. So a few more questions.
> > Apologies
> > > >>> if
> > > >>> > >> you've
> > > >>> > >> > > answered these before in other threads.
>
> > > >>> > >> > > 1) What MPM is your Apache using?
>
> > > >>> > >> > I'm using the default for httpd 2.2.3 CentOS. I believe this
> > is
> > > >>> > >> > 'prefork'.
>
> > > >>> > >> > > 2) And is this fastcgi or mod_python?
>
> > > >>> > >> > mod_python (mod_python-3.2.8-3.1)
>
> > > >>> > >> > > 3) Are you absolutely sure Review Board is using your
> > memcached
> > > >>> > >> server? Go
> > > >>> > >> > > in the admin dashboard and click Server Cache, then copy and
> > > >>> paste the
> > > >>> > >> info
> > > >>> > >> > > and paste it here.
>
> > > >>> > >> > Server Cache
> > > >>> > >> > Cache backend: django.core.cache.backends.memcached
> > > >>> > >> > Statistics
> > > >>> > >> > 127.0.0.1:11211 Memory usage:   45.7 MB
> > > >>> > >> > Keys in cache:  1767 of 1777
> > > >>> > >> > Cache hits:     3996 of 7464: 53%
> > > >>> > >> > Cache misses:   3468 of 7464: 46%
> > > >>> > >> > Cache evictions:        0
> > > >>> > >> > Cache traffic:  46.1 MB in, 103.8 MB out
> > > >>> > >> > Uptime:         53009
>
> > > >>> > >> > > We might be able to do something smarter with deleted diffs,
> > but
> > > >>> that
> > > >>> > >> would
> > > >>> > >> > > require a good bit of work, and we'd have to think carefully
> > > >>> about it.
>
> > > >>> > >> > > We talked about splitting up the reviews page, but decided
> > it
> > > >>> wouldn't
> > > >>> > >> make
> > > >>> > >> > > much sense due to how it's ordered. We came up with a method
> > for
> > > >>> > >> collapsing,
> > > >>> > >> > > but  I'm probably holding off until after 1.0 to finish
> > this,
> > > >>> because
> > > >>> > >> it
> > > >>> > >> > > needs a lot more thought and testing.
>
> > > >>> > >> > > Christian
>
> > > >>> > >> > > --
> > > >>> > >> > > Christian Hammond - chip...@chipx86.com
> > > >>> > >> > > Review Board -http://www.review-board.org
> > > >>> > >> > > VMware, Inc. -http://www.vmware.com
>
> > > >>> > >> > > On Thu, Mar 19, 2009 at 11:39 AM, ciaomary <
> > ciaom...@gmail.com>
> > > >>> > >> wrote:
>
> > > >>> > >> > > > Thanks for your response. Please see comments inline
> > below...
>
> > > >>> > >> > > > On Mar 18, 11:35 am, Christian Hammond <
> > chip...@chipx86.com>
> > > >>> wrote:
> > > >>> > >> > > > > Not immediately, no.
>
> > > >>> > >> > > > > I can't even determine right now whether we can do
> > anything
> > > >>> about
> > > >>> > >> this.
> > > >>> > >> > > > > You're seeing crazy CPU loads that we're definitely not
> > > >>> seeing,
> > > >>> > >> and most
> > > >>> > >> > > > of
> > > >>> > >> > > > > this has to do with the browser itself. We're loading
> > the
>
> ...
>
> read more »
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"reviewboard" group.
To post to this group, send email to reviewboard@googlegroups.com
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply via email to