it seems that the syntax highlighting process takes quite a bit of time. I know it is 3rd party. Maybe that tool can be sped up as well.
On Apr 1, 4:37 pm, Martin <mkoeb...@gmail.com> wrote: > for us it takes approximately a minute to view a diff of a file > containing 24,260 lines of C code. > Is 1 minute reasonable for a file that size? (thankfully, files of > that size are an exception). > After turning off syntax highlighting, it now takes only 15 seconds. > > I turned on profiling and I saw the following log (prefixed with line > numbers) > > 1 2009-04-01 13:32:19,874 - DEBUG - Begin: Patching file devel/foms/ > programs/source/stdb3090.c > 2 2009-04-01 13:32:19,974 - DEBUG - End: Patching file devel/foms/ > programs/source/stdb3090.c > 3 2009-04-01 13:32:19,976 - DEBUG - Patching file devel/foms/programs/ > source/stdb3090.c took 0.99753 seconds > 4 2009-04-01 13:32:23,129 - DEBUG - Generating diff chunks for > filediff id 31114 > 5 2009-04-01 13:32:28,008 - DEBUG - Done generating diff chunks for > filediff id 31114 > 6 2009-04-01 13:32:34,837 - DEBUG - End: Generating diff file info > for diffset id 2094, filediff 31114 > 7 2009-04-01 13:32:34,838 - CRITICAL - Generating diff file info for > diffset id 2094, filediff 31114 took 15.94912 seconds > > My first question is, what does it do in the 3 seconds between line 3 > and 4? > I suppose generating the diff chunks is the expensive part, taking 5 > seconds. > Then, what is it doing in the 6 seconds between line 5 and 6? > > Thanks, > Martin > > On Mar 30, 6:26 pm, Christian Hammond <chip...@chipx86.com> wrote:> Thanks, > it's great to hear this feedback :) I was really hoping those > > optimizations would improve things that noticeably. > > > We'll continue to profile the review page and try to come up with > > improvements. I'm actually hoping we're "good enough" for 1.0, since we need > > to freeze things and concentrate on bug fixes. If we can find any easy fixes > > for improving speed, we'll of course do that, but if we were to implement > > review collapsing (as discussed earlier), then we'd need to hold off until a > > 1.1 or something. > > > Anyhow, I'll keep looking into ways of making this faster. Thanks again! > > > Christian > > > -- > > Christian Hammond - chip...@chipx86.com > > Review Board -http://www.review-board.org > > VMware, Inc. -http://www.vmware.com > > > On Mon, Mar 30, 2009 at 2:12 PM, ciaomary <ciaom...@gmail.com> wrote: > > > > Wow, great work! Beta has been a big improvment help for us! The diff > > > viewer looks to be about 5 to 10 times faster in page load based on > > > preliminary testing! > > > > The only remaining perf issue we have left is on the page load of the > > > review page but I believe this is limited to when there are many > > > hundreds of comments. Your recent changes in nightlies and beta are a > > > big improvement but we still have a few huge reviews that take 30 > > > seconds due to the large number of reviews. > > > > Keep up the great work! > > > > On Mar 28, 4:54 pm, Christian Hammond <chip...@chipx86.com> wrote: > > > > I've made some optimizations today that will be going into beta 1. > > > > > I created a change with some deleted files and put about 15 comment > > > > flags > > > on > > > > one of the files. I'm sure this isn't anywhere near the amount on your > > > > reviews, but it was a good sample to start with. > > > > > The comment loading code took, on average, 1 second for that particular > > > file > > > > (which was a file deletion, so 100% changes and expanded). > > > > > After my changes, that went down to ~155 msec. > > > > > I'll be curious to see if you guys will notice a measurable impact on > > > > performance in beta 1. > > > > > Christian > > > > > -- > > > > Christian Hammond - chip...@chipx86.com > > > > Review Board -http://www.review-board.org > > > > VMware, Inc. -http://www.vmware.com > > > > > On Fri, Mar 27, 2009 at 2:39 PM, Mary B <ciaom...@gmail.com> wrote: > > > > > That's great to hear, thanks! I will stay tuned... > > > > > > On Fri, Mar 27, 2009 at 2:27 PM, Christian Hammond < > > > chip...@chipx86.com>wrote: > > > > > >> Okay, I have a couple ideas now where the problem lies. > > > > > >> I want to give you a heads up that we'll be releasing beta 1 soon, > > > > >> and > > > it > > > > >> won't include a fix for this. This is going to be a much larger > > > problem to > > > > >> tackle and I don't want to hold up this release (since it has many > > > very > > > > >> important fixes in it). I hope to get to this one for beta 2. > > > > > >> Christian > > > > > >> -- > > > > >> Christian Hammond - chip...@chipx86.com > > > > >> Review Board -http://www.review-board.org > > > > >> VMware, Inc. -http://www.vmware.com > > > > > >> On Fri, Mar 27, 2009 at 10:27 AM, ciaomary <ciaom...@gmail.com> > > > wrote: > > > > > >>> We typically only see perf problems on reviews with large diffs and > > > > >>> large number of reviews. One that has given us a lot of trouble has > > > > >>> a > > > > >>> downloadable diff of 1.6MB, 138 different files (with an average > > > > >>> file > > > > >>> size of ~30K), and hundreds+ review comments. (Bringing up diffs for > > > > >>> deleted files give the biggest load and slow down.) > > > > > >>> Under Admin Settings I've changed the "Paginate by" value from 20 > > > down > > > > >>> to just 3 which has helped a lot. It is set to the default 5 "Lines > > > of > > > > >>> Context". Also increasing the RAM on our server from 4G to 16G has > > > > >>> helped - RB takes a lot of server memory under this kind of load! > > > > >>> After all this we're still seeing these big reviews taking 45-60 > > > > >>> seconds to load, pegging local cpu during that time. > > > > > >>> I suspect you could reproduce this by setting the "Paginate by" > > > > >>> value > > > > >>> to something very high. (Try 55 for your 55 page diff.) I believe > > > > >>> the > > > > >>> problem is only on reviews with lots of comments as well. Our use > > > case > > > > >>> has been that if there is a really large diff, there are always a > > > > >>> lot > > > > >>> of review comments as well. > > > > > >>> Thanks! > > > > > >>> On Mar 27, 1:38 am, Christian Hammond <chip...@chipx86.com> wrote: > > > > >>> > So I just looked at a diff on our server that spans 55 pages, and > > > as > > > > >>> far as > > > > >>> > loading diffs goes, it's not at all slow. How many pages did you > > > say > > > > >>> yours > > > > >>> > were on average? > > > > > >>> > When looking at diffs, do you have it set to expand the entire > > > file, or > > > > >>> show > > > > >>> > the collapsed version? > > > > > >>> > Are you seeing slowdown on large diffs that don't have any > > > > >>> > reviews, > > > or > > > > >>> is it > > > > >>> > just when there's a lot of reviews for that revision of the diff? > > > > > >>> > Christian > > > > > >>> > -- > > > > >>> > Christian Hammond - chip...@chipx86.com > > > > >>> > Review Board -http://www.review-board.org > > > > >>> > VMware, Inc. -http://www.vmware.com > > > > > >>> > On Mon, Mar 23, 2009 at 5:02 PM, Christian Hammond < > > > > >>> chip...@chipx86.com>wrote: > > > > > >>> > > I'm still looking into it. So far I have not seen the slowdown > > > you > > > > >>> have > > > > >>> > > described, but will continue to try to figure out what's going > > > on. I > > > > >>> know > > > > >>> > > this is inconvenient. We just haven't seen usage like this yet. > > > > > >>> > > Christian > > > > > >>> > > -- > > > > >>> > > Christian Hammond - chip...@chipx86.com > > > > >>> > > Review Board -http://www.review-board.org > > > > >>> > > VMware, Inc. -http://www.vmware.com > > > > > >>> > > On Mon, Mar 23, 2009 at 1:19 PM, ciaomary <ciaom...@gmail.com> > > > > >>> wrote: > > > > > >>> > >> Any follow up on this? Any configuration guidance? > > > > > >>> > >> We've moved to a 16 G + 4 cpu system and still have > > > > >>> > >> unacceptable > > > > >>> > >> response times (~30 seconds) for really large diffs with > > > hundreds of > > > > >>> > >> review comments. > > > > > >>> > >> If you can't reproduce this, I recommend you set the number of > > > diff > > > > >>> > >> files per page from 20 to something really high (like 100) and > > > then > > > > >>> > >> post a review with the same number of files deleted. For us, > > > this > > > > >>> puts > > > > >>> > >> a huge load on the local browser and the server VM during load. > > > > > >>> > >> On Mar 19, 12:41 pm, ciaomary <ciaom...@gmail.com> wrote: > > > > >>> > >> > Thanks again. I'm really not that familiar with Apache 2 w/ > > > > >>> regards to > > > > >>> > >> > performance issues. At this time, I just have a more or less > > > > >>> out-of- > > > > >>> > >> > the-box Apache2 setup on CentOS 5. Any recommendations here > > > would > > > > >>> be > > > > >>> > >> > hugely appreciated. Further answers inline below... > > > > > >>> > >> > On Mar 19, 12:09 pm, Christian Hammond <chip...@chipx86.com> > > > > >>> wrote: > > > > > >>> > >> > > We should figure out first why your backend server is being > > > hit > > > > >>> so > > > > >>> > >> hard. We > > > > >>> > >> > > have over a thousand people at VMware submitting good sized > > > > >>> diffs and > > > > >>> > >> aren't > > > > >>> > >> > > hitting any of these issues. So a few more questions. > > > Apologies > > > > >>> if > > > > >>> > >> you've > > > > >>> > >> > > answered these before in other threads. > > > > > >>> > >> > > 1) What MPM is your Apache using? > > > > > >>> > >> > I'm using the default for httpd 2.2.3 CentOS. I believe this > > > is > > > > >>> > >> > 'prefork'. > > > > > >>> > >> > > 2) And is this fastcgi or mod_python? > > > > > >>> > >> > mod_python (mod_python-3.2.8-3.1) > > > > > >>> > >> > > 3) Are you absolutely sure Review Board is using your > > > memcached > > > > >>> > >> server? Go > > > > >>> > >> > > in the admin dashboard and click Server Cache, then copy > > > > >>> > >> > > and > > > > >>> paste the > > > > >>> > >> info > > > > >>> > >> > > and paste it here. > > > > > >>> > >> > Server Cache > > > > >>> > >> > Cache backend: django.core.cache.backends.memcached > > > > >>> > >> > Statistics > > > > >>> > >> > 127.0.0.1:11211 Memory usage: 45.7 MB > > > > >>> > >> > Keys in cache: 1767 of 1777 > > > > >>> > >> > Cache hits: 3996 of 7464: 53% > > > > >>> > >> > Cache misses: 3468 of 7464: 46% > > > > >>> > >> > Cache evictions: 0 > > > > >>> > >> > Cache traffic: 46.1 MB in, 103.8 MB out > > > > >>> > >> > Uptime: 53009 > > > > > >>> > >> > > We might be able to do something smarter with deleted > > > > >>> > >> > > diffs, > > > but > > > > >>> that > > > > >>> > >> would > > > > >>> > >> > > require a good bit of work, and we'd have to think > > > > >>> > >> > > carefully > > > > >>> about it. > > > > > >>> > >> > > We talked about splitting up the reviews page, but decided > > > it > > > > >>> wouldn't > > > > >>> > >> make > > > > >>> > >> > > much sense due to how it's ordered. We came up with a > > > > >>> > >> > > method > > > for > > > > >>> > >> collapsing, > > > > >>> > >> > > but I'm probably holding off until after 1.0 to finish > > > this, > > > > >>> because > > > > >>> > >> it > > > > >>> > >> > > needs a lot more thought and testing. > > > > > >>> > >> > > Christian > > > > > >>> > >> > > -- > > > > >>> > >> > > Christian Hammond - chip...@chipx86.com > > > > >>> > >> > > Review Board -http://www.review-board.org > > > > >>> > >> > > VMware, Inc. -http://www.vmware.com > > > > > >>> > >> > > On Thu, Mar 19, 2009 at 11:39 AM, ciaomary < > > > ciaom...@gmail.com> > > > > >>> > >> wrote: > > > > > >>> > >> > > > Thanks for your response. Please see comments inline > > > below... > > > > > >>> > >> > > > On Mar 18, 11:35 am, Christian Hammond < > > > chip...@chipx86.com> > > > > >>> wrote: > > > > >>> > >> > > > > Not immediately, no. > > > > > >>> > >> > > > > I can't even determine right now whether we can do > > > anything > > > > >>> about > > > > >>> > >> this. > > > > >>> > >> > > > > You're seeing crazy CPU loads that we're definitely not > > > > >>> seeing, > > > > >>> > >> and most > > > > >>> > >> > > > of > > > > >>> > >> > > > > this has to do with the browser itself. We're loading > > > the > > > ... > > > read more » --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "reviewboard" group. To post to this group, send email to reviewboard@googlegroups.com To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~----------~----~----~----~------~----~------~--~---
