Wow, great work! Beta has been a big improvment help for us! The diff viewer looks to be about 5 to 10 times faster in page load based on preliminary testing!
The only remaining perf issue we have left is on the page load of the review page but I believe this is limited to when there are many hundreds of comments. Your recent changes in nightlies and beta are a big improvement but we still have a few huge reviews that take 30 seconds due to the large number of reviews. Keep up the great work! On Mar 28, 4:54 pm, Christian Hammond <chip...@chipx86.com> wrote: > I've made some optimizations today that will be going into beta 1. > > I created a change with some deleted files and put about 15 comment flags on > one of the files. I'm sure this isn't anywhere near the amount on your > reviews, but it was a good sample to start with. > > The comment loading code took, on average, 1 second for that particular file > (which was a file deletion, so 100% changes and expanded). > > After my changes, that went down to ~155 msec. > > I'll be curious to see if you guys will notice a measurable impact on > performance in beta 1. > > Christian > > -- > Christian Hammond - chip...@chipx86.com > Review Board -http://www.review-board.org > VMware, Inc. -http://www.vmware.com > > > > On Fri, Mar 27, 2009 at 2:39 PM, Mary B <ciaom...@gmail.com> wrote: > > That's great to hear, thanks! I will stay tuned... > > > On Fri, Mar 27, 2009 at 2:27 PM, Christian Hammond > > <chip...@chipx86.com>wrote: > > >> Okay, I have a couple ideas now where the problem lies. > > >> I want to give you a heads up that we'll be releasing beta 1 soon, and it > >> won't include a fix for this. This is going to be a much larger problem to > >> tackle and I don't want to hold up this release (since it has many very > >> important fixes in it). I hope to get to this one for beta 2. > > >> Christian > > >> -- > >> Christian Hammond - chip...@chipx86.com > >> Review Board -http://www.review-board.org > >> VMware, Inc. -http://www.vmware.com > > >> On Fri, Mar 27, 2009 at 10:27 AM, ciaomary <ciaom...@gmail.com> wrote: > > >>> We typically only see perf problems on reviews with large diffs and > >>> large number of reviews. One that has given us a lot of trouble has a > >>> downloadable diff of 1.6MB, 138 different files (with an average file > >>> size of ~30K), and hundreds+ review comments. (Bringing up diffs for > >>> deleted files give the biggest load and slow down.) > > >>> Under Admin Settings I've changed the "Paginate by" value from 20 down > >>> to just 3 which has helped a lot. It is set to the default 5 "Lines of > >>> Context". Also increasing the RAM on our server from 4G to 16G has > >>> helped - RB takes a lot of server memory under this kind of load! > >>> After all this we're still seeing these big reviews taking 45-60 > >>> seconds to load, pegging local cpu during that time. > > >>> I suspect you could reproduce this by setting the "Paginate by" value > >>> to something very high. (Try 55 for your 55 page diff.) I believe the > >>> problem is only on reviews with lots of comments as well. Our use case > >>> has been that if there is a really large diff, there are always a lot > >>> of review comments as well. > > >>> Thanks! > > >>> On Mar 27, 1:38 am, Christian Hammond <chip...@chipx86.com> wrote: > >>> > So I just looked at a diff on our server that spans 55 pages, and as > >>> far as > >>> > loading diffs goes, it's not at all slow. How many pages did you say > >>> yours > >>> > were on average? > > >>> > When looking at diffs, do you have it set to expand the entire file, or > >>> show > >>> > the collapsed version? > > >>> > Are you seeing slowdown on large diffs that don't have any reviews, or > >>> is it > >>> > just when there's a lot of reviews for that revision of the diff? > > >>> > Christian > > >>> > -- > >>> > Christian Hammond - chip...@chipx86.com > >>> > Review Board -http://www.review-board.org > >>> > VMware, Inc. -http://www.vmware.com > > >>> > On Mon, Mar 23, 2009 at 5:02 PM, Christian Hammond < > >>> chip...@chipx86.com>wrote: > > >>> > > I'm still looking into it. So far I have not seen the slowdown you > >>> have > >>> > > described, but will continue to try to figure out what's going on. I > >>> know > >>> > > this is inconvenient. We just haven't seen usage like this yet. > > >>> > > Christian > > >>> > > -- > >>> > > Christian Hammond - chip...@chipx86.com > >>> > > Review Board -http://www.review-board.org > >>> > > VMware, Inc. -http://www.vmware.com > > >>> > > On Mon, Mar 23, 2009 at 1:19 PM, ciaomary <ciaom...@gmail.com> > >>> wrote: > > >>> > >> Any follow up on this? Any configuration guidance? > > >>> > >> We've moved to a 16 G + 4 cpu system and still have unacceptable > >>> > >> response times (~30 seconds) for really large diffs with hundreds of > >>> > >> review comments. > > >>> > >> If you can't reproduce this, I recommend you set the number of diff > >>> > >> files per page from 20 to something really high (like 100) and then > >>> > >> post a review with the same number of files deleted. For us, this > >>> puts > >>> > >> a huge load on the local browser and the server VM during load. > > >>> > >> On Mar 19, 12:41 pm, ciaomary <ciaom...@gmail.com> wrote: > >>> > >> > Thanks again. I'm really not that familiar with Apache 2 w/ > >>> regards to > >>> > >> > performance issues. At this time, I just have a more or less > >>> out-of- > >>> > >> > the-box Apache2 setup on CentOS 5. Any recommendations here would > >>> be > >>> > >> > hugely appreciated. Further answers inline below... > > >>> > >> > On Mar 19, 12:09 pm, Christian Hammond <chip...@chipx86.com> > >>> wrote: > > >>> > >> > > We should figure out first why your backend server is being hit > >>> so > >>> > >> hard. We > >>> > >> > > have over a thousand people at VMware submitting good sized > >>> diffs and > >>> > >> aren't > >>> > >> > > hitting any of these issues. So a few more questions. Apologies > >>> if > >>> > >> you've > >>> > >> > > answered these before in other threads. > > >>> > >> > > 1) What MPM is your Apache using? > > >>> > >> > I'm using the default for httpd 2.2.3 CentOS. I believe this is > >>> > >> > 'prefork'. > > >>> > >> > > 2) And is this fastcgi or mod_python? > > >>> > >> > mod_python (mod_python-3.2.8-3.1) > > >>> > >> > > 3) Are you absolutely sure Review Board is using your memcached > >>> > >> server? Go > >>> > >> > > in the admin dashboard and click Server Cache, then copy and > >>> paste the > >>> > >> info > >>> > >> > > and paste it here. > > >>> > >> > Server Cache > >>> > >> > Cache backend: django.core.cache.backends.memcached > >>> > >> > Statistics > >>> > >> > 127.0.0.1:11211 Memory usage: 45.7 MB > >>> > >> > Keys in cache: 1767 of 1777 > >>> > >> > Cache hits: 3996 of 7464: 53% > >>> > >> > Cache misses: 3468 of 7464: 46% > >>> > >> > Cache evictions: 0 > >>> > >> > Cache traffic: 46.1 MB in, 103.8 MB out > >>> > >> > Uptime: 53009 > > >>> > >> > > We might be able to do something smarter with deleted diffs, but > >>> that > >>> > >> would > >>> > >> > > require a good bit of work, and we'd have to think carefully > >>> about it. > > >>> > >> > > We talked about splitting up the reviews page, but decided it > >>> wouldn't > >>> > >> make > >>> > >> > > much sense due to how it's ordered. We came up with a method for > >>> > >> collapsing, > >>> > >> > > but I'm probably holding off until after 1.0 to finish this, > >>> because > >>> > >> it > >>> > >> > > needs a lot more thought and testing. > > >>> > >> > > Christian > > >>> > >> > > -- > >>> > >> > > Christian Hammond - chip...@chipx86.com > >>> > >> > > Review Board -http://www.review-board.org > >>> > >> > > VMware, Inc. -http://www.vmware.com > > >>> > >> > > On Thu, Mar 19, 2009 at 11:39 AM, ciaomary <ciaom...@gmail.com> > >>> > >> wrote: > > >>> > >> > > > Thanks for your response. Please see comments inline below... > > >>> > >> > > > On Mar 18, 11:35 am, Christian Hammond <chip...@chipx86.com> > >>> wrote: > >>> > >> > > > > Not immediately, no. > > >>> > >> > > > > I can't even determine right now whether we can do anything > >>> about > >>> > >> this. > >>> > >> > > > > You're seeing crazy CPU loads that we're definitely not > >>> seeing, > >>> > >> and most > >>> > >> > > > of > >>> > >> > > > > this has to do with the browser itself. We're loading the > >>> files > >>> > >> > > > > asynchronously, but it appears that for whatever reason your > >>> > >> browser is > >>> > >> > > > > blocking until it's all loaded (if I'm understanding right). > > >>> > >> > > > Just for clarity, the page can sometimes load partially, but > >>> since > >>> > >> the > >>> > >> > > > CPU is at 99% it's of little value. I don't get the CPU back > >>> until > >>> > >> the > >>> > >> > > > page is fully loaded. > > >>> > >> > > > We now understand that our backend server is also getting > >>> maxed out > >>> > >> to > >>> > >> > > > the point of failure. One request of this page demands 800 M > >>> of > >>> > >> > > > virtual memory from Apache! Once the request is fully loaded, > >>> 400 M > >>> > >> of > >>> > >> > > > that memory is freed again - but since we have 200 reviewboard > >>> users > >>> > >> > > > this isn't very scalable. > > >>> > >> > > > > I'm wondering > >>> > >> > > > > if there's something with the configuration in your browsers > >>> where > >>> > >> you > >>> > >> > > > work > >>> > >> > > > > that is causing some of this. > > >>> > >> > > > I really don't think this is a browser configuration issue > >>> because > >>> > >> > > > users across many different OS and browsers are seeing this. I > >>> see > >>> > >> it > >>> > >> > > > on default FireFox 3.0.7 and IE 7.0. Users have reported that > >>> Safari > >>> > >> > > > works the best, but I have not confirmed. FireFox 2 and IE 6 > >>> users > >>> > >> > > > have reported the same problem. > > >>> > >> > > > > You say this is happening on several different browsers? In > >>> the > >>> > >> case of > >>> > >> > > > > Firefox, what extensions are loaded? > > >>> > >> > > > Just the default FireFox 3.0.7 install. > > >>> > >> > > > > We definitely would like to fix this. But we just don't know > >>> > >> enough about > >>> > >> > > > > what the problem is right now to determine if it's in our > >>> control. > > >>> > >> > > > Apache must load up a huge amount of data in VM for these > >>> large > >>> > >> diffs > >>> > >> > > > (I believe this is especially true when diffs contain > > ... > > read more »- Hide quoted text - > > - Show quoted text - --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "reviewboard" group. To post to this group, send email to reviewboard@googlegroups.com To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -~----------~----~----~----~------~----~------~--~---
