Re: ReviewBoard example instance
Hi Stephen, Sorry, I missed the e-mail where you attached the profile log. I'm looking through it now. The render_to_response was misleading. The slowdown is actually within precompute_objects, which does some database queries. That primarily does database queries, but the SQL logs show that they're not taking very long individually. Certainly not a total of 2 seconds. The rendering doesn't appear to be the slow part either. I don't know much about PostgreSQL, but it almost sounds like there's some issue in the configuration or something. If there's a delay in talking to the database (locking? bad connection?) then I could see that causing this. The low SQL query time is interesting. That makes me wonder if it's just an issue in establishing the first connection. Christian -- Christian Hammond - chip...@chipx86.com Review Board - http://www.reviewboard.org VMware, Inc. - http://www.vmware.com On Mon, Jul 19, 2010 at 5:08 AM, Stephen Gallagher step...@gallagherhome.com wrote: On 07/08/2010 07:17 AM, Stephen Gallagher wrote: On 07/07/2010 06:04 PM, Christian Hammond wrote: Do you happen to have the raw profiling log file too? I want to see the SQL queries as well. There were some heavy optimizations made in Djblets for the datagrid rendering, but I'm not sure if all that made it into beta 2 or not. Christian Sure, attached is the matching .prof file (gzipped since it was rather large uncompressed) *crickets* Christian, have you had a chance to look at this? I'm really unsure how to track this issue down. -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
On 07/19/2010 03:00 PM, Christian Hammond wrote: Hi Stephen, Sorry, I missed the e-mail where you attached the profile log. I'm looking through it now. The render_to_response was misleading. The slowdown is actually within precompute_objects, which does some database queries. That primarily does database queries, but the SQL logs show that they're not taking very long individually. Certainly not a total of 2 seconds. The rendering doesn't appear to be the slow part either. I don't know much about PostgreSQL, but it almost sounds like there's some issue in the configuration or something. If there's a delay in talking to the database (locking? bad connection?) then I could see that causing this. The low SQL query time is interesting. That makes me wonder if it's just an issue in establishing the first connection. Well, the problem is that I'm seeing a lag of about 8 seconds for every page load. I can't imagine the SQL query time being that slow... Furthermore, there are other Django-based web applications talking to this same database server without any obvious performance issues. (Specifically, it's a Transifex instance, so it's low-traffic. That wouldn't be wasting DB time). -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
On 07/19/2010 03:12 PM, Christian Hammond wrote: Can you verify that the settings_local.py in Review Board and the settings.py in other Django sites are using the exact same database backend? Are all the sites running on the same server? The logs you provided show a lag of ~2 seconds rather than 8, though. If you're seeing 8 from your end, but the logs are showing 2, then there's a whole different issue happening, but I'm assuming it just varies on attempt and page. Well, I think I may be seeing 2s of lag for each component being displayed on the dashboard, which would account for the 8 seconds total. I just confirmed that both Transifex and ReviewBoard are both using postgresql-psycopg2 to talk to the database. Transifex and ReviewBoard themselves are both running on separate servers. Transifex has a dedicated machine at the moment, while ReviewBoard is shared with a number of Trac instances (but the load is low and there's plenty of free memory). -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
On 06/12/2010 03:39 AM, Christian Hammond wrote: Hmm, it might be due to not having cProfile installed? Does the reviewboard.log show anything? Christian -- Christian Hammond - chip...@chipx86.com mailto:chip...@chipx86.com Review Board - http://www.reviewboard.org VMware, Inc. - http://www.vmware.com On Fri, Jun 11, 2010 at 1:30 PM, Stephen Gallagher step...@gallagherhome.com mailto:step...@gallagherhome.com wrote: On 06/10/2010 03:24 PM, Christian Hammond wrote: Hi Stephen, We're running on MySQL. I don't know why you'd be seeing such a huge delay with PostgreSQL though. Can you tell me which pages are showing delays? (Is it all of them?) if it's the dashboard, there are some known slowdowns when reaching large numbers of review requests. If the diff viewer, make sure your memcached is caching aggressively. One thing you could try doing is going into Settings - Logging and turn on Profiling. Then you can visit a page with slowdowns and run with ?profiling=1. This should output to a reviewboard.prof file in the logging directory with some details on the SQL queries and the code paths hit. If you could pass that on to me, I could take a look. Still working on this, but I thought I should mention. I get a 500 error on every page that I try to use ?profiling=1 on. Is this a known bug on 1.5 beta 2? Okay, it turned out that I needed to hack python-djblets to use profile instead of cProfile (since cProfile requires python 2.5 and RHEL5 only has 2.4) I got the following profile results (graphed for easier consumption): http://sgallagh.fedorapeople.org/rb-profile.svg If I'm reading it right, it looks like the majority of the time is being lost in grids:696:render_to_response * That SVG should be viewable in Firefox, but if not I can generate a PNG as well - it's just much larger. -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
On 06/10/2010 03:24 PM, Christian Hammond wrote: Hi Stephen, We're running on MySQL. I don't know why you'd be seeing such a huge delay with PostgreSQL though. Can you tell me which pages are showing delays? (Is it all of them?) if it's the dashboard, there are some known slowdowns when reaching large numbers of review requests. If the diff viewer, make sure your memcached is caching aggressively. One thing you could try doing is going into Settings - Logging and turn on Profiling. Then you can visit a page with slowdowns and run with ?profiling=1. This should output to a reviewboard.prof file in the logging directory with some details on the SQL queries and the code paths hit. If you could pass that on to me, I could take a look. Still working on this, but I thought I should mention. I get a 500 error on every page that I try to use ?profiling=1 on. Is this a known bug on 1.5 beta 2? -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
It is probably between SQLite and PostgreSQL. But this is delay is proportional to scalable. SQLite is not scalable at all, PostgreSQL is most scalable free SQL database. On Jun 9, 9:39 pm, Stephen Gallagher step...@gallagherhome.com wrote: I'm curious, what database is thehttp://reviews.reviewboard.orgserver using? We deployed on PostgreSQL recently, and I've noticed a distinct performance issue with it. I can only directly compare it to SQLite, but of course that's local access. But we went from 1s responses on webpages to 4-8s per page when backed by Postgresql (on a private network with very little traffic). How is the performance when backed by MySQL? -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
Well, right now 4-8s for a page refresh is just barely within the tolerable limits of my users. I'm trying to determine whether it makes more sense to migrate to MySQL (if it would be better performance) I was also curious whether ReviewBoard's reference implementation was running on postgresql and whether there were any tuning tricks I should be aware of. On 06/10/2010 02:53 PM, Jan Koprowski wrote: It is probably between SQLite and PostgreSQL. But this is delay is proportional to scalable. SQLite is not scalable at all, PostgreSQL is most scalable free SQL database. On Jun 9, 9:39 pm, Stephen Gallagherstep...@gallagherhome.com wrote: I'm curious, what database is thehttp://reviews.reviewboard.orgserver using? We deployed on PostgreSQL recently, and I've noticed a distinct performance issue with it. I can only directly compare it to SQLite, but of course that's local access. But we went from 1s responses on webpages to 4-8s per page when backed by Postgresql (on a private network with very little traffic). How is the performance when backed by MySQL? -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
Hi Stephen, We're running on MySQL. I don't know why you'd be seeing such a huge delay with PostgreSQL though. Can you tell me which pages are showing delays? (Is it all of them?) if it's the dashboard, there are some known slowdowns when reaching large numbers of review requests. If the diff viewer, make sure your memcached is caching aggressively. One thing you could try doing is going into Settings - Logging and turn on Profiling. Then you can visit a page with slowdowns and run with ?profiling=1. This should output to a reviewboard.prof file in the logging directory with some details on the SQL queries and the code paths hit. If you could pass that on to me, I could take a look. Christian -- Christian Hammond - chip...@chipx86.com Review Board - http://www.reviewboard.org VMware, Inc. - http://www.vmware.com On Thu, Jun 10, 2010 at 11:56 AM, Stephen Gallagher step...@gallagherhome.com wrote: Well, right now 4-8s for a page refresh is just barely within the tolerable limits of my users. I'm trying to determine whether it makes more sense to migrate to MySQL (if it would be better performance) I was also curious whether ReviewBoard's reference implementation was running on postgresql and whether there were any tuning tricks I should be aware of. On 06/10/2010 02:53 PM, Jan Koprowski wrote: It is probably between SQLite and PostgreSQL. But this is delay is proportional to scalable. SQLite is not scalable at all, PostgreSQL is most scalable free SQL database. On Jun 9, 9:39 pm, Stephen Gallagherstep...@gallagherhome.com wrote: I'm curious, what database is thehttp://reviews.reviewboard.orgserver using? We deployed on PostgreSQL recently, and I've noticed a distinct performance issue with it. I can only directly compare it to SQLite, but of course that's local access. But we went from 1s responses on webpages to 4-8s per page when backed by Postgresql (on a private network with very little traffic). How is the performance when backed by MySQL? -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
On 06/10/2010 03:24 PM, Christian Hammond wrote: Hi Stephen, We're running on MySQL. I don't know why you'd be seeing such a huge delay with PostgreSQL though. Can you tell me which pages are showing delays? (Is it all of them?) Yeah, it's every page but the login page. That includes the admin UI. if it's the dashboard, there are some known slowdowns when reaching large numbers of review requests. If the diff viewer, make sure your memcached is caching aggressively. For the record, our test instance currently has fewer than thirty review requests so far, so I guarantee it's not this. One thing you could try doing is going into Settings - Logging and turn on Profiling. Then you can visit a page with slowdowns and run with ?profiling=1. This should output to a reviewboard.prof file in the logging directory with some details on the SQL queries and the code paths hit. If you could pass that on to me, I could take a look. Yeah, I'm going to do some additional profiling. Something I noticed when doing testing on django_evolution (using the test suite you directed me to) was that I saw a difference of 7s vs. 45s for the time it took to run the suite. This is leading me to believe that the problem is more likely to be in python-pyscopg2 or Django itself, rather than ReviewBoard. I'll let you know what I find. -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
You can check Your efficient of memcached caching. This could be critical. Read about getting statistic from memcached: http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt this is the fastest element of ReviewBoard. If You still belive this is database engine bottle neck here is good article witch talk about Postgres tuning also: http://jacobian.org/writing/django-performance-tips/ On Jun 10, 8:56 pm, Stephen Gallagher step...@gallagherhome.com wrote: Well, right now 4-8s for a page refresh is just barely within the tolerable limits of my users. I'm trying to determine whether it makes more sense to migrate to MySQL (if it would be better performance) I was also curious whether ReviewBoard's reference implementation was running on postgresql and whether there were any tuning tricks I should be aware of. On 06/10/2010 02:53 PM, Jan Koprowski wrote: It is probably between SQLite and PostgreSQL. But this is delay is proportional to scalable. SQLite is not scalable at all, PostgreSQL is most scalable free SQL database. On Jun 9, 9:39 pm, Stephen Gallagherstep...@gallagherhome.com wrote: I'm curious, what database is thehttp://reviews.reviewboard.orgserver using? We deployed on PostgreSQL recently, and I've noticed a distinct performance issue with it. I can only directly compare it to SQLite, but of course that's local access. But we went from 1s responses on webpages to 4-8s per page when backed by Postgresql (on a private network with very little traffic). How is the performance when backed by MySQL? -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
Re: ReviewBoard example instance
The reviews.reviewboard.org server is running under mysql. We've done a little tuning but not much. I haven't tried with pgsql at all, personally. -David On Jun 10, 2010, at 11:56 AM, Stephen Gallagher step...@gallagherhome.com wrote: Well, right now 4-8s for a page refresh is just barely within the tolerable limits of my users. I'm trying to determine whether it makes more sense to migrate to MySQL (if it would be better performance) I was also curious whether ReviewBoard's reference implementation was running on postgresql and whether there were any tuning tricks I should be aware of. On 06/10/2010 02:53 PM, Jan Koprowski wrote: It is probably between SQLite and PostgreSQL. But this is delay is proportional to scalable. SQLite is not scalable at all, PostgreSQL is most scalable free SQL database. On Jun 9, 9:39 pm, Stephen Gallagherstep...@gallagherhome.com wrote: I'm curious, what database is thehttp:// reviews.reviewboard.orgserver using? We deployed on PostgreSQL recently, and I've noticed a distinct performance issue with it. I can only directly compare it to SQLite, but of course that's local access. But we went from 1s responses on webpages to 4-8s per page when backed by Postgresql (on a private network with very little traffic). How is the performance when backed by MySQL? -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en -- Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/ Happy user? Let us know at http://www.reviewboard.org/users/ -~--~~~~--~~--~--~--- To unsubscribe from this group, send email to reviewboard+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
