Re: ReviewBoard example instance

2010-07-19 Thread Stephen Gallagher
On 07/08/2010 07:17 AM, Stephen Gallagher wrote:
 On 07/07/2010 06:04 PM, Christian Hammond wrote:
   
 Do you happen to have the raw profiling log file too? I want to see
 the SQL queries as well.

 There were some heavy optimizations made in Djblets for the datagrid
 rendering, but I'm not sure if all that made it into beta 2 or not.

 Christian
 
 Sure, attached is the matching .prof file (gzipped since it was rather
 large uncompressed)
   
*crickets*

Christian, have you had a chance to look at this? I'm really unsure how
to track this issue down.

-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


Re: ReviewBoard example instance

2010-07-19 Thread Christian Hammond
Hi Stephen,

Sorry, I missed the e-mail where you attached the profile log. I'm looking
through it now.

The render_to_response was misleading. The slowdown is actually within
precompute_objects, which does some database queries. That primarily does
database queries, but the SQL logs show that they're not taking very long
individually. Certainly not a total of 2 seconds. The rendering doesn't
appear to be the slow part either.

I don't know much about PostgreSQL, but it almost sounds like there's some
issue in the configuration or something. If there's a delay in talking to
the database (locking? bad connection?) then I could see that causing this.
The low SQL query time is interesting. That makes me wonder if it's just an
issue in establishing the first connection.

Christian

-- 
Christian Hammond - chip...@chipx86.com
Review Board - http://www.reviewboard.org
VMware, Inc. - http://www.vmware.com


On Mon, Jul 19, 2010 at 5:08 AM, Stephen Gallagher 
step...@gallagherhome.com wrote:

 On 07/08/2010 07:17 AM, Stephen Gallagher wrote:
  On 07/07/2010 06:04 PM, Christian Hammond wrote:
 
  Do you happen to have the raw profiling log file too? I want to see
  the SQL queries as well.
 
  There were some heavy optimizations made in Djblets for the datagrid
  rendering, but I'm not sure if all that made it into beta 2 or not.
 
  Christian
 
  Sure, attached is the matching .prof file (gzipped since it was rather
  large uncompressed)
 
 *crickets*

 Christian, have you had a chance to look at this? I'm really unsure how
 to track this issue down.

 --
 Want to help the Review Board project? Donate today at
 http://www.reviewboard.org/donate/
 Happy user? Let us know at http://www.reviewboard.org/users/
 -~--~~~~--~~--~--~---
 To unsubscribe from this group, send email to
 reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/reviewboard?hl=en


-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en

Re: ReviewBoard example instance

2010-07-19 Thread Stephen Gallagher
On 07/19/2010 03:00 PM, Christian Hammond wrote:
 Hi Stephen,

 Sorry, I missed the e-mail where you attached the profile log. I'm
 looking through it now.

 The render_to_response was misleading. The slowdown is actually within
 precompute_objects, which does some database queries. That primarily
 does database queries, but the SQL logs show that they're not taking
 very long individually. Certainly not a total of 2 seconds. The
 rendering doesn't appear to be the slow part either.

 I don't know much about PostgreSQL, but it almost sounds like there's
 some issue in the configuration or something. If there's a delay in
 talking to the database (locking? bad connection?) then I could see
 that causing this. The low SQL query time is interesting. That makes
 me wonder if it's just an issue in establishing the first connection.

Well, the problem is that I'm seeing a lag of about 8 seconds for every
page load. I can't imagine the SQL query time being that slow...
Furthermore, there are other Django-based web applications talking to
this same database server without any obvious performance issues.
(Specifically, it's a Transifex instance, so it's low-traffic. That
wouldn't be wasting DB time).

-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


Re: ReviewBoard example instance

2010-07-19 Thread Christian Hammond
There is one query in the list that happens twice and takes 0.352s on
average each time. That's the query to get the auth info from the Fedora
auth backend. I don't know why it'd be so slow, but it might also point to
an issue in talking to the database.

Christian

-- 
Christian Hammond - chip...@chipx86.com
Review Board - http://www.reviewboard.org
VMware, Inc. - http://www.vmware.com


On Mon, Jul 19, 2010 at 12:12 PM, Christian Hammond chip...@chipx86.comwrote:

 The SQL query time isn't that slow, but something definitely is. Especially
 if it's happening on every page. The function that's really taking all that
 time is precompute_objects. It does the following:

 1) Builds a list of fields to sort in order, which will be used in the SQL
 query (this is fairly simple and shouldn't be slow)

 2) Builds a paginator, which is just a simple object used to increment the
 offset in the query. Also simple, part of Django, and used in lots of
 installs.

 3) Fetches the list of IDs for the objects that will be shown in the
 dashboard.

 4) Takes those IDs and uses them to get the actual objects in one batch.

 5) Renders them (the logs show this to not be taking a lot of time).

 So basically, some simple logic to split a string and stuff it into a
 dictionary (which will at most have 2 items, and usually 0 or 1), perform
 the query (which the logs say are actually fairly quick, once the database
 connection is established), and then render it (which the logs say are also
 quick). So I can't imagine it being anything other than the PostgreSQL
 configuration.

 Can you verify that the settings_local.py in Review Board and the
 settings.py in other Django sites are using the exact same database backend?

 Are all the sites running on the same server?

 The logs you provided show a lag of ~2 seconds rather than 8, though. If
 you're seeing 8 from your end, but the logs are showing 2, then there's a
 whole different issue happening, but I'm assuming it just varies on attempt
 and page.


 Christian

 --
 Christian Hammond - chip...@chipx86.com
 Review Board - http://www.reviewboard.org
 VMware, Inc. - http://www.vmware.com


 On Mon, Jul 19, 2010 at 12:04 PM, Stephen Gallagher 
 step...@gallagherhome.com wrote:

 On 07/19/2010 03:00 PM, Christian Hammond wrote:
  Hi Stephen,
 
  Sorry, I missed the e-mail where you attached the profile log. I'm
  looking through it now.
 
  The render_to_response was misleading. The slowdown is actually within
  precompute_objects, which does some database queries. That primarily
  does database queries, but the SQL logs show that they're not taking
  very long individually. Certainly not a total of 2 seconds. The
  rendering doesn't appear to be the slow part either.
 
  I don't know much about PostgreSQL, but it almost sounds like there's
  some issue in the configuration or something. If there's a delay in
  talking to the database (locking? bad connection?) then I could see
  that causing this. The low SQL query time is interesting. That makes
  me wonder if it's just an issue in establishing the first connection.

 Well, the problem is that I'm seeing a lag of about 8 seconds for every
 page load. I can't imagine the SQL query time being that slow...
 Furthermore, there are other Django-based web applications talking to
 this same database server without any obvious performance issues.
 (Specifically, it's a Transifex instance, so it's low-traffic. That
 wouldn't be wasting DB time).

 --
 Want to help the Review Board project? Donate today at
 http://www.reviewboard.org/donate/
 Happy user? Let us know at http://www.reviewboard.org/users/
 -~--~~~~--~~--~--~---
 To unsubscribe from this group, send email to
 reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/reviewboard?hl=en




-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en

Re: ReviewBoard example instance

2010-07-19 Thread Stephen Gallagher
On 07/19/2010 03:12 PM, Christian Hammond wrote:
 Can you verify that the settings_local.py in Review Board and the
 settings.py in other Django sites are using the exact same database
 backend?

 Are all the sites running on the same server?

 The logs you provided show a lag of ~2 seconds rather than 8, though.
 If you're seeing 8 from your end, but the logs are showing 2, then
 there's a whole different issue happening, but I'm assuming it just
 varies on attempt and page.

Well, I think I may be seeing 2s of lag for each component being
displayed on the dashboard, which would account for the 8 seconds total.

I just confirmed that both Transifex and ReviewBoard are both using
postgresql-psycopg2 to talk to the database.

Transifex and ReviewBoard themselves are both running on separate
servers. Transifex has a dedicated machine at the moment, while
ReviewBoard is shared with a number of Trac instances (but the load is
low and there's plenty of free memory).


-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


Re: ReviewBoard example instance

2010-07-07 Thread Stephen Gallagher
On 06/12/2010 03:39 AM, Christian Hammond wrote:
 Hmm, it might be due to not having cProfile installed? Does the
 reviewboard.log show anything?

 Christian

 -- 
 Christian Hammond - chip...@chipx86.com mailto:chip...@chipx86.com
 Review Board - http://www.reviewboard.org
 VMware, Inc. - http://www.vmware.com


 On Fri, Jun 11, 2010 at 1:30 PM, Stephen Gallagher
 step...@gallagherhome.com mailto:step...@gallagherhome.com wrote:

 On 06/10/2010 03:24 PM, Christian Hammond wrote:

 Hi Stephen,

 We're running on MySQL. I don't know why you'd be seeing such
 a huge delay with PostgreSQL though. Can you tell me which
 pages are showing delays? (Is it all of them?)

 if it's the dashboard, there are some known slowdowns when
 reaching large numbers of review requests. If the diff viewer,
 make sure your memcached is caching aggressively.

 One thing you could try doing is going into Settings -
 Logging and turn on Profiling. Then you can visit a page with
 slowdowns and run with ?profiling=1. This should output to a
 reviewboard.prof file in the logging directory with some
 details on the SQL queries and the code paths hit. If you
 could pass that on to me, I could take a look.


 Still working on this, but I thought I should mention. I get a 500
 error on every page that I try to use ?profiling=1 on. Is this a
 known bug on 1.5 beta 2?



Okay, it turned out that I needed to hack python-djblets to use profile
instead of cProfile (since cProfile requires python 2.5 and RHEL5 only
has 2.4)

I got the following profile results (graphed for easier consumption):
http://sgallagh.fedorapeople.org/rb-profile.svg

If I'm reading it right, it looks like the majority of the time is being
lost in grids:696:render_to_response

* That SVG should be viewable in Firefox, but if not I can generate a
PNG as well - it's just much larger.

-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


Re: ReviewBoard example instance

2010-06-11 Thread Stephen Gallagher

On 06/10/2010 03:24 PM, Christian Hammond wrote:

Hi Stephen,

We're running on MySQL. I don't know why you'd be seeing such a huge 
delay with PostgreSQL though. Can you tell me which pages are showing 
delays? (Is it all of them?)


if it's the dashboard, there are some known slowdowns when reaching 
large numbers of review requests. If the diff viewer, make sure your 
memcached is caching aggressively.


One thing you could try doing is going into Settings - Logging and 
turn on Profiling. Then you can visit a page with slowdowns and run 
with ?profiling=1. This should output to a reviewboard.prof file in 
the logging directory with some details on the SQL queries and the 
code paths hit. If you could pass that on to me, I could take a look.


Still working on this, but I thought I should mention. I get a 500 error 
on every page that I try to use ?profiling=1 on. Is this a known bug on 
1.5 beta 2?


--
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


Re: ReviewBoard example instance

2010-06-10 Thread Jan Koprowski
It is probably between SQLite and PostgreSQL. But this is delay is
proportional to scalable. SQLite is not scalable at all, PostgreSQL is
most scalable free SQL database.

On Jun 9, 9:39 pm, Stephen Gallagher step...@gallagherhome.com
wrote:
 I'm curious, what database is thehttp://reviews.reviewboard.orgserver
 using? We deployed on PostgreSQL recently, and I've noticed a distinct
 performance issue with it.

 I can only directly compare it to SQLite, but of course that's local
 access. But we went from  1s responses on webpages to 4-8s per page
 when backed by Postgresql (on a private network with very little traffic).

 How is the performance when backed by MySQL?

-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


Re: ReviewBoard example instance

2010-06-10 Thread Stephen Gallagher
Well, right now 4-8s for a page refresh is just barely within the 
tolerable limits of my users. I'm trying to determine whether it makes 
more sense to migrate to MySQL (if it would be better performance)


I was also curious whether ReviewBoard's reference implementation was 
running on postgresql and whether there were any tuning tricks I should 
be aware of.


On 06/10/2010 02:53 PM, Jan Koprowski wrote:

It is probably between SQLite and PostgreSQL. But this is delay is
proportional to scalable. SQLite is not scalable at all, PostgreSQL is
most scalable free SQL database.

On Jun 9, 9:39 pm, Stephen Gallagherstep...@gallagherhome.com
wrote:
   

I'm curious, what database is thehttp://reviews.reviewboard.orgserver
using? We deployed on PostgreSQL recently, and I've noticed a distinct
performance issue with it.

I can only directly compare it to SQLite, but of course that's local
access. But we went from  1s responses on webpages to 4-8s per page
when backed by Postgresql (on a private network with very little traffic).

How is the performance when backed by MySQL?
 
   


--
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


Re: ReviewBoard example instance

2010-06-10 Thread Christian Hammond
Hi Stephen,

We're running on MySQL. I don't know why you'd be seeing such a huge delay
with PostgreSQL though. Can you tell me which pages are showing delays? (Is
it all of them?)

if it's the dashboard, there are some known slowdowns when reaching large
numbers of review requests. If the diff viewer, make sure your memcached is
caching aggressively.

One thing you could try doing is going into Settings - Logging and turn on
Profiling. Then you can visit a page with slowdowns and run with
?profiling=1. This should output to a reviewboard.prof file in the logging
directory with some details on the SQL queries and the code paths hit. If
you could pass that on to me, I could take a look.

Christian

-- 
Christian Hammond - chip...@chipx86.com
Review Board - http://www.reviewboard.org
VMware, Inc. - http://www.vmware.com


On Thu, Jun 10, 2010 at 11:56 AM, Stephen Gallagher 
step...@gallagherhome.com wrote:

 Well, right now 4-8s for a page refresh is just barely within the tolerable
 limits of my users. I'm trying to determine whether it makes more sense to
 migrate to MySQL (if it would be better performance)

 I was also curious whether ReviewBoard's reference implementation was
 running on postgresql and whether there were any tuning tricks I should be
 aware of.


 On 06/10/2010 02:53 PM, Jan Koprowski wrote:

 It is probably between SQLite and PostgreSQL. But this is delay is
 proportional to scalable. SQLite is not scalable at all, PostgreSQL is
 most scalable free SQL database.

 On Jun 9, 9:39 pm, Stephen Gallagherstep...@gallagherhome.com
 wrote:


 I'm curious, what database is thehttp://reviews.reviewboard.orgserver
 using? We deployed on PostgreSQL recently, and I've noticed a distinct
 performance issue with it.

 I can only directly compare it to SQLite, but of course that's local
 access. But we went from  1s responses on webpages to 4-8s per page
 when backed by Postgresql (on a private network with very little
 traffic).

 How is the performance when backed by MySQL?





 --
 Want to help the Review Board project? Donate today at
 http://www.reviewboard.org/donate/
 Happy user? Let us know at http://www.reviewboard.org/users/
 -~--~~~~--~~--~--~---
 To unsubscribe from this group, send email to
 reviewboard+unsubscr...@googlegroups.comreviewboard%2bunsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/reviewboard?hl=en


-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en

Re: ReviewBoard example instance

2010-06-10 Thread Stephen Gallagher

On 06/10/2010 03:24 PM, Christian Hammond wrote:

Hi Stephen,

We're running on MySQL. I don't know why you'd be seeing such a huge 
delay with PostgreSQL though. Can you tell me which pages are showing 
delays? (Is it all of them?)


Yeah, it's every page but the login page. That includes the admin UI.



if it's the dashboard, there are some known slowdowns when reaching 
large numbers of review requests. If the diff viewer, make sure your 
memcached is caching aggressively.


For the record, our test instance currently has fewer than thirty review 
requests so far, so I guarantee it's not this.




One thing you could try doing is going into Settings - Logging and 
turn on Profiling. Then you can visit a page with slowdowns and run 
with ?profiling=1. This should output to a reviewboard.prof file in 
the logging directory with some details on the SQL queries and the 
code paths hit. If you could pass that on to me, I could take a look.




Yeah, I'm going to do some additional profiling. Something I noticed 
when doing testing on django_evolution (using the test suite you 
directed me to) was that I saw a difference of 7s vs. 45s for the time 
it took to run the suite. This is leading me to believe that the problem 
is more likely to be in python-pyscopg2 or Django itself, rather than 
ReviewBoard.


I'll let you know what I find.

--
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


Re: ReviewBoard example instance

2010-06-10 Thread Jan Koprowski
You can check Your efficient of memcached caching. This could be
critical. Read about getting statistic from memcached:
http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt
this is the fastest element of ReviewBoard.
If You still belive this is database engine bottle neck here is good
article witch talk about Postgres tuning also:
http://jacobian.org/writing/django-performance-tips/


On Jun 10, 8:56 pm, Stephen Gallagher step...@gallagherhome.com
wrote:
 Well, right now 4-8s for a page refresh is just barely within the
 tolerable limits of my users. I'm trying to determine whether it makes
 more sense to migrate to MySQL (if it would be better performance)

 I was also curious whether ReviewBoard's reference implementation was
 running on postgresql and whether there were any tuning tricks I should
 be aware of.

 On 06/10/2010 02:53 PM, Jan Koprowski wrote:



  It is probably between SQLite and PostgreSQL. But this is delay is
  proportional to scalable. SQLite is not scalable at all, PostgreSQL is
  most scalable free SQL database.

  On Jun 9, 9:39 pm, Stephen Gallagherstep...@gallagherhome.com
  wrote:

  I'm curious, what database is thehttp://reviews.reviewboard.orgserver
  using? We deployed on PostgreSQL recently, and I've noticed a distinct
  performance issue with it.

  I can only directly compare it to SQLite, but of course that's local
  access. But we went from  1s responses on webpages to 4-8s per page
  when backed by Postgresql (on a private network with very little traffic).

  How is the performance when backed by MySQL?

-- 
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


Re: ReviewBoard example instance

2010-06-10 Thread David Trowbridge
The reviews.reviewboard.org server is running under mysql. We've done  
a little tuning but not much. I haven't tried with pgsql at all,  
personally.


-David

On Jun 10, 2010, at 11:56 AM, Stephen Gallagher step...@gallagherhome.com 
 wrote:


Well, right now 4-8s for a page refresh is just barely within the  
tolerable limits of my users. I'm trying to determine whether it  
makes more sense to migrate to MySQL (if it would be better  
performance)


I was also curious whether ReviewBoard's reference implementation  
was running on postgresql and whether there were any tuning tricks I  
should be aware of.


On 06/10/2010 02:53 PM, Jan Koprowski wrote:

It is probably between SQLite and PostgreSQL. But this is delay is
proportional to scalable. SQLite is not scalable at all, PostgreSQL  
is

most scalable free SQL database.

On Jun 9, 9:39 pm, Stephen Gallagherstep...@gallagherhome.com
wrote:

I'm curious, what database is thehttp:// 
reviews.reviewboard.orgserver
using? We deployed on PostgreSQL recently, and I've noticed a  
distinct

performance issue with it.

I can only directly compare it to SQLite, but of course that's local
access. But we went from  1s responses on webpages to 4-8s per page
when backed by Postgresql (on a private network with very little  
traffic).


How is the performance when backed by MySQL?





--
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en


--
Want to help the Review Board project? Donate today at 
http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~--~~~~--~~--~--~---
To unsubscribe from this group, send email to 
reviewboard+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/reviewboard?hl=en