Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Heikki Linnakangas
Douglas J Hunley wrote: On Sunday 03 June 2007 16:39:51 Luke Lonergan wrote: When you initdb, a config file is edited from the template by initdb to reflect your machine config. I didn't realize that. I'll have to harass the rest of the team to see if someone overwrote that file or not. In

Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Luke Lonergan
Absolutely! A summary of relevant comments so far are: - enable-mergejoin - shared-buffers - fsync Another to consider if you use indexes is random-page-cost. What would be helpful is if you could identify a slow query and post the explain analyze here. The concurrent performance of many

Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Douglas J Hunley
On Sunday 03 June 2007 16:39:51 Luke Lonergan wrote: When you initdb, a config file is edited from the template by initdb to reflect your machine config. I didn't realize that. I'll have to harass the rest of the team to see if someone overwrote that file or not. In the interim, I did an

Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Reid Thompson
On Sun, 2007-06-03 at 23:17 -0400, Tom Lane wrote: Douglas J Hunley [EMAIL PROTECTED] writes: On Saturday 02 June 2007 11:25:11 Tom Lane wrote: Another thing that seems strange is that the 8.2 config file does not seem to have been processed by initdb --- or did you explicitly comment out

[PERFORM] dbt2 NOTPM numbers

2007-06-04 Thread Markus Schiltknecht
Hi, I'm currently playing with dbt2 and am wondering, if the results I'm getting are reasonable. I'm testing a 2x Dual Core Xeon system with 4 GB of RAM and 8 SATA HDDs attached via Areca RAID Controller w/ battery backed write cache. Seven of the eight platters are configured as one RAID6,

Re: [PERFORM] dbt2 NOTPM numbers

2007-06-04 Thread Heikki Linnakangas
Markus Schiltknecht wrote: I'm currently playing with dbt2 and am wondering, if the results I'm getting are reasonable. I'm testing a 2x Dual Core Xeon system with 4 GB of RAM and 8 SATA HDDs attached via Areca RAID Controller w/ battery backed write cache. Seven of the eight platters are

Re: [PERFORM] dbt2 NOTPM numbers

2007-06-04 Thread Markus Schiltknecht
Hi, Heikki Linnakangas wrote: There's clearly something wrong. The response times are ridiculously high, they should be 5 seconds (except for stock level transaction) to pass a TPC-C test. I wonder if you built any indexes at all? Hm.. according to the output/5/db/plan0.out, all queries use

Re: [PERFORM] dbt2 NOTPM numbers

2007-06-04 Thread PFC
I'll run a bonnie++ first. As the CPUs seem to be idle most of the time (see the vmstat.out below), I'm suspecting the RAID or disks. You have a huge amount of iowait ! Did you put the xlog on a separate disk ? What filesystem do you use ? Did you check that

Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Douglas J Hunley
On Sunday 03 June 2007 18:30:17 Greg Smith wrote: To be equivalent to the 7.4 config the 8.2 config would need: I've taken all the wonderful advise offered thus far, and put the attached into use. Our initial testing shows a 66% improvement in page load times for our app. I have the customer

[PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Thomas Andrews
I have several thousand clients. Our clients do surveys, and each survey has two tables for the client data, responders responses Frequent inserts into both table. Right now, we are seeing significant time during inserts to these two tables. Some of the indices in tableA and tableB do

Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Alvaro Herrera
Douglas J Hunley wrote: On a side note, is there any real benefit to using autovacuum over a periodically scheduled vacuum? I ask because we have the latter already coded up and cron'd and it seems to keep things fairly optimized. No, not really. Maybe autovacuum could get to specific

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Mark Lewis
On Mon, 2007-06-04 at 13:40 -0400, Thomas Andrews wrote: I have several thousand clients. Our clients do surveys, and each survey has two tables for the client data, responders responses Frequent inserts into both table. Right now, we are seeing significant time during inserts

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Thomas Andrews
Oh, and we vacuum every day. Not sure about REINDEX, but I doubt we have done that. =thomas Mark Lewis wrote: On Mon, 2007-06-04 at 13:40 -0400, Thomas Andrews wrote: I have several thousand clients. Our clients do surveys, and each survey has two tables for the client data, responders

Re: [PERFORM] dbt2 NOTPM numbers

2007-06-04 Thread Markus Schiltknecht
Hi, PFC wrote: You have a huge amount of iowait ! Yup. Did you put the xlog on a separate disk ? No, it's all one big RAID6 for the sake of simplicity (plus I doubt somewhat, that 2 disks for WAL + 5 for data + 1 spare would be much faster than 7 disks for WAL and data + 1 spare

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Thomas Andrews
We're running 7.4 but will be upgrading to 8.2. The responses table has 20,000,000 records. Sometimes (but not all the time) an insert into the responses table can take 5-6 seconds. I guess my real question is, does it ever make sense to create thousands of tables like this? =thomas Mark

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Gregory Stark
Thomas Andrews [EMAIL PROTECTED] writes: I guess my real question is, does it ever make sense to create thousands of tables like this? Sometimes. But usually it's not a good idea. What you're proposing is basically partitioning, though you may not actually need to put all the partitions

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Thomas Andrews
On 6/4/07 3:43 PM, Gregory Stark [EMAIL PROTECTED] wrote: Thomas Andrews [EMAIL PROTECTED] writes: I guess my real question is, does it ever make sense to create thousands of tables like this? Sometimes. But usually it's not a good idea. What you're proposing is basically

Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Douglas J Hunley
On Saturday 02 June 2007 11:21:41 Michael Fuhr wrote: If you post an example query and the EXPLAIN ANALYZE output then we might be able to see if the slowness is due to query plans. Query 1: (SELECT project.path AS rbac_project_path_string, role_operation.resource_name AS

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Y Sidhu
On 6/4/07, Thomas Andrews [EMAIL PROTECTED] wrote: On 6/4/07 3:43 PM, Gregory Stark [EMAIL PROTECTED] wrote: Thomas Andrews [EMAIL PROTECTED] writes: I guess my real question is, does it ever make sense to create thousands of tables like this? Sometimes. But usually it's not a good

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Thomas Andrews
Linux 2.4.9, if I¹m reading this right. =thomas On 6/4/07 4:08 PM, Y Sidhu [EMAIL PROTECTED] wrote: On 6/4/07, Thomas Andrews [EMAIL PROTECTED] wrote: On 6/4/07 3:43 PM, Gregory Stark [EMAIL PROTECTED] wrote: Thomas Andrews [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] writes:

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Gregory Stark
Thomas Andrews [EMAIL PROTECTED] writes: Clustering sounds like it might be a really good solution. How long does a cluster command usually take on a table with 50,000,000 records? Is it something that can be run daily/weekly? ouch, ok, with 50M records cluster isn't going to be quick

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread PFC
can tell you our vacuum every night is taking 2 hours and that disk IO is the real killer - the CPU rarely gets higher than 20% or so. How many gigabytes of stuff do you have in this database ? ( du -sh on the *right* directory will suffice, don't include the logs etc, aim for

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread Scott Marlowe
Gregory Stark wrote: Thomas Andrews [EMAIL PROTECTED] writes: I guess my real question is, does it ever make sense to create thousands of tables like this? Sometimes. But usually it's not a good idea. What you're proposing is basically partitioning, though you may not actually

Re: [PERFORM] dbt2 NOTPM numbers

2007-06-04 Thread Heikki Linnakangas
Markus Schiltknecht wrote: Hi, Heikki Linnakangas wrote: There's clearly something wrong. The response times are ridiculously high, they should be 5 seconds (except for stock level transaction) to pass a TPC-C test. I wonder if you built any indexes at all? Hm.. according to the

Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Gregory Stark
Those plans look like they have a lot of casts to text in them. How have you defined your indexes? Are your id columns really text? And you don't have a 7.4 install around to compare the plans do you? -- Gregory Stark EnterpriseDB http://www.enterprisedb.com

Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Heikki Linnakangas
Gregory Stark wrote: Those plans look like they have a lot of casts to text in them. How have you defined your indexes? Are your id columns really text? And did you use the same encoding and locale? Text operations on multibyte encodings are much more expensive. -- Heikki Linnakangas

Re: [PERFORM] upgraded to pgsql 8.2.4, getting worse performance then 7.4.x

2007-06-04 Thread Kenneth Marshall
On Sat, Jun 02, 2007 at 09:13:32AM -0400, Douglas J Hunley wrote: Hello great gurus of performance: Our 'esteemed' Engr group recently informed a customer that in their testing, upgrading to 8.2.x improved the performance of our J2EE application approximately 20%, so of course, the customer

[PERFORM] PostgreSQL not fully utilizing system resources?

2007-06-04 Thread Gregory Stewart
Hello List, We've been running PostgreSQL as our web application database for almost a year and it has noticeably slowed down over last few months. Our current setup and pgsql configuration looks like this: 8.1.2 on Ubuntu 4 on Opteron Dual Core with 2 GBytes RAM. This is a dedicated DB

Re: [PERFORM] PostgreSQL not fully utilizing system resources?

2007-06-04 Thread Bill Moran
Gregory Stewart [EMAIL PROTECTED] wrote: Hello List, We've been running PostgreSQL as our web application database for almost a year and it has noticeably slowed down over last few months. Just going to go through your email and address each point inline. First off, you say nothing of your

Re: [PERFORM] Thousands of tables versus on table?

2007-06-04 Thread david
On Mon, 4 Jun 2007, Scott Marlowe wrote: Gregory Stark wrote: Thomas Andrews [EMAIL PROTECTED] writes: I guess my real question is, does it ever make sense to create thousands of tables like this? Sometimes. But usually it's not a good idea. What you're proposing is

Re: [PERFORM] PostgreSQL not fully utilizing system resources?

2007-06-04 Thread Greg Smith
On Fri, 1 Jun 2007, Gregory Stewart wrote: Is our configuration flawed? For sure. The bad news is that you have a good chunk of work to do; the good news is that you should quickly see a dramatic improvement as that progresses. Anyways, watching the system processes we realized that

[PERFORM] postgresql running on a virtual cluster

2007-06-04 Thread Dave Cramer
Does anyone have any experience running pg on multiple IBM 3950's set up as a single machine ? Dave ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED]

[PERFORM] Question about SQL performance

2007-06-04 Thread Jason Lustig
I have some questions about the performance of certain types of SQL statements. What sort of speed increase is there usually with binding parameters (and thus preparing statements) v. straight sql with interpolated variables? Will Postgresql realize that the following queries are