Re: [PERFORM] Large number of short lived connections - could a connection pool help?

2011-11-15 Thread Mario Weilguni
Am 15.11.2011 01:42, schrieb Cody Caughlan: We have anywhere from 60-80 background worker processes connecting to Postgres, performing a short task and then disconnecting. The lifetime of these tasks averages 1-3 seconds. I know that there is some connection overhead to Postgres, but I dont

Re: [PERFORM] avoiding seq scans when two columns are very correlated

2011-11-15 Thread Ruslan Zakirov
On Tue, Nov 15, 2011 at 11:08 AM, Stuart Bishop stu...@stuartbishop.net wrote: On Fri, Nov 11, 2011 at 10:01 PM, Ruslan Zakirov r...@bestpractical.com wrote: Hello, [snip] In application such wrong estimation result in seq scan of this table winning leading position in execution plans over

Re: [PERFORM] What's the state of postgresql on ext4 now?

2011-11-15 Thread Josh Berkus
On 11/14/11 2:00 AM, Alexandru wrote: I know there were a lot of performance issues with ext4, but i don't know the state of it now. I have a private openstreetmap server installed on a ubuntu 11.10 64bit pc with both partitions (/ and /home) formated with ext4. My problem is that the

Re: [PERFORM] unlogged tables

2011-11-15 Thread Josh Berkus
Unlogged table can increase speed, this table has about 1.6 millions of update per hour, but unlogged with a chance of loss all information on a crash are not a good idea for this. pg_dump -t 'tablename' from a cron job? (Make sure to rotate dump file names, maybe with day of week or

Re: [PERFORM] What's the state of postgresql on ext4 now?

2011-11-15 Thread Scott Marlowe
On Tue, Nov 15, 2011 at 1:48 PM, Josh Berkus j...@agliodbs.com wrote: On 11/14/11 2:00 AM, Alexandru wrote: I know there were a lot of performance issues with ext4, but i don't know the state of it now. I have a private openstreetmap server installed on a ubuntu 11.10 64bit pc with both

Re: [PERFORM] Slow queries / commits, mis-configuration or hardware issues?

2011-11-15 Thread Tomas Vondra
Dne 15.11.2011 01:13, Cody Caughlan napsal(a): The first two are what I would think would be largely read operations (certainly the SELECT) so its not clear why a SELECT consumes write time. Here is the output of some pg_stat_bgwriter stats from the last couple of hours:

Re: [PERFORM] Slow queries / commits, mis-configuration or hardware issues?

2011-11-15 Thread Tomas Vondra
Dne 14.11.2011 22:58, Cody Caughlan napsal(a): I ran bonnie++ on a slave node, doing active streaming replication but otherwise idle: http://batch-files-test.s3.amazonaws.com/sql03.prod.html bonnie++ on the master node: http://batch-files-test.s3.amazonaws.com/sql01.prod.html If I am

Re: [PERFORM] Slow queries / commits, mis-configuration or hardware issues?

2011-11-15 Thread Cody Caughlan
On Tue, Nov 15, 2011 at 5:16 PM, Tomas Vondra t...@fuzzy.cz wrote: Dne 14.11.2011 22:58, Cody Caughlan napsal(a): I ran bonnie++ on a slave node, doing active streaming replication but otherwise idle: http://batch-files-test.s3.amazonaws.com/sql03.prod.html bonnie++ on the master node:

Re: [PERFORM] Slow queries / commits, mis-configuration or hardware issues?

2011-11-15 Thread Tomas Vondra
On 16 Listopad 2011, 2:21, Cody Caughlan wrote: How did you build your RAID array? Maybe I have a fundamental flaw / misconfiguration. I am doing it via: $ yes | mdadm --create /dev/md0 --level=10 -c256 --raid-devices=4 /dev/xvdb /dev/xvdc /dev/xvdd /dev/xvde $ pvcreate /dev/md0 $ vgcreate

Re: [PERFORM] What's the state of postgresql on ext4 now?

2011-11-15 Thread Greg Smith
On 11/14/2011 05:00 AM, Alexandru wrote: I know there were a lot of performance issues with ext4, but i don't know the state of it now. I have a private openstreetmap server installed on a ubuntu 11.10 64bit pc with both partitions (/ and /home) formated with ext4. My problem is that the

Re: [PERFORM] What's the state of postgresql on ext4 now?

2011-11-15 Thread Scott Marlowe
On Tue, Nov 15, 2011 at 8:48 PM, Greg Smith g...@2ndquadrant.com wrote: When you turn off the synchronous_commit parameter in the postgresql.conf, the database will stop asking the filesystem to ensure things are on disk this way.  You can lose some data in the event of a crash, but things will

Re: [PERFORM] What's the state of postgresql on ext4 now?

2011-11-15 Thread Aidan Van Dyk
On Tue, Nov 15, 2011 at 10:48 PM, Greg Smith g...@2ndquadrant.com wrote:   In just about every other way but commit performance, ext4 is faster than most other filesystems. As someone who is looked at as an expert and knowledgable my many of us, are

Re: [PERFORM] Slow queries / commits, mis-configuration or hardware issues?

2011-11-15 Thread Greg Smith
On 11/14/2011 01:16 PM, Cody Caughlan wrote: We're starting to see some slow queries, especially COMMITs that are happening more frequently. The slow queries are against seemingly well-indexed tables. Slow commits like: 2011-11-14 17:47:11 UTC pid:14366 (44/0-0) LOG: duration: 3062.784 ms

Re: [PERFORM] What's the state of postgresql on ext4 now?

2011-11-15 Thread Greg Smith
On 11/15/2011 10:54 PM, Aidan Van Dyk wrote: are you getting to the point of migrating large XFS filesystems to ext4 for production databases yet? Or at least using ext4 in new large-scale filesystems for PG? Not really. Last time I checked (a few months ago), there were still some