Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Greg Smith
On Tue, 12 Aug 2008, Ron Mayer wrote: Really old software (notably 2.4 linux kernels) didn't send cache synchronizing commands for SCSI nor either ATA; but it seems well thought through in the 2.6 kernels as described in the Linux kernel documentation.

Re: [PERFORM] long transaction

2008-08-13 Thread Sabin Coanda
have you considered importing to a temporary 'holding' table with copy, then doing 'big' sql statements on it to check constraints, etc? Yes I considered it, but the problem is the data is very tight related between different tables and is important to keep the import order of each entity

Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Scott Marlowe
On Tue, Aug 12, 2008 at 10:28 PM, Ron Mayer [EMAIL PROTECTED] wrote: Scott Marlowe wrote: I can attest to the 2.4 kernel not being able to guarantee fsync on IDE drives. Sure. But note that it won't for SCSI either; since AFAICT the write barrier support was implemented at the same time

Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Matthew Wakeling
On Tue, 12 Aug 2008, Ron Mayer wrote: Really old software (notably 2.4 linux kernels) didn't send cache synchronizing commands for SCSI nor either ATA; Surely not true. Write cache flushing has been a known problem in the computer science world for several tens of years. The difference is

Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Ron Mayer
Scott Marlowe wrote: On Tue, Aug 12, 2008 at 10:28 PM, Ron Mayer ...wrote: Scott Marlowe wrote: I can attest to the 2.4 kernel ... ...SCSI...AFAICT the write barrier support... Tested both by pulling the power plug. The SCSI was pulled 10 times while running 600 or so concurrent pgbench

Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Ron Mayer
Greg Smith wrote: The below disk writes impossibly fast when I issue a sequence of fsync 'k. I've got some homework. I'll be trying to reproduce similar with md raid, old IDE drives, etc to see if I can reproduce them. I assume test_fsync in the postgres source distribution is a decent way to

Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Scott Marlowe
On Wed, Aug 13, 2008 at 8:41 AM, Ron Mayer [EMAIL PROTECTED] wrote: Greg Smith wrote: But I still am looking for any evidence that there were any widely shipped SATA (or even IDE drives) that were at fault, as opposed to filesystem bugs and poor settings of defaults. Well, if they're getting

Re: [PERFORM] query plan, index scan cost

2008-08-13 Thread Decibel!
On Jul 18, 2008, at 5:28 AM, Stefan Zweig wrote: CREATE TABLE nw_tla_2008_4_deu ( ID bigint NOT NULL, NET2CLASS smallint, FOW smallint, CONSTRAINT nw_tla_2008_4_deu_pkey PRIMARY KEY (ID), ) WITHOUT OIDS; You might want to give up on the double-quotes... you'll have to use them everywhere.

Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Decibel!
On Aug 11, 2008, at 9:01 AM, Jeff wrote: On Aug 11, 2008, at 5:17 AM, Henrik wrote: OK, changed the SAS RAID 10 to RAID 5 and now my random writes are handing 112 MB/ sek. So it is almsot twice as fast as the RAID10 with the same disks. Any ideas why? Is the iozone tests faulty? does

Re: [PERFORM] Incorrect estimates on correlated filters

2008-08-13 Thread Decibel!
On Aug 12, 2008, at 4:59 PM, Chris Kratz wrote: Ran into a re-occuring performance problem with some report queries again today. In a nutshell, we have filters on either multiple joined tables, or multiple columns on a single table that are highly correlated. So, the estimates come out

Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Greg Smith
On Wed, 13 Aug 2008, Ron Mayer wrote: I assume test_fsync in the postgres source distribution is a decent way to see? Not really. It takes too long (runs too many tests you don't care about) and doesn't spit out the results the way you want them--TPS, not average time. You can do it with

Re: [PERFORM] Incorrect estimates on correlated filters

2008-08-13 Thread Chris Kratz
On Wed, Aug 13, 2008 at 10:59 AM, Decibel! [EMAIL PROTECTED] wrote: On Aug 12, 2008, at 4:59 PM, Chris Kratz wrote: Ran into a re-occuring performance problem with some report queries again today. In a nutshell, we have filters on either multiple joined tables, or multiple columns on a

Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Henrik
13 aug 2008 kl. 17.13 skrev Decibel!: On Aug 11, 2008, at 9:01 AM, Jeff wrote: On Aug 11, 2008, at 5:17 AM, Henrik wrote: OK, changed the SAS RAID 10 to RAID 5 and now my random writes are handing 112 MB/ sek. So it is almsot twice as fast as the RAID10 with the same disks. Any ideas

Re: [PERFORM] Incorrect estimates on correlated filters

2008-08-13 Thread Alvaro Herrera
Chris Kratz wrote: Unfortunately, if I don't think the sorting idea would help in the one case I'm looking at which involves filters on two tables that are joined together. The filters happen to be correlated such that about 95% of the rows from each filtered table are actually returned

[PERFORM] autovacuum: use case for indenpedent TOAST table autovac settings

2008-08-13 Thread Alvaro Herrera
Hackers and PG users, Does anyone see a need for having TOAST tables be individually configurable for autovacuum? I've finally come around to looking at being able to use ALTER TABLE for autovacuum settings, and I'm wondering if we need to support that case. -- Alvaro Herrera

Re: [PERFORM] Filesystem benchmarking for pg 8.3.3 server

2008-08-13 Thread Ron Mayer
Scott Marlowe wrote: IDE came up corrupted every single time. Greg Smith wrote: you've drank the kool-aid ... completely ridiculous ...unsafe fsync ... md0 RAID-1 array (aren't there issues with md and the barriers?) Alright - I'll eat my words. Or mostly. I still haven't found IDE drives

Re: [PERFORM] autovacuum: use case for indenpedent TOAST table autovac settings

2008-08-13 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: Does anyone see a need for having TOAST tables be individually configurable for autovacuum? I've finally come around to looking at being able to use ALTER TABLE for autovacuum settings, and I'm wondering if we need to support that case. It seems like

Re: [HACKERS] [PERFORM] autovacuum: use case for indenpedent TOAST table autovac settings

2008-08-13 Thread Alvaro Herrera
Tom Lane wrote: Alvaro Herrera [EMAIL PROTECTED] writes: Does anyone see a need for having TOAST tables be individually configurable for autovacuum? I've finally come around to looking at being able to use ALTER TABLE for autovacuum settings, and I'm wondering if we need to support that

Re: [HACKERS] [PERFORM] autovacuum: use case for indenpedent TOAST table autovac settings

2008-08-13 Thread Tom Lane
Alvaro Herrera [EMAIL PROTECTED] writes: Tom Lane wrote: It seems like we'll want to do it somehow. Perhaps the cleanest way is to incorporate toast-table settings in the reloptions of the parent table. Otherwise dump/reload is gonna be a mess. My question is whether there is interest in

Re: [PERFORM] Incorrect estimates on correlated filters

2008-08-13 Thread Craig Ringer
Decibel! wrote: Well... you could try and convince certain members of the community that we actually do need some kind of a query hint mechanism... ;) It strikes me that there are really two types of query hint possible here. One tells the planner (eg) prefer a merge join here. The other