Re: [PERFORM] Question about shared_buffers and cpu usage

2008-02-20 Thread Scott Marlowe
On Wed, Feb 20, 2008 at 11:13 PM, bh yuan <[EMAIL PROTECTED]> wrote: > Hi > > I am using Postgres8.3 on 8G memory , Xeon X5355 Quad Core x 2 > processer RH5 machine with 10G data. (with some table which have > about 2,000,000~ 5,000,000 rows ) > > I have two quesion. > 1. how to set the share

[PERFORM] planner favors seq scan too early

2008-02-20 Thread Markus Bertheau
Given the following query: SELECT fi.pub_date FROM ext_feeder_item fi WHERE fi.feed_id IN (SELECT id FROM ext_feeder_feed ff WHERE ff.is_system) ORDER BY pub_date DESC; I'm getting a plan that uses a sequential scan on ext_feeder_item instead of several index sc

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Chris
Gregory Stark wrote: "Chris" <[EMAIL PROTECTED]> writes: When I said "obfuscating" I meant it. I'm pretty familiar with sh scripting and I'm not even sure what the && behaviour would do. It chains commands together so if the first fails the second doesn't happen. I meant in this case, not in

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Gregory Stark
"Chris" <[EMAIL PROTECTED]> writes: >> When I said "obfuscating" I meant it. I'm pretty familiar with sh scripting >> and I'm not even sure what the && behaviour would do. > > It chains commands together so if the first fails the second doesn't happen. I meant in this case, not in general. That i

[PERFORM] Question about shared_buffers and cpu usage

2008-02-20 Thread bh yuan
Hi I am using Postgres8.3 on 8G memory , Xeon X5355 Quad Core x 2 processer RH5 machine with 10G data. (with some table which have about 2,000,000~ 5,000,000 rows ) I have two quesion. 1. how to set the shared_buffers and other postgresql.conf parameter for best performance? I only run the Postg

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Chris
When I said "obfuscating" I meant it. I'm pretty familiar with sh scripting and I'm not even sure what the && behaviour would do. It chains commands together so if the first fails the second doesn't happen. $ echo 1 && echo 2 1 2 $ echo '1234' > /etc/file_that_doesnt_exist && echo 2 -bash: /

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > Erik Jones <[EMAIL PROTECTED]> writes: >> On Feb 20, 2008, at 8:14 AM, Gregory Stark wrote: >>> I would suggest leaving out the && which only obfuscate what's >>> going on here. >>> >>> PGOPTIONS=... pg_restore ... >>> >>> would work just as well and be

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Tom Lane
Matthew <[EMAIL PROTECTED]> writes: > We have a process here that dumps a large quantity of data into an empty > database, much like pg_restore, and then creates all the indexes at the > end. In order to speed up that bit, I initially made it spawn off several > threads, and make each thread run

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Jeff Davis
On Wed, 2008-02-20 at 18:18 +, Matthew wrote: > On Wed, 20 Feb 2008, Jeff Davis wrote: > > However, building indexes in parallel would allow better CPU > > utilization. > > We have a process here that dumps a large quantity of data into an empty > database, much like pg_restore, and then crea

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Matthew
On Wed, 20 Feb 2008, Jeff Davis wrote: However, building indexes in parallel would allow better CPU utilization. We have a process here that dumps a large quantity of data into an empty database, much like pg_restore, and then creates all the indexes at the end. In order to speed up that bit,

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Jeff Davis
On Wed, 2008-02-20 at 14:31 +0530, Pavan Deolasee wrote: > I think it would be interesting if we can build these indexes in parallel. > Each index build requires a seq scan on the table. If the table does > not fit in shared buffers, each index build would most likely result > in lots of IO. He's

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Erik Jones
On Feb 20, 2008, at 10:54 AM, Tom Lane wrote: Erik Jones <[EMAIL PROTECTED]> writes: On Feb 20, 2008, at 8:14 AM, Gregory Stark wrote: I would suggest leaving out the && which only obfuscate what's going on here. PGOPTIONS=... pg_restore ... would work just as well and be clearer about what

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Matthew
On Wed, 20 Feb 2008, Tom Lane wrote: Erik Jones <[EMAIL PROTECTED]> writes: On Feb 20, 2008, at 8:14 AM, Gregory Stark wrote: I would suggest leaving out the && which only obfuscate what's going on here. PGOPTIONS=... pg_restore ... would work just as well and be clearer about what's going on

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Tom Lane
Erik Jones <[EMAIL PROTECTED]> writes: > On Feb 20, 2008, at 8:14 AM, Gregory Stark wrote: >> I would suggest leaving out the && which only obfuscate what's >> going on here. >> >> PGOPTIONS=... pg_restore ... >> >> would work just as well and be clearer about what's going on. > Right, that's

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Erik Jones
On Feb 20, 2008, at 8:14 AM, Gregory Stark wrote: "Douglas J Hunley" <[EMAIL PROTECTED]> writes: On Tuesday 19 February 2008 16:32:02 Erik Jones wrote: pg_restore is a postgres client app that uses libpq to connect and, thus, will pick up anything in your $PGOPTIONS env variable. So, PGOPT

Re: [PERFORM] Anyone using a SAN?

2008-02-20 Thread Michael Stone
On Wed, Feb 20, 2008 at 02:52:42PM +0100, C. Bergström wrote: Can be, but may I point to a recent posting on Beowulf ml [1] and the article it references [2] Showing that the per node price of SDR IB has come down far enough to in some cases compete with GigE. ymmv, but I'm in the planning phase

Re: [PERFORM] Anyone using a SAN?

2008-02-20 Thread Michael Stone
On Mon, Feb 18, 2008 at 03:44:40PM -0600, Peter Koczan wrote: One big reason we're really looking into a SAN option is that we have a lot of unused disk space. The cost of the SAN interfaces probably exceeds the cost of the wasted space, and the performance will probably be lower for a lot of

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Gregory Stark
"Douglas J Hunley" <[EMAIL PROTECTED]> writes: > On Tuesday 19 February 2008 16:32:02 Erik Jones wrote: >> pg_restore is a postgres client app that uses libpq to connect and,   >> thus, will pick up anything in your $PGOPTIONS env variable.  So, >> >> PGOPTONS="-c maintenance_work_mem=512MB" && pg

Re: [PERFORM] Anyone using a SAN?

2008-02-20 Thread C.
On Wed, 2008-02-20 at 13:41 +, Matthew wrote: > On Mon, 18 Feb 2008, Peter Koczan wrote: > > One of the other things I was interested in was the "hidden costs" of > > a SAN. For instance, we'd probably have to invest in more UPS capacity > > to protect our data. Are there any other similar poi

Re: [PERFORM] Anyone using a SAN?

2008-02-20 Thread Matthew
On Mon, 18 Feb 2008, Peter Koczan wrote: One of the other things I was interested in was the "hidden costs" of a SAN. For instance, we'd probably have to invest in more UPS capacity to protect our data. Are there any other similar points that people don't initially consider regarding a SAN? You

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Douglas J Hunley
On Tuesday 19 February 2008 17:53:45 Greg Smith wrote: > On Tue, 19 Feb 2008, Douglas J Hunley wrote: > > The db resides on a HP Modular Storage Array 500 G2. 4x72.8Gb 15k rpm > > disks. 1 raid 6 logical volume. Compaq Smart Array 6404 controller > > You might consider doing some simple disk tests

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Douglas J Hunley
On Tuesday 19 February 2008 16:32:02 Erik Jones wrote: > pg_restore is a postgres client app that uses libpq to connect and,   > thus, will pick up anything in your $PGOPTIONS env variable.  So, > > PGOPTONS="-c maintenance_work_mem=512MB" && pg_restore now that's just plain cool /me updates

Re: [PERFORM] Need Help selecting Large Data From PQSQL

2008-02-20 Thread Richard Huxton
[EMAIL PROTECTED] wrote: The report contains more than 70,000 records but it takes more than half an hour and some time no result. Please help me to generating the report fast. You'll need to provide some more information before anyone can help. Something along the lines of:

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread ohp
On Wed, 20 Feb 2008, Pavan Deolasee wrote: > Date: Wed, 20 Feb 2008 14:31:09 +0530 > From: Pavan Deolasee <[EMAIL PROTECTED]> > To: Jeff Davis <[EMAIL PROTECTED]> > Cc: Douglas J Hunley <[EMAIL PROTECTED]>, > pgsql-performance@postgresql.org > Subject: Re: [PERFORM] 7 hrs for a pg_restore? >

Re: [PERFORM] mis-estimate in nested query causes slow runtimes

2008-02-20 Thread Chris Kratz
On 2/18/08, Chris Kratz <[EMAIL PROTECTED]> wrote: > > On 2/11/08, Tom Lane <[EMAIL PROTECTED]> wrote: > > > > Chris Kratz <[EMAIL PROTECTED]> writes: > > > The first frustration is that I can't get the transaction details scan > > > to get any more accurate. It thinks it will find 1407 records, >

[PERFORM] Need Help selecting Large Data From PQSQL

2008-02-20 Thread shilpa.raghavendra
Hi, I am using Postgresql 8.1 for handling large data. I am having One Parent Table and Child Table I.e.inherits from parent table. The constraint for partitioning table is date range. I have to generate monthly report, report generation query contains union with

Re: [PERFORM] 7 hrs for a pg_restore?

2008-02-20 Thread Pavan Deolasee
On Feb 19, 2008 11:53 PM, Jeff Davis <[EMAIL PROTECTED]> wrote: > > > Keep in mind, if you have several GB worth of indexes, they take up > basically no space in the logical dump (just the "CREATE INDEX" command, > and that's it). But they can take a lot of processor time to build up > again, espec

Re: [PERFORM] Anyone using a SAN?

2008-02-20 Thread Sven Geisler
Hi Peter, Peter Koczan schrieb: One of the other things I was interested in was the "hidden costs" of a SAN. For instance, we'd probably have to invest in more UPS capacity to protect our data. Are there any other similar points that people don't initially consider regarding a SAN? There