[PERFORM] Massive update, memory usage

2010-10-28 Thread Trenta sis
There are about 100.000 inserts and 30 updates. Without transaction it seems that works, but with a transaction no. Witt about only 300.000 updates it seems that can finish correctly, but last 20% is slow because is using swap... Any tunning to do in this configuration or it is correct?

Re: [PERFORM] Select count(*), the sequel

2010-10-28 Thread Pierre C
Pierre C li...@peufeu.com wrote: in-page compression How would that be different from the in-page compression done by TOAST now? Or are you just talking about being able to make it more aggressive? -Kevin Well, I suppose lzo-style compression would be better used on data that is written a

Re: [PERFORM] AIX slow buffer reads

2010-10-28 Thread André Volpato
- Mensagem original - | André Volpato wrote: | | | | If it is being spent in the bitmap index scan, try setting | | effective_io_concurrency to 0 for Linux, and see what effect that | | has. | | I disabled effective_io_concurrency at AIX but it made no changes on | bitmap index

Re: [PERFORM] Postgres insert performance and storage requirement compared to Oracle

2010-10-28 Thread Merlin Moncure
On Wed, Oct 27, 2010 at 3:47 PM, Jesper Krogh jes...@krogh.cc wrote: On 2010-10-27 20:51, Merlin Moncure wrote: Yes, I am quite aware of how the o/s page cache works.  All else being equal, I more compact database obviously would be preferred.  However 'all else' is not necessarily equal.  I

Re: [PERFORM] Select count(*), the sequel

2010-10-28 Thread Kenneth Marshall
On Wed, Oct 27, 2010 at 05:49:42PM -0400, Tom Lane wrote: Kenneth Marshall k...@rice.edu writes: Just keeping the hope alive for faster compression. Is there any evidence that that's something we should worry about? I can't recall ever having seen a code profile that shows the

Re: [PERFORM] temporary tables, indexes, and query plans

2010-10-28 Thread Jon Nelson
On Wed, Oct 27, 2010 at 5:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jon Nelson jnelson+pg...@jamponi.net writes: I'd like to zoom out a little bit and, instead of focusing on the specifics, ask more general questions: .. - is there some way for me to determine /why/ the planner chooses a

Re: [PERFORM] temporary tables, indexes, and query plans

2010-10-28 Thread Tom Lane
Jon Nelson jnelson+pg...@jamponi.net writes: On Wed, Oct 27, 2010 at 5:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: It thinks it's faster, or there is some reason why it *can't* use the index, like a datatype mismatch.  You could tell which by trying set enable_seqscan = off to see if that will

Re: [PERFORM] Slow Query- Simple taking

2010-10-28 Thread Robert Haas
On Tue, Oct 19, 2010 at 2:21 PM, Ozer, Pam po...@automotive.com wrote: I have the following query running on 8.4, which takes 3516 ms.  It is very straight forward.  It brings back 116412 records.  The explain only takes 1348ms select VehicleUsed.VehicleUsedId as VehicleUsedId ,

Re: [PERFORM] Slow Query- Simple taking

2010-10-28 Thread Robert Haas
On Thu, Oct 28, 2010 at 10:39 AM, Robert Haas robertmh...@gmail.com wrote: Can someone tell me why after it runs the index scan it hen runs a bitmap heap scan?  It should not take this long to run should it?  If I limit the results it comes back in 300ms. It doesn't.  The EXPLAIN output shows

Re: [PERFORM] Slow Query- Simple taking

2010-10-28 Thread Robert Haas
On Tue, Oct 19, 2010 at 6:05 PM, Ozer, Pam po...@automotive.com wrote: On mysql the same query only takes milliseconds not seconds.  That's a big difference. I can believe that MySQL is faster, because they probably don't need to do the bitmap heap scan. There is a much-anticipated feature

Re: [PERFORM] Slow Query- Simple taking

2010-10-28 Thread Mladen Gogala
On 10/28/2010 10:42 AM, Robert Haas wrote: I can believe that MySQL is faster, because they probably don't need to do the bitmap heap scan. There is a much-anticipated feature called index-only scans that we don't have yet in PG, which would help cases like this a great deal. Yyesss! Any time

Re: [PERFORM] Slow Query- Simple taking

2010-10-28 Thread Richard Broersma
On Thu, Oct 28, 2010 at 7:51 AM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Yyesss! Any time frame on that? Can you make it into 9.0.2? Maybe 9.1.0 or 9.2.0 :) 9.0's features are already frozen. -- Regards, Richard Broersma Jr. Visit the Los Angeles PostgreSQL Users Group (LAPUG)

Re: [PERFORM] Slow Query- Simple taking

2010-10-28 Thread Mladen Gogala
On 10/28/2010 10:53 AM, Richard Broersma wrote: On Thu, Oct 28, 2010 at 7:51 AM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Yyesss! Any time frame on that? Can you make it into 9.0.2? Maybe 9.1.0 or 9.2.0 :) 9.0's features are already frozen. Well, with all this global warming around

Re: [PERFORM] Postgres insert performance and storage requirement compared to Oracle

2010-10-28 Thread Jesper Krogh
On 2010-10-28 15:13, Merlin Moncure wrote: On Wed, Oct 27, 2010 at 3:47 PM, Jesper Kroghjes...@krogh.cc wrote: On 2010-10-27 20:51, Merlin Moncure wrote: Yes, I am quite aware of how the o/s page cache works. All else being equal, I more compact database obviously would be

Re: [PERFORM] How does PG know if data is in memory?

2010-10-28 Thread Cédric Villemain
2010/10/28 Francisco Reyes li...@stringsutils.com: Greg Smith writes: heard privately from two people who have done similar experiments on Linux and found closer to 8GB to be the point where performance started So on a machine with 72GB is 8GB still the recommended value? Yes, as a maximum,

Re: [PERFORM] Massive update, memory usage

2010-10-28 Thread Cédric Villemain
2010/10/28 Trenta sis trenta@gmail.com: There are about 100.000 inserts and 30 updates. Without transaction it seems that works, but with a transaction no. Witt about only 300.000 updates it seems that can finish correctly, but last 20% is slow because is using swap... Any tunning

Re: [PERFORM] Slow Query- Simple taking

2010-10-28 Thread Robert Haas
On Thu, Oct 28, 2010 at 11:23 AM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: On 10/28/2010 10:53 AM, Richard Broersma wrote: On Thu, Oct 28, 2010 at 7:51 AM, Mladen Gogala mladen.gog...@vmsinfo.com  wrote: Yyesss! Any time frame on that? Can you make it into 9.0.2? Maybe 9.1.0 or 9.2.0

Re: [PERFORM] Query much faster with enable_seqscan=0

2010-10-28 Thread Robert Haas
On Tue, Oct 12, 2010 at 10:28 PM, Samuel Gendler sgend...@ideasculptor.com wrote: On Tue, Sep 21, 2010 at 4:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: Ogden li...@darkstatic.com writes: SELECT tr.id, tr.sid             FROM             test_registration tr,             INNER JOIN

[PERFORM] partitioning question 1

2010-10-28 Thread Ben
hello -- my last email was apparently too long to respond to so i'll split it up into shorter pieces. my first question : my understanding of how range partitioning and constraint exclusion works leads me to believe that it does not buy any query performance that a clustered index doesn't

Re: [PERFORM] how to get the total number of records in report

2010-10-28 Thread Robert Haas
On Tue, Oct 19, 2010 at 7:56 PM, Merlin Moncure mmonc...@gmail.com wrote: On Mon, Oct 18, 2010 at 1:16 AM, AI Rumman rumman...@gmail.com wrote: At present for reporting I use following types of query: select crm.*, crm_cnt.cnt from crm, (select count(*) as cnt from crm) crm_cnt; Here count

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Joshua D. Drake
On Thu, 2010-10-28 at 09:36 -0700, Ben wrote: hello -- my last email was apparently too long to respond to so i'll split it up into shorter pieces. my first question : my understanding of how range partitioning and constraint exclusion works leads me to believe that it does not buy any

Re: [PERFORM] how to get the total number of records in report

2010-10-28 Thread Merlin Moncure
On Thu, Oct 28, 2010 at 1:05 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Oct 19, 2010 at 7:56 PM, Merlin Moncure mmonc...@gmail.com wrote: On Mon, Oct 18, 2010 at 1:16 AM, AI Rumman rumman...@gmail.com wrote: At present for reporting I use following types of query: select crm.*,

Re: [PERFORM] how to get the total number of records in report

2010-10-28 Thread AI Rumman
But I am using Postgresql 8.1 and it is not possible to write query as your one here. On Thu, Oct 28, 2010 at 11:05 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Oct 19, 2010 at 7:56 PM, Merlin Moncure mmonc...@gmail.com wrote: On Mon, Oct 18, 2010 at 1:16 AM, AI Rumman

Re: [PERFORM] how to get the total number of records in report

2010-10-28 Thread Merlin Moncure
On Thu, Oct 28, 2010 at 1:49 PM, AI Rumman rumman...@gmail.com wrote: But I am using Postgresql 8.1 and it is not possible to write query as your one here. with 8.1, you are limited to subquery approach, application derived count, plpgsql hacks, etc. merlin -- Sent via pgsql-performance

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Ben
thanks for the prompt response. some comments / questions below : On Oct 28, 2010, at 10:31 AM, Joshua D. Drake wrote: ...constraint exclusion is able to eliminate table partitions. the I/O advantages of having queries target small subtables are the same as the I/O advantages of clustering

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Joshua D. Drake
On Thu, 2010-10-28 at 11:44 -0700, Ben wrote: Yes the constraints have to be static. Not sure about the operator question honestly. this seems to severely restrict their usefulness -- our queries are data warehouse analytical -type queries, so the constraints are usually data-driven

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Ben
On Oct 28, 2010, at 11:50 AM, Joshua D. Drake wrote: Yes the constraints have to be static. Not sure about the operator question honestly. this seems to severely restrict their usefulness -- our queries are data warehouse analytical -type queries, so the constraints are usually

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Joshua D. Drake
On Thu, 2010-10-28 at 12:25 -0700, Ben wrote: i think we are talking about two different things here: the constraints on the table, and the where-clause constraints in a query which may or may not trigger constraint exclusion. i understand that table constraints have to be constants -- it

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Ben
On Oct 28, 2010, at 12:44 PM, Joshua D. Drake wrote: My tests show you are incorrect: part_test=# explain analyze select * from foo join bar using (i) where i=9; QUERY PLAN

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Reid Thompson
On Thu, 2010-10-28 at 12:59 -0700, Ben wrote: explain analyze select * from foo join bar using (i); vs explain analyze select * from foo join bar using (i) where i=9;

Re: [PERFORM] BBU Cache vs. spindles

2010-10-28 Thread James Mansion
Tom Lane wrote: The other and probably worse problem is that there's no application control over how soon changes to mmap'd pages get to disk. An msync will flush them out, but the kernel is free to write dirty pages sooner. So if they're depending for consistency on writes not happening until

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Joshua D. Drake
On Thu, 2010-10-28 at 12:59 -0700, Ben wrote: On Oct 28, 2010, at 12:44 PM, Joshua D. Drake wrote: My tests show you are incorrect: part_test=# explain analyze select * from foo join bar using (i) where i=9; QUERY PLAN

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Ben
whoops, didn't see the i=9 (linebreak! linebreak!) nonetheless that is a static constant constraint on the column i, and i was asking if constraint exclusions would work for dynamic constraints (like those derived from a table joined against.) so for example the bar table has only 0-9 in its

Re: [PERFORM] BBU Cache vs. spindles

2010-10-28 Thread Tom Lane
James Mansion ja...@mansionfamily.plus.com writes: Tom Lane wrote: The other and probably worse problem is that there's no application control over how soon changes to mmap'd pages get to disk. An msync will flush them out, but the kernel is free to write dirty pages sooner. So if they're

Re: [PERFORM] Massive update, memory usage

2010-10-28 Thread Trenta sis
Well, I have solved executing with more RAM, and then works correctly Thanks 2010/10/28 Cédric Villemain cedric.villemain.deb...@gmail.com 2010/10/28 Trenta sis trenta@gmail.com: There are about 100.000 inserts and 30 updates. Without transaction it seems that works, but

Re: [PERFORM] Massive update, memory usage

2010-10-28 Thread Emanuele Bracci Poste
Scusa, scadenze a parte, ma non vi è sembrato il caso di chiedere a chi sta gestendo il progetto prima di rimuovere una risorsa? Grazie comunque. Emanuele Il giorno 28/ott/2010, alle ore 23.48, Trenta sis ha scritto: Well, I have solved executing with more RAM, and then works correctly