Re: [PERFORM] Fragmentation/Vacuum, Analyze, Re-Index

2010-01-22 Thread Reid Thompson
On 1/22/2010 2:27 PM, Richard Neill wrote: DM wrote: Is there any script/tool to identify if the table requires full vacuum? or to re-index an existing index table? http://pgsql.tapoueh.org/site/html/news/20080131.bloat.html The bucardo project has released its nagios plugins for PostgreSQL a

Re: [PERFORM] Inserting 8MB bytea: just 25% of disk perf used?

2010-01-22 Thread Scott Carey
On Jan 22, 2010, at 12:42 PM, fka...@googlemail.com wrote: > > 'Writing twice': That is the most interesting point I > believe. Why is the data disk doing 40 MB/s *not* including > WAL, however, having 20 MB/s write thoughput in fact. Seems > like: 20 MB for data, 20 MB for X, 20 MB for WAL. >

Re: [PERFORM] Inserting 8MB bytea: just 25% of disk perf used?

2010-01-22 Thread Scott Carey
On Jan 21, 2010, at 12:35 AM, Greg Smith wrote: > Scott Carey wrote: >> On Jan 20, 2010, at 5:32 AM, fka...@googlemail.com wrote: >> >> >>> In the attachement you'll find 2 screenshots perfmon34.png >>> and perfmon35.png (I hope 2x14 kb is o.k. for the mailing >>> list). >>> >>> > > I don't

Re: [PERFORM] Inserting 8MB bytea: just 25% of disk perf used?

2010-01-22 Thread Scott Carey
On Jan 21, 2010, at 12:35 AM, Greg Smith wrote: > Scott Carey wrote: >> On Jan 20, 2010, at 5:32 AM, fka...@googlemail.com wrote: >> >> >>> In the attachement you'll find 2 screenshots perfmon34.png >>> and perfmon35.png (I hope 2x14 kb is o.k. for the mailing >>> list). >>> >>> > > I don't

Re: [PERFORM] Data Set Growth causing 26+hour runtime, on what we believe to be very simple SQL

2010-01-22 Thread Craig Ringer
On 23/01/2010 1:59 AM, Tory M Blue wrote: It's this query, run 6000 times with a diff makeid's / / /SELECT COUNT(DISTINCT uid ) AS active_users FROM pixelpool.userstatsWHERE makeid ='bmw-ferman' AND tagged =true/ /Plan/ / "Aggregate (cost=49467.00..49467.01 rows=1 width=

Re: [PERFORM] Inserting 8MB bytea: just 25% of disk perf used?

2010-01-22 Thread fka...@googlemail.com
Scott Carey: > Well, something is causing the system to alternate between > CPU and disk bound here. (see below). > It would be useful to see what affect the index has. Ok, I simply deleted the index and repeated the test: I did not notice any difference. This is probably so because in fact I am

Re: [PERFORM] Fragmentation/Vacuum, Analyze, Re-Index

2010-01-22 Thread Richard Neill
DM wrote: Is there any script/tool to identify if the table requires full vacuum? or to re-index an existing index table? Don't know if there is a script to specifically do this, though you may find this query a useful one: SELECT relname, reltuples, relpages FROM pg_class ORDER BY relpage

Re: [PERFORM] Data Set Growth causing 26+hour runtime, on what we believe to be very simple SQL

2010-01-22 Thread Tory M Blue
On Fri, Jan 22, 2010 at 11:06 AM, Tory M Blue wrote: > On Fri, Jan 22, 2010 at 10:26 AM, Matthew Wakeling > wrote: >> >> On Fri, 22 Jan 2010, Tory M Blue wrote: >>> >>> But the same sql that returns maybe 500 rows is pretty fast, it's the return >>> of 10K+ rows that seems to stall and is CPU Bo

Re: [PERFORM] Data Set Growth causing 26+hour runtime, on what we believe to be very simple SQL

2010-01-22 Thread Tory M Blue
On Fri, Jan 22, 2010 at 10:26 AM, Matthew Wakeling wrote: > > On Fri, 22 Jan 2010, Tory M Blue wrote: >> >> But the same sql that returns maybe 500 rows is pretty fast, it's the return >> of 10K+ rows that seems to stall and is CPU Bound. > > Okay, so you have two differing cases. Show us the EXPL

Re: [PERFORM] Data Set Growth causing 26+hour runtime, on what we believe to be very simple SQL

2010-01-22 Thread Scott Marlowe
On Fri, Jan 22, 2010 at 10:59 AM, Tory M Blue wrote: > On Thu, Jan 21, 2010 at 7:46 PM, Craig Ringer > wrote: >> > Any assistance would be appreciated, don't worry about slapping me >> > around I need to figure this out. Otherwise I'm buying new hardware >> > where it may not be required. >> >> W

Re: [PERFORM] Data Set Growth causing 26+hour runtime, on what we believe to be very simple SQL

2010-01-22 Thread Matthew Wakeling
On Fri, 22 Jan 2010, Tory M Blue wrote: But the same sql that returns maybe 500 rows is pretty fast, it's the return of 10K+ rows that seems to stall and is CPU Bound. Okay, so you have two differing cases. Show us the EXPLAIN ANALYSE for both of them, and we will see what the difference is.

Re: [PERFORM] Data Set Growth causing 26+hour runtime, on what we believe to be very simple SQL

2010-01-22 Thread Richard Huxton
On 22/01/10 18:03, Tory M Blue wrote: On Fri, Jan 22, 2010 at 1:42 AM, Richard Huxton wrote: On 21/01/10 22:15, Tory M Blue wrote: 2. If it's mostly tagged=true you are interested in you can always use a partial index: CREATE INDEX ... (makeid) WHERE tagged This might be a win even if you n

Re: [PERFORM] Data Set Growth causing 26+hour runtime, on what we believe to be very simple SQL

2010-01-22 Thread Tory M Blue
On Fri, Jan 22, 2010 at 1:42 AM, Richard Huxton wrote: > On 21/01/10 22:15, Tory M Blue wrote: > >> · Data distribution = In the 98mill records, there are 7000 unique >> >> makeid's, and 21mill unique UID's. About 41mill of the records have >> tagged=true >> >> · Time to execute t

Re: [PERFORM] Data Set Growth causing 26+hour runtime, on what we believe to be very simple SQL

2010-01-22 Thread Tory M Blue
On Thu, Jan 21, 2010 at 7:46 PM, Craig Ringer wrote: > > > Any assistance would be appreciated, don't worry about slapping me > > around I need to figure this out. Otherwise I'm buying new hardware > > where it may not be required. > > What is the reporting query that takes 26 hours? You didn't se

Re: [PERFORM] New server to improve performance on our large and busy DB - advice?

2010-01-22 Thread Carlo Stonebanks
Hi Greg, As a follow up to this suggestion: I don't see effective_cache_size listed there. If that's at the default, I wouldn't be surprised that you're seeing sequential scans instead of indexed ones far too often. I found an article written by you http://www.westnet.com/~gsmith/content/p

Re: [PERFORM] Fragmentation/Vacuum, Analyze, Re-Index

2010-01-22 Thread DM
Is there any script/tool to identify if the table requires full vacuum? or to re-index an existing index table? Thanks Deepak On Fri, Jan 22, 2010 at 12:11 AM, DM wrote: > Hello All, > > How to identify if a table requires full vacuum? How to identify when to do > re-index on an existing index

Re: [PERFORM] Slow update query

2010-01-22 Thread Robert Haas
On Thu, Jan 21, 2010 at 11:14 AM, elias ghanem wrote: > So the average execution time of the function is around 2.5 mins, meaning > that the update query (+ the vacuum) takes 2.5 mins to execute. So is this a > normal behavior? (The same function in oracle with the same environment > (with our vac

[PERFORM] Slow update query

2010-01-22 Thread elias ghanem
Hi, For the explain analyze here's the output: "Seq Scan on in_sortie (cost=0.00..171140.19 rows=114449 width=84) (actual time=15.074..28461.349 rows=99611 loops=1)" " Output: type, site_id, fiche_produit_id, numero_commande, ligne_commande, date_sortie, quantite_sortie, date_livraison_souh

Re: [PERFORM] Data Set Growth causing 26+hour runtime, on what we believe to be very simple SQL

2010-01-22 Thread Richard Huxton
On 21/01/10 22:15, Tory M Blue wrote: · Data distribution = In the 98mill records, there are 7000 unique makeid's, and 21mill unique UID's. About 41mill of the records have tagged=true · Time to execute the following query with indices on makeid and tagged = 90-120 seconds. The p

[PERFORM] Fragmentation/Vacuum, Analyze, Re-Index

2010-01-22 Thread DM
Hello All, How to identify if a table requires full vacuum? How to identify when to do re-index on an existing index of a table? Is there any tool for the above? Thanks Deepak Murthy