Re: [PERFORM] Monitoring Transaction Log size

2007-01-18 Thread Simon Riggs
On Wed, 2007-01-17 at 23:03 +0500, Shoaib Mir wrote: > archive_timeout (came in ver 8.2) might help you with customizing the > size for log files. I'm not sure that it will. If anything it could produce more log files, which could lead to a backlog if the archive_command isn't functioning for som

Re: [PERFORM] Monitoring Transaction Log size

2007-01-18 Thread Shoaib Mir
Suggested in case he wants to do a log switch after certain amount of time... --- Shoaib Mir EnterpriseDB (www.enterprisedb.com) On 1/18/07, Simon Riggs <[EMAIL PROTECTED]> wrote: On Wed, 2007-01-17 at 23:03 +0500, Shoaib Mir wrote: > archive_timeout (came in ver 8.2) might help you wi

Re: [PERFORM] Configuration Advice

2007-01-18 Thread Arjen van der Meijden
On 18-1-2007 0:37 Adam Rich wrote: 4) Complex queries that might take advantage of the MySQL "Query Cache" since the base data never changes Have you ever compared MySQL's performance with complex queries to PostgreSQL's? I once had a query which would operate on a recordlist and see whether

[PERFORM] Vacuum v/s Autovacuum

2007-01-18 Thread Gauri Kanekar
Hi List, Can anybody help me out with this - is autovacuum similar to vacuum full analyse verbose. -- Regards Gauri

Re: [PERFORM] Vacuum v/s Autovacuum

2007-01-18 Thread Michael Glaesemann
On Jan 18, 2007, at 22:24 , Gauri Kanekar wrote: is autovacuum similar to vacuum full analyse verbose. http://www.postgresql.org/docs/8.2/interactive/routine- vacuuming.html#AUTOVACUUM Apparently, no FULL, no VERBOSE (which is only really useful if you want to see the results, not for ro

Re: [PERFORM] Vacuum v/s Autovacuum

2007-01-18 Thread Gauri Kanekar
Hi Thanks. We have autovacuum ON , but still postgres server warns to increas max_fsm_pages value. Do autovacuum release space after it is over? so how can we tackle it. On 1/18/07, Michael Glaesemann <[EMAIL PROTECTED]> wrote: On Jan 18, 2007, at 22:24 , Gauri Kanekar wrote: > is auto

Re: [PERFORM] Vacuum v/s Autovacuum

2007-01-18 Thread Bill Moran
In response to "Gauri Kanekar" <[EMAIL PROTECTED]>: > On 1/18/07, Michael Glaesemann <[EMAIL PROTECTED]> wrote: > > > > On Jan 18, 2007, at 22:24 , Gauri Kanekar wrote: > > > > > is autovacuum similar to vacuum full analyse verbose. > > > > http://www.postgresql.org/docs/8.2/interactive/routine- >

Re: [PERFORM] Vacuum v/s Autovacuum

2007-01-18 Thread Shoaib Mir
You will need to properly tune the thresholds for VACUUM and ANALYZE in case of autovacuuming process, so that you do not need to increase the max_fsm_pages oftenly... - Shoaib Mir EnterpriseDB (www.enterprisedb.com) On 1/18/07, Bill Moran <[EMAIL PROTECTED]> wrote: In response to

Re: [PERFORM] Configuration Advice

2007-01-18 Thread Scott Marlowe
On Wed, 2007-01-17 at 18:27, Steve wrote: > > Generally speaking, once you've gotten to the point of swapping, even a > > little, you've gone too far. A better approach is to pick some > > conservative number, like 10-25% of your ram for shared_buffers, and 1 > > gig or so for maintenance work_mem

Re: [PERFORM] Configuration Advice

2007-01-18 Thread Scott Marlowe
On Thu, 2007-01-18 at 04:24, Arjen van der Meijden wrote: > On 18-1-2007 0:37 Adam Rich wrote: > > 4) Complex queries that might take advantage of the MySQL "Query Cache" > > since the base data never changes > > Have you ever compared MySQL's performance with complex queries to > PostgreSQL's? I

Re: [PERFORM] Configuration Advice

2007-01-18 Thread Arjen van der Meijden
On 18-1-2007 17:20 Scott Marlowe wrote: Besides that, mysql rewrites the entire table for most table-altering statements you do (including indexes). Note that this applies to the myisam table type. innodb works quite differently. It is more like pgsql in behaviour, and is an mvcc storage A

Re: [PERFORM] Configuration Advice

2007-01-18 Thread Jeremy Haile
> I once had a query which would operate on a recordlist and > see whether there were any gaps larger than 1 between consecutive > primary keys. Would you mind sharing the query you described? I am attempting to do something similar now. ---(end of broadcast)--

[PERFORM] Autoanalyze settings with zero scale factor

2007-01-18 Thread Jeremy Haile
Some of my very large tables (10 million rows) need to be analyzed by autovacuum on a frequent basis. Rather than specifying this as a percentage of table size + base threshold, I wanted to specify it as an explicit number of rows. I changed the table-specific settings so that the ANALYZE base th

Re: [PERFORM] Autoanalyze settings with zero scale factor

2007-01-18 Thread Matthew T. O'Connor
Jeremy Haile wrote: I changed the table-specific settings so that the ANALYZE base threshold was 5000 and the ANALYZE scale factor is 0. According to the documented formula: analyze threshold = analyze base threshold + analyze scale factor * number of tuples, I assumed that this would cause the

Re: [PERFORM] Autoanalyze settings with zero scale factor

2007-01-18 Thread Jeremy Haile
> Unless it's just a bug, my only guess is that autovacuum may be getting > busy at times (vacuuming large tables for example) and hasn't had a > chance to even look at that table for a while, and by the time it gets > to it, there have been tens of thousands of inserts. Does that sounds > pla

[PERFORM] Postgres and really huge tables

2007-01-18 Thread Brian Hurt
Is there any experience with Postgresql and really huge tables? I'm talking about terabytes (plural) here in a single table. Obviously the table will be partitioned, and probably spread among several different file systems. Any other tricks I should know about? We have a problem of that for

Re: [PERFORM] [pgsql-advocacy] Postgres and really huge tables

2007-01-18 Thread Joshua D. Drake
Brian Hurt wrote: > Is there any experience with Postgresql and really huge tables? I'm > talking about terabytes (plural) here in a single table. Obviously the > table will be partitioned, and probably spread among several different > file systems. Any other tricks I should know about? > > We

Re: [PERFORM] Autoanalyze settings with zero scale factor

2007-01-18 Thread Matthew T. O'Connor
Jeremy Haile wrote: Also, are other auto-vacuums and auto-analyzes showing up in the pg_stats table? Maybe it's a stats system issue. No tables have been vacuumed or analyzed today. I had thought that this problem was due to my pg_autovacuum changes, but perhaps not. I restarted Postgre

Re: [PERFORM] Postgres and really huge tables

2007-01-18 Thread Scott Marlowe
On Thu, 2007-01-18 at 14:31, Brian Hurt wrote: > Is there any experience with Postgresql and really huge tables? I'm > talking about terabytes (plural) here in a single table. Obviously the > table will be partitioned, and probably spread among several different > file systems. Any other tric

Re: [PERFORM] Configuration Advice

2007-01-18 Thread Arjen van der Meijden
On 18-1-2007 18:28 Jeremy Haile wrote: I once had a query which would operate on a recordlist and see whether there were any gaps larger than 1 between consecutive primary keys. Would you mind sharing the query you described? I am attempting to do something similar now. Well it was over a

Re: [PERFORM] Autoanalyze settings with zero scale factor

2007-01-18 Thread Tom Lane
"Jeremy Haile" <[EMAIL PROTECTED]> writes: > No tables have been vacuumed or analyzed today. I had thought that this > problem was due to my pg_autovacuum changes, but perhaps not. I > restarted PostgreSQL (in production - yikes) About a minute after being > restarted, the autovac process fired

Re: [PERFORM] [pgsql-advocacy] Postgres and really huge tables

2007-01-18 Thread Chris Mair
Is there any experience with Postgresql and really huge tables? I'm talking about terabytes (plural) here in a single table. Obviously the table will be partitioned, and probably spread among several different file systems. Any other tricks I should know about? We have a problem of that f

Re: [PERFORM] Postgres and really huge tables

2007-01-18 Thread Tom Lane
Brian Hurt <[EMAIL PROTECTED]> writes: > Is there any experience with Postgresql and really huge tables? I'm > talking about terabytes (plural) here in a single table. The 2MASS sky survey point-source catalog http://www.ipac.caltech.edu/2mass/releases/allsky/doc/sec2_2a.html is 470 million rows

Re: [PERFORM] Autoanalyze settings with zero scale factor

2007-01-18 Thread Jeremy Haile
Well - it hadn't run on any table in over 24 hours (according to pg_stat_user_tables). My tables are constantly being inserted into and deleted from, and the autovacuum settings are pretty aggressive. I also had not seen the autovac process running in the past 24 hours. (although I wasn't watchin

Re: [PERFORM] Configuration Advice

2007-01-18 Thread Tom Lane
Arjen van der Meijden <[EMAIL PROTECTED]> writes: > PS, In case any of the planner-hackers are reading, here are the plans > of the first two queries, just to see if something can be done to > decrease the differences between them. Increase work_mem? It's not taking the hash because it thinks i

Re: [PERFORM] [pgsql-advocacy] Postgres and really huge tables

2007-01-18 Thread Luke Lonergan
Chris, On 1/18/07 1:42 PM, "Chris Mair" <[EMAIL PROTECTED]> wrote: > A lot of data, but not a lot of records... I don't know if that's > valid. I guess the people at Greenplum and/or Sun have more exciting > stories ;) You guess correctly :-) Given that we're Postgres 8.2, etc compatible, that

Re: [PERFORM] Postgres and really huge tables

2007-01-18 Thread Gavin Sherry
Hi Brian, On Thu, 18 Jan 2007, Brian Hurt wrote: > Is there any experience with Postgresql and really huge tables? I'm > talking about terabytes (plural) here in a single table. Obviously the > table will be partitioned, and probably spread among several different > file systems. Any other tri

Re: [PERFORM] Configuration Advice

2007-01-18 Thread Arjen van der Meijden
On 18-1-2007 23:11 Tom Lane wrote: Increase work_mem? It's not taking the hash because it thinks it won't fit in memory ... When I increase it to 128MB in the session (arbitrarily selected relatively large value) it indeed has the other plan. Best regards, Arjen --