Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Gregory Stark
Wait, rereading the original queries I seem to have misunderstood something. The individual parts of the partitioned tables are being accessed in timestamp order. So what's missing is some way for the optimizer to know that the resulting append results will still be in order. If it knew that all t

Re: [PERFORM] Feature Request --- was: PostgreSQL Performance Tuning

2007-04-30 Thread Kevin Hunter
At 12:18p -0400 on 30 Apr 2007, Craig A. James wrote: 1. Generating a resonable starting configuration for neophyte users who have installed Postgres for the first time. I recognize that PostgreSQL and MySQL try to address different problem-areas, but is this one reason why a lot of people w

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Steinar H. Gunderson
On Mon, Apr 30, 2007 at 03:29:30PM +0200, Andreas Haumer wrote: > This already gives a row matching the given WHERE clause. > It makes no sense to scan the other tables, as the query > asks for one row only and all the other tables have timestamps > larger than all the timestamps in table t_mv_2006

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Andreas Haumer
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi! Tom Lane schrieb: [...] > As already pointed out, this is only going to be able to exclude > partitions that are strictly after the limit-time, since you have no > WHERE clause that excludes anything before. Can you set a reasonable > upper bound

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Andreas Haumer
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi! Neil Peter Braggio schrieb: > Just cast the value in the WHERE clause: > > select ts from mwdb.t_mv where zr=3622 and ts > '2006-01-01 00:00:00' > ::TIMESTAMP order by ts asc limit 1; > > This search only into the right partitioned tables if you

Re: [PERFORM] Feature Request --- was: PostgreSQL Performance Tuning

2007-04-30 Thread Craig A. James
Greg Smith wrote: If you're going to the trouble of building a tool for offering configuration advice, it can be widly more effective if you look inside the database after it's got data in it, and preferably after it's been running under load for a while, and make your recommendations based on

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Gregory Stark
"Guillaume Cottenceau" <[EMAIL PROTECTED]> writes: > I think this is the last claimed point which is incorrect. Pg has > no general guarantee the partitions actually create a disjoint > set, even with the CHECK constraints. Pg can only optimize by > avoiding scanning the partitions inside which no

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Tom Lane
Andreas Haumer <[EMAIL PROTECTED]> writes: > A simple example: Get the timestamp of a measurement value for time > series 3622 which is right before the measurement value with time > stamp '2007-04-22 00:00:00': > testdb_std=> select ts from mwdb.t_mv where zr=3622 and ts < '2007-04-22 > 00:00:00

[PERFORM] sytem log audit/reporting and psql

2007-04-30 Thread Fei Liu
Hello group, I need to design and develop a web reporting system to let users query/view syslog files on a unix host. For now, I am concentrating on the authentication file that has user logon (success/failure) and logoff records. The log file is logrotated every week or so. My reporting system

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Neil Peter Braggio
Just cast the value in the WHERE clause: select ts from mwdb.t_mv where zr=3622 and ts > '2006-01-01 00:00:00' ::TIMESTAMP order by ts asc limit 1; This search only into the right partitioned tables if you build the rules based in the ts field. Neil Peter Braggio [EMAIL PROTECTED] On 4/

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Guillaume Cottenceau
Andreas Haumer writes: > > Well, how can the planner know inside which partition the wanted > > row is? There might be no data, say, inside a couple of > > partitions in the past before finding the wanted row, in which > > case 3 partitions in the past must be scanned. > > > > I think the plann

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Richard Huxton
Andreas Haumer wrote: I think the planner could do the following: a) It could make a better decision in which direction to scan the partitions (depending on sort order involved in the query) b) It could stop scanning as soon as there can not be any further resulting row according to the

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Andreas Haumer
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi! Guillaume Cottenceau schrieb: > Andreas Haumer writes: [...] > >> Now my question is: Does the query planner in the case of partitioned tables >> really have to scan all indexes in order to get the next timestamp smaller >> (or larger) than a gi

Re: [PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Guillaume Cottenceau
Andreas Haumer writes: [...] > testdb_part=> explain analyze select ts from mwdb.t_mv where zr=3622 and ts < > '2007-04-22 00:00:00' order by ts desc limit 1; > > QUERY PLAN >

[PERFORM] Query performance problems with partitioned tables

2007-04-30 Thread Andreas Haumer
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi! I'm currently experimenting with PostgreSQL 8.2.4 and table partitioning in order to improve the performance of an application I'm working on. My application is about managing measurement values (lots of!) I have one table "t_mv" which stores all

Re: [PERFORM] Feature Request --- was: PostgreSQL Performance Tuning

2007-04-30 Thread Greg Smith
On Fri, 27 Apr 2007, Josh Berkus wrote: *Everyone* wants this. The problem is that it's very hard code to write given the number of variables There's lots of variables, and there are at least three major ways to work on improving someone's system: 1) Collect up data about their system (mem