Re: [HACKERS] On partitioning

2014-09-01 Thread Hannu Krosing
On 09/01/2014 05:52 PM, Andres Freund wrote: On 2014-08-29 20:12:16 +0200, Hannu Krosing wrote: It would need to replace plain tid (pagenr, tupnr) with triple of (partid, pagenr, tupnr). Cross-partition indexes are especially needed if we want to allow putting UNIQUE constraints on

Re: [HACKERS] On partitioning

2014-09-01 Thread Craig Ringer
On 09/01/2014 11:52 PM, Andres Freund wrote: I personally think that implementing cross partition indexes has a low enough cost/benefit ratio that I doubt it's wise to tackle it anytime soon. UNIQUE constraints on partitioned tables (and thus foreign key constraints pointing to partitioned

Re: [HACKERS] On partitioning

2014-09-01 Thread Craig Ringer
On 09/01/2014 04:03 AM, Tom Lane wrote: I think one of the key design decisions we have to make is whether partitions are all constrained to have exactly the same set of indexes. ... and a lot of that comes down to what use cases the partitioning is meant to handle, and what people are

Re: [HACKERS] On partitioning

2014-08-31 Thread Tom Lane
Another thought about this general topic: Alvaro Herrera alvhe...@2ndquadrant.com writes: ... Allowed actions on a RELKIND_PARTITION: * CREATE INDEX .. ON PARTITION n ON TABLE xyz ... Still To Be Designed * Are indexes/constraints inherited from the parent rel? I

Re: [HACKERS] On partitioning

2014-08-31 Thread Hannu Krosing
On 08/31/2014 10:03 PM, Tom Lane wrote: Another thought about this general topic: Alvaro Herrera alvhe...@2ndquadrant.com writes: ... Allowed actions on a RELKIND_PARTITION: * CREATE INDEX .. ON PARTITION n ON TABLE xyz ... Still To Be Designed * Are

Re: [HACKERS] On partitioning

2014-08-31 Thread Martijn van Oosterhout
On Fri, Aug 29, 2014 at 12:35:50PM -0400, Tom Lane wrote: Each partition is assigned an Expression that receives a tuple and returns boolean. This expression returns true if a given tuple belongs into it, false otherwise. -1, in fact minus a lot. One of the core problems of the current

Re: [HACKERS] On partitioning

2014-08-29 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: [ partition sketch ] In this design, partitions are first-class objects, not normal tables in inheritance hierarchies. There are no pg_inherits entries involved at all. Hm, actually I'd say they are *not* first class objects; the problem with

Re: [HACKERS] On partitioning

2014-08-29 Thread Greg Stark
On Fri, Aug 29, 2014 at 4:56 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: For scan plans, we need to prepare Append lists which are used to scan for tuples in a partitioned relation. We can setup fake constraint expressions based on the partitioning expressions, which let the planner

Re: [HACKERS] On partitioning

2014-08-29 Thread Pavel Stehule
2014-08-29 18:35 GMT+02:00 Tom Lane t...@sss.pgh.pa.us: Alvaro Herrera alvhe...@2ndquadrant.com writes: [ partition sketch ] In this design, partitions are first-class objects, not normal tables in inheritance hierarchies. There are no pg_inherits entries involved at all. Hm, actually

Re: [HACKERS] On partitioning

2014-08-29 Thread Tom Lane
Greg Stark st...@mit.edu writes: And I think there are still unanswered questions about indexes. One other interesting thought that occurs to me: are we going to support UPDATEs that cause a row to belong to a different partition? If so, how are we going to handle the update chain links?

Re: [HACKERS] On partitioning

2014-08-29 Thread Alvaro Herrera
Tom Lane wrote: Greg Stark st...@mit.edu writes: And I think there are still unanswered questions about indexes. One other interesting thought that occurs to me: are we going to support UPDATEs that cause a row to belong to a different partition? If so, how are we going to handle the

Re: [HACKERS] On partitioning

2014-08-29 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: Tom Lane wrote: One other interesting thought that occurs to me: are we going to support UPDATEs that cause a row to belong to a different partition? If so, how are we going to handle the update chain links? Bah, I didn't mention it? My

Re: [HACKERS] On partitioning

2014-08-29 Thread Andres Freund
On 2014-08-29 13:15:16 -0400, Tom Lane wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Tom Lane wrote: One other interesting thought that occurs to me: are we going to support UPDATEs that cause a row to belong to a different partition? If so, how are we going to handle the update

Re: [HACKERS] On partitioning

2014-08-29 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-08-29 13:15:16 -0400, Tom Lane wrote: Hm. I certainly agree that it's a case that could be disallowed for a first cut, but it'd be nice to have some clue about how we might allow it eventually. Not pretty, but we could set t_ctid to some

Re: [HACKERS] On partitioning

2014-08-29 Thread Alvaro Herrera
Tom Lane wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Tom Lane wrote: One other interesting thought that occurs to me: are we going to support UPDATEs that cause a row to belong to a different partition? If so, how are we going to handle the update chain links? Bah, I

Re: [HACKERS] On partitioning

2014-08-29 Thread Andres Freund
On 2014-08-29 13:29:19 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-08-29 13:15:16 -0400, Tom Lane wrote: Hm. I certainly agree that it's a case that could be disallowed for a first cut, but it'd be nice to have some clue about how we might allow it

Re: [HACKERS] On partitioning

2014-08-29 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-08-29 13:29:19 -0400, Tom Lane wrote: An actual fix would presumably involve adding a partition number to the ctid chain field in tuples in partitioned tables. The reason I bring it up now is that we'd have to commit to doing that (or at

Re: [HACKERS] On partitioning

2014-08-29 Thread Hannu Krosing
On 08/29/2014 07:15 PM, Tom Lane wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Tom Lane wrote: One other interesting thought that occurs to me: are we going to support UPDATEs that cause a row to belong to a different partition? If so, how are we going to handle the update chain

Re: [HACKERS] On partitioning

2014-08-29 Thread Hannu Krosing
On 08/29/2014 05:56 PM, Alvaro Herrera wrote: Prompted by a comment in the UPDATE/LIMIT thread, I saw Marko Tiikkaja reference Tom's post http://www.postgresql.org/message-id/1598.1399826...@sss.pgh.pa.us which mentions the possibility of a different partitioning implementation than what we

Re: [HACKERS] On partitioning

2014-08-29 Thread Alvaro Herrera
Hannu Krosing wrote: Cross-partition indexes are especially needed if we want to allow putting UNIQUE constraints on non-partition-key columns. I'm not going to implement cross-partition indexes in the first patch. They are a huge can of worms. -- Álvaro Herrera

Re: [HACKERS] On partitioning

2014-08-29 Thread Robert Haas
On Fri, Aug 29, 2014 at 11:56 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: In this design, partitions are first-class objects, not normal tables in inheritance hierarchies. There are no pg_inherits entries involved at all. Whoa. I always assumed that table inheritance was a

Re: [HACKERS] On partitioning

2014-08-29 Thread Amit Langote
On Sat, Aug 30, 2014 at 12:56 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Prompted by a comment in the UPDATE/LIMIT thread, I saw Marko Tiikkaja reference Tom's post http://www.postgresql.org/message-id/1598.1399826...@sss.pgh.pa.us which mentions the possibility of a different

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Yuri Levinsky
: [HACKERS] Hash partitioning. On 06/26/2013 05:46 PM, Heikki Linnakangas wrote: We could also allow a large query to search a single table in parallel. A seqscan would be easy to divide into N equally-sized parts that can be scanned in parallel. It's more difficult for index scans, but even

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Nicolas Barbier
2013/6/26 Heikki Linnakangas hlinnakan...@vmware.com: On 26.06.2013 16:41, Yuri Levinsky wrote: Heikki, As far as I understand the height of the btree will affect the number of I/Os necessary. The height of the tree does not increase linearly with the number of records. Now let's compare

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Nicolas Barbier
2013/6/27 Nicolas Barbier nicolas.barb...@gmail.com: When each index requires one extra I/O (because each index is one level taller), that is 50 extra I/Os. In the partitioned case, each index would require the normal smaller amount of I/Os. [..] Using those other indexes (both for look-ups

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Ants Aasma
On Jun 27, 2013 12:24 PM, Nicolas Barbier nicolas.barb...@gmail.com wrote: 2013/6/27 Nicolas Barbier nicolas.barb...@gmail.com: When each index requires one extra I/O (because each index is one level taller), that is 50 extra I/Os. In the partitioned case, each index would require the

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Markus Wanner
On 06/27/2013 11:12 AM, Nicolas Barbier wrote: Imagine that there are a lot of indexes, e.g., 50. Although a lookup (walking one index) is equally fast, an insertion must update al 50 indexes. When each index requires one extra I/O (because each index is one level taller), that is 50 extra

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Nicolas Barbier
2013/6/27 Markus Wanner mar...@bluegap.ch: On 06/27/2013 11:12 AM, Nicolas Barbier wrote: Imagine that there are a lot of indexes, e.g., 50. Although a lookup (walking one index) is equally fast, an insertion must update al 50 indexes. When each index requires one extra I/O (because each

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Markus Wanner
On 06/27/2013 06:35 PM, Nicolas Barbier wrote: I am assuming that this (comparatively very small and super-hot) index is cached all the time, while for the other indexes (that are supposedly super-huge) only the top part stays cached. I am mostly just trying to find out where Yuri’s

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Jeff Janes
On Wed, Jun 26, 2013 at 8:55 AM, Markus Wanner mar...@bluegap.ch wrote: On 06/26/2013 05:46 PM, Heikki Linnakangas wrote: We could also allow a large query to search a single table in parallel. A seqscan would be easy to divide into N equally-sized parts that can be scanned in parallel.

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Jeff Janes
On Wed, Jun 26, 2013 at 11:14 AM, Claudio Freire klaussfre...@gmail.comwrote: Now I just have two indices. One that indexes only hot tuples, it's very heavily queried and works blazingly fast, and one that indexes by (hotness, key). I include the hotness value on the query, and still works

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Jeff Janes
On Thu, Jun 27, 2013 at 2:12 AM, Nicolas Barbier nicolas.barb...@gmail.comwrote: 2013/6/26 Heikki Linnakangas hlinnakan...@vmware.com: On 26.06.2013 16:41, Yuri Levinsky wrote: Heikki, As far as I understand the height of the btree will affect the number of I/Os necessary. The height

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Jeff Janes
On Thu, Jun 27, 2013 at 9:35 AM, Nicolas Barbier nicolas.barb...@gmail.comwrote: My reasoning was: To determine which index block to update (typically one in both the partitioned and non-partitioned cases), one needs to walk the index first, which supposedly causes one additional (read) I/O

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Markus Wanner
On 06/27/2013 11:13 PM, Jeff Janes wrote: Wouldn't any IO system being used on a high-end system be fairly good about making this work through interleaved read-ahead algorithms? To some extent, certainly. It cannot possibly get better than a fully sequential load, though. That sounds like it

Re: [HACKERS] Hash partitioning.

2013-06-27 Thread Claudio Freire
On Thu, Jun 27, 2013 at 6:20 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Wed, Jun 26, 2013 at 11:14 AM, Claudio Freire klaussfre...@gmail.com wrote: Now I just have two indices. One that indexes only hot tuples, it's very heavily queried and works blazingly fast, and one that indexes by

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Yuri Levinsky
: Tuesday, June 25, 2013 10:33 PM To: Christopher Browne Cc: Yuri Levinsky; Robert Haas; Bruce Momjian; PostgreSQL Mailing Lists Subject: Re: [HACKERS] Hash partitioning. Christopher Browne cbbro...@gmail.com writes: There would indeed be merit in improving the partitioning apparatus, and actually

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Heikki Linnakangas
On 26.06.2013 11:17, Yuri Levinsky wrote: The main purpose of partitioning in my world is to store billions of rows and be able to search by date, hour or even minute as fast as possible. Hash partitioning sounds like a bad fit for that use case. A regular b-tree, possibly with range

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Bruce Momjian
On Tue, Jun 25, 2013 at 02:52:33PM -0700, Kevin Grittner wrote: Claudio Freire klaussfre...@gmail.com wrote: Did you try select * from foo where (a % 16) = (1::int % 16)? I did.  Using Robert's hashed partitioning table definitions: test=# explain select * from foo where a = 1 and (a %

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Yuri Levinsky
: [HACKERS] Hash partitioning. On 26.06.2013 11:17, Yuri Levinsky wrote: The main purpose of partitioning in my world is to store billions of rows and be able to search by date, hour or even minute as fast as possible. Hash partitioning sounds like a bad fit for that use case. A regular b

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Markus Wanner
On 06/25/2013 11:52 PM, Kevin Grittner wrote: At least until we have parallel query execution. At *that* point this all changes. Can you elaborate on that, please? I currently have a hard time imagining how partitions can help performance in that case, either. At least compared to modern RAID

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread k...@rice.edu
On Wed, Jun 26, 2013 at 03:47:43PM +0200, Markus Wanner wrote: On 06/25/2013 11:52 PM, Kevin Grittner wrote: At least until we have parallel query execution. At *that* point this all changes. Can you elaborate on that, please? I currently have a hard time imagining how partitions can

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Heikki Linnakangas
On 26.06.2013 16:41, Yuri Levinsky wrote: Heikki, As far as I understand the height of the btree will affect the number of I/Os necessary. The height of the tree does not increase linearly with the number of records. The height of a b-tree is O(log n), where n is the number of records.

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Yuri Levinsky
; Yuri Levinsky; PostgreSQL-Dev Subject: Re: [HACKERS] Hash partitioning. On Wed, Jun 26, 2013 at 03:47:43PM +0200, Markus Wanner wrote: On 06/25/2013 11:52 PM, Kevin Grittner wrote: At least until we have parallel query execution. At *that* point this all changes. Can you elaborate

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Bruce Momjian
On Wed, Jun 26, 2013 at 05:10:00PM +0300, Heikki Linnakangas wrote: In practice, there might be a lot of quirks and inefficiencies and locking contention etc. involved in various DBMS's, that you might be able to work around with hash partitioning. But from a theoretical point of view, there

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Yuri Levinsky
; Bruce Momjian; PostgreSQL Mailing Lists Subject: Re: [HACKERS] Hash partitioning. On 26.06.2013 16:41, Yuri Levinsky wrote: Heikki, As far as I understand the height of the btree will affect the number of I/Os necessary. The height of the tree does not increase linearly with the number

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: On 26.06.2013 11:17, Yuri Levinsky wrote: When you dealing with company, which has ~350.000.000 users, and you don't want to use key/value data stores: you need hash partitioned tables and hash partitioned table clusters to perform fast

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Markus Wanner
On 06/26/2013 04:10 PM, Yuri Levinsky wrote: You typically don't want to use b-tree index when yo select more when ~1-2% of your data. Agreed. Indices on columns with very low selectivity don't perform well. (Postgres knows that and uses a sequential scan based on selectivity estimates. Being

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Markus Wanner
On 06/26/2013 04:01 PM, k...@rice.edu wrote: I think he is referring to the fact that with parallel query execution, multiple partitions can be processed simultaneously instead of serially as they are now with the resulting speed increase. Processing simultaneously is the purpose of parallel

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Bruce Momjian
On Wed, Jun 26, 2013 at 05:04:11PM +0200, Markus Wanner wrote: On 06/26/2013 04:01 PM, k...@rice.edu wrote: I think he is referring to the fact that with parallel query execution, multiple partitions can be processed simultaneously instead of serially as they are now with the resulting

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Kevin Grittner
Markus Wanner mar...@bluegap.ch wrote: On 06/25/2013 11:52 PM, Kevin Grittner wrote: At least until we have parallel query execution.  At *that* point this all changes. Can you elaborate on that, please? I currently have a hard time imagining how partitions can help performance in that case,

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Heikki Linnakangas
On 26.06.2013 18:34, Kevin Grittner wrote: Markus Wannermar...@bluegap.ch wrote: On 06/25/2013 11:52 PM, Kevin Grittner wrote: At least until we have parallel query execution. At *that* point this all changes. Can you elaborate on that, please? I currently have a hard time imagining how

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Markus Wanner
On 06/26/2013 05:46 PM, Heikki Linnakangas wrote: We could also allow a large query to search a single table in parallel. A seqscan would be easy to divide into N equally-sized parts that can be scanned in parallel. It's more difficult for index scans, but even then it might be possible at

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Claudio Freire
On Wed, Jun 26, 2013 at 11:14 AM, Bruce Momjian br...@momjian.us wrote: On Wed, Jun 26, 2013 at 05:10:00PM +0300, Heikki Linnakangas wrote: In practice, there might be a lot of quirks and inefficiencies and locking contention etc. involved in various DBMS's, that you might be able to work

Re: [HACKERS] Hash partitioning.

2013-06-26 Thread Jeff Janes
On Wed, Jun 26, 2013 at 7:01 AM, k...@rice.edu k...@rice.edu wrote: On Wed, Jun 26, 2013 at 03:47:43PM +0200, Markus Wanner wrote: On 06/25/2013 11:52 PM, Kevin Grittner wrote: At least until we have parallel query execution. At *that* point this all changes. Can you elaborate on

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Bruce Momjian
On Tue, Jun 25, 2013 at 03:48:19PM +0300, Yuri Levinsky wrote: Hi, Do we have any plans to implement Hash Partitioning, maybe I missing this feature? You can do it by writing your own constraint and trigger functions that control the hashing. -- Bruce Momjian br...@momjian.us

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Yuri Levinsky
: +972 54 6107703, Office: +972 9 9710239; Fax: +972 9 9710222 -Original Message- From: Bruce Momjian [mailto:br...@momjian.us] Sent: Tuesday, June 25, 2013 4:21 PM To: Yuri Levinsky Cc: pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Hash partitioning. On Tue, Jun 25, 2013 at 03:48

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Bruce Momjian
On Tue, Jun 25, 2013 at 05:19:47PM +0300, Yuri Levinsky wrote: Bruce, Many thanks. According to PostgreSQL documentation it's only range and list partitions are supported. My question is: when I am following your advice, is PostgreSQL will do partitioning pruning on select? My expectation is:

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Robert Haas
On Tue, Jun 25, 2013 at 9:21 AM, Bruce Momjian br...@momjian.us wrote: On Tue, Jun 25, 2013 at 03:48:19PM +0300, Yuri Levinsky wrote: Hi, Do we have any plans to implement Hash Partitioning, maybe I missing this feature? You can do it by writing your own constraint and trigger functions

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Bruce Momjian
On Tue, Jun 25, 2013 at 11:02:40AM -0400, Robert Haas wrote: On Tue, Jun 25, 2013 at 9:21 AM, Bruce Momjian br...@momjian.us wrote: On Tue, Jun 25, 2013 at 03:48:19PM +0300, Yuri Levinsky wrote: Hi, Do we have any plans to implement Hash Partitioning, maybe I missing this feature?

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Robert Haas
On Tue, Jun 25, 2013 at 11:06 AM, Bruce Momjian br...@momjian.us wrote: Not really. Constraint exclusion won't kick in for a constraint like CHECK (hashme(a) % 16 == 3) and a WHERE clause of the form a = 42. Uh, I thought we checked the constant against every CHECK constraint and only

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Bruce Momjian
On Tue, Jun 25, 2013 at 11:15:24AM -0400, Robert Haas wrote: On Tue, Jun 25, 2013 at 11:06 AM, Bruce Momjian br...@momjian.us wrote: Not really. Constraint exclusion won't kick in for a constraint like CHECK (hashme(a) % 16 == 3) and a WHERE clause of the form a = 42. Uh, I thought we

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Robert Haas
On Tue, Jun 25, 2013 at 11:45 AM, Bruce Momjian br...@momjian.us wrote: On Tue, Jun 25, 2013 at 11:15:24AM -0400, Robert Haas wrote: On Tue, Jun 25, 2013 at 11:06 AM, Bruce Momjian br...@momjian.us wrote: Not really. Constraint exclusion won't kick in for a constraint like CHECK (hashme(a)

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: I looked at predtest.c but I can't see how we accept = and = ranges, but not CHECK (a % 16 == 3). It is the '%' operator? I am not sure why the hashme() function is there. Wouldn't it work if hashme() was an immutable function? No. Robert's

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Yuri Levinsky
, Israel Mobile: +972 54 6107703, Office: +972 9 9710239; Fax: +972 9 9710222 -Original Message- From: Robert Haas [mailto:robertmh...@gmail.com] Sent: Tuesday, June 25, 2013 6:55 PM To: Bruce Momjian Cc: Yuri Levinsky; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] Hash partitioning

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Bruce Momjian
On Tue, Jun 25, 2013 at 12:08:34PM -0400, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: I looked at predtest.c but I can't see how we accept = and = ranges, but not CHECK (a % 16 == 3). It is the '%' operator? I am not sure why the hashme() function is there. Wouldn't it work if

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Alvaro Herrera
Yuri Levinsky escribió: As former Oracle DBA and now simple beginner PostgreSQL DBA I would like to say: the current partitioning mechanism might be improved. Sorry, it seems to me far behind yesterday requirements. I don't think you'll find anybody that disagrees with this. -- Álvaro

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Claudio Freire
On Tue, Jun 25, 2013 at 12:55 PM, Robert Haas robertmh...@gmail.com wrote: Let me back up a minute. You told the OP that he could make hash partitioning by writing his own constraint and trigger functions. I think that won't work. But I'm happy to be proven wrong. Do you have an example

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Christopher Browne
On Tue, Jun 25, 2013 at 12:08 PM, Yuri Levinsky yu...@celltick.com wrote: Guys, I am sorry for taking your time. The reason for my question is: As former Oracle DBA and now simple beginner PostgreSQL DBA I would like to say: the current partitioning mechanism might be improved. Sorry, it

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Tom Lane
Christopher Browne cbbro...@gmail.com writes: There would indeed be merit in improving the partitioning apparatus, and actually, I think it's been a couple of years since there has been serious discussion of this. We could certainly use a partitioning mechanism that's easier to use than what

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Claudio Freire
On Tue, Jun 25, 2013 at 4:32 PM, Tom Lane t...@sss.pgh.pa.us wrote: However, I find it hard to think that hash partitioning as such is very high on the to-do list. As was pointed out upthread, the main practical advantage of partitioning is *not* performance of routine queries, but improved

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Kevin Grittner
Claudio Freire klaussfre...@gmail.com wrote: Did you try select * from foo where (a % 16) = (1::int % 16)? I did.  Using Robert's hashed partitioning table definitions: test=# explain select * from foo where a = 1 and (a % 16) = (1 % 16); QUERY PLAN 

Re: [HACKERS] Hash partitioning.

2013-06-25 Thread Claudio Freire
On Tue, Jun 25, 2013 at 6:52 PM, Kevin Grittner kgri...@ymail.com wrote: I agree though, that having an index implementation that can do the first level split faster than any partitioning mechanism can do is better, and that the main benefits of partitioning are in administration, *not*

Re: [HACKERS] New partitioning WAS: Check constraints on partition parents only?

2011-07-28 Thread Robert Haas
On Tue, Jul 26, 2011 at 7:58 PM, Josh Berkus j...@agliodbs.com wrote: Jim, That's why I'd be opposed to any partitioning scheme that removed the ability to have different fields in different children. We've found that ability to be very useful. Likewise, I think we need to have intelligent

Re: [HACKERS] New partitioning WAS: Check constraints on partition parents only?

2011-07-28 Thread Josh Berkus
Robert, If the value is less than v1, put it in a partition called p1. If the value is less than v2, put it in a position called p2. repeat ad nauseum, and then, optionally: If the value is not less than any of the above, put it in a partition called poverflow. I like that design, not

Re: [HACKERS] New partitioning WAS: Check constraints on partition parents only?

2011-07-28 Thread Aidan Van Dyk
On Thu, Jul 28, 2011 at 12:53 PM, Josh Berkus j...@agliodbs.com wrote: Robert, If the value is less than v1, put it in a partition called p1. If the value is less than v2, put it in a position called p2. repeat ad nauseum, and then, optionally: If the value is not less than any of the above,

Re: [HACKERS] New partitioning WAS: Check constraints on partition parents only?

2011-07-28 Thread Martijn van Oosterhout
On Thu, Jul 28, 2011 at 10:20:57AM -0400, Robert Haas wrote: What Itagaki Takahiro proposed a year ago was basically something where you would say, OK, I want to partition on this column (or maybe expression). And then you say: If the value is less than v1, put it in a partition called p1.

Re: [HACKERS] New partitioning WAS: Check constraints on partition parents only?

2011-07-28 Thread Tom Lane
Aidan Van Dyk ai...@highrise.ca writes: On Thu, Jul 28, 2011 at 12:53 PM, Josh Berkus j...@agliodbs.com wrote: Second, the key-based partitioning I described would actually be preferred to what you describe by a lot of users I know, because it's even simpler than what you propose, which means

Re: [HACKERS] New partitioning WAS: Check constraints on partition parents only?

2011-07-26 Thread Josh Berkus
Jim, That's why I'd be opposed to any partitioning scheme that removed the ability to have different fields in different children. We've found that ability to be very useful. Likewise, I think we need to have intelligent plans involving a parent table that's either completely empty or

Re: [HACKERS] Table Partitioning

2011-06-23 Thread Robert Haas
On Tue, Jun 21, 2011 at 1:52 PM, David Fetter da...@fetter.org wrote: Still, I think a pretty clear way forward here is to try to figure out a way to add some explicit syntax for range partitions, so that you can say... foo_a is for all rows where foo starts with 'a' foo_b is for all rows

Re: [HACKERS] Table Partitioning

2011-06-23 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: David Fetter da...@fetter.org wrote: Does The Standard* have anything to say? I don't know I dug around in the standard a bit and didn't find anything. Given that the standard doesn't even touch the issue of indexes because that a performance

Re: [HACKERS] Table Partitioning

2011-06-21 Thread Robert Haas
On Mon, Jun 20, 2011 at 5:42 PM, David Fetter da...@fetter.org wrote: I noticed that we have some nice new speed optimizations (more properly, de-pessimizations) for partitioned tables in 9.1. /me sticks tongue out at dfetter. Anybody care to look over the table partitioning stuff on the wiki

Re: [HACKERS] Table Partitioning

2011-06-21 Thread David Fetter
On Tue, Jun 21, 2011 at 01:07:17PM -0400, Robert Haas wrote: On Mon, Jun 20, 2011 at 5:42 PM, David Fetter da...@fetter.org wrote: I noticed that we have some nice new speed optimizations (more properly, de-pessimizations) for partitioned tables in 9.1. /me sticks tongue out at dfetter.

Re: [HACKERS] Table partitioning - is anything coming?

2010-07-05 Thread Greg Smith
Igor Kryltsov wrote: I am not asking any firm dates but when (and if) do you think roughly it will be any enhancements on automating partitioning in Postgres? The earliest possible date for that is the summer of 2011 when PostgreSQL 9.1 might be released:

Re: [HACKERS] Table Partitioning Feature

2009-02-10 Thread Amit Gupta
Thanks for your feedback, Emmanuel. Here are my comments: On 2/10/09, Emmanuel Cecchet m...@frogthinker.org wrote: Hi Amit, I will be traveling until next Tuesday and will have no access to email so don't be surprised if I don't follow up this week. The overall approach seems sound. The

Re: [HACKERS] Table Partitioning Feature

2009-02-10 Thread Amit Gupta
Hi Robert, I am a little fuzzy on what you're proposing here, but I think you're saying that you're only going to support range partitioning on integers or dates and that you plan to use the text type to store the integer or date values. FWIW, those don't seem like very good decisions

Re: [HACKERS] Table Partitioning Feature

2009-02-09 Thread Amit Gupta
Hi Emmanuel, We are considering to following approach: 1. metadata table pg_partitions is defined as follows: CATALOG(pg_partitions,2336) BKI_WITHOUT_OIDS { Oid partrelid; // partition table Oid Oid parentrelid; // Parent table Oid int4parttype; // Type

Re: [HACKERS] Table Partitioning Feature

2009-02-09 Thread Emmanuel Cecchet
Hi Amit, I will be traveling until next Tuesday and will have no access to email so don't be surprised if I don't follow up this week. The overall approach seems sound. The metadata table should help also for DDL to find out overlapping ranges or duplicate list entries. So far, I have not

Re: [HACKERS] Table Partitioning Feature

2009-02-09 Thread Robert Haas
On Mon, Feb 9, 2009 at 9:16 AM, Amit Gupta amit.pc.gu...@gmail.com wrote: Hi Emmanuel, We are considering to following approach: 1. metadata table pg_partitions is defined as follows: CATALOG(pg_partitions,2336) BKI_WITHOUT_OIDS { Oid partrelid; // partition table Oid Oid

Re: [HACKERS] Table Partitioning Feature

2009-01-27 Thread Amit Gupta
Hi Emmanuel, On 1/26/09, Emmanuel Cecchet m...@frogthinker.org wrote: Hi Amit, I overlooked the fact that you dropped composite partitions and subpartitions template from the proposal presented in http://archives.postgresql.org/pgsql-hackers/2008-01/msg00413.php. Is it because this is too

Re: [HACKERS] Table Partitioning Feature

2009-01-25 Thread Emmanuel Cecchet
Hi Amit, I overlooked the fact that you dropped composite partitions and subpartitions template from the proposal presented in http://archives.postgresql.org/pgsql-hackers/2008-01/msg00413.php. Is it because this is too hard to support? or you don't see any immediate need for it? Thanks,

Re: [HACKERS] Table Partitioning Feature

2009-01-23 Thread Amit Gupta
Hi Emmanuel, Please find my comments in-lined: On 1/23/09, Emmanuel Cecchet m...@frogthinker.org wrote: Amit, You might want to put this on the http://wiki.postgresql.org/wiki/Table_partitioning wiki page. Sure. How does your timeline look like for this implementation? The

Re: [HACKERS] Table Partitioning Feature

2009-01-22 Thread Josh Berkus
Amit, Wow, thanks! As you probably know, we're already in freeze for 8.4. So this patch will need to go on the first commitfest for 8.5, in May or June. --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Table Partitioning Feature

2009-01-22 Thread Emmanuel Cecchet
Amit, You might want to put this on the http://wiki.postgresql.org/wiki/Table_partitioning wiki page. How does your timeline look like for this implementation? I would be happy to contribute C triggers to your implementation. From what I understood in

Re: [HACKERS] Declarative partitioning grammar

2008-01-20 Thread Simon Riggs
On Tue, 2008-01-15 at 17:12 +, Gregory Stark wrote: Wouldn't Segment Exclusion (maybe together with a specialized form of CLUSTERing) handle that case much better than partitioning? Without the need to name all those thousands of partitions and manage them manually. Firstly we

Re: [HACKERS] Declarative partitioning grammar

2008-01-17 Thread Gavin Sherry
On Mon, Jan 14, 2008 at 10:45:28PM -0500, Tom Lane wrote: Jeff Cohen [EMAIL PROTECTED] writes: In the proposed solution, hash and list partitions work for all types that support an equality operator, and range partitions work for all types that support fully-ordered comparison.

Re: [HACKERS] Declarative partitioning grammar

2008-01-16 Thread Zeugswetter Andreas ADI SD
Personally I find the automatic partition idea intriguing, where you only have to choose an expression that equates to one value (value group) per partition (and possibly a way to derive a partition name). IMO, better go right to a fully automated approach. Or why would you need

Re: [HACKERS] Declarative partitioning grammar

2008-01-15 Thread Markus Schiltknecht
Hi Jeff, Jeff Cohen wrote: If you don't define a default partition to handle outliers, the insert should fail with an error. IMO, you should always have a default partition, then, so as not to violate the constraints (by rejecting tuples which are correct according to the constraints).

Re: [HACKERS] Declarative partitioning grammar

2008-01-15 Thread Hannu Krosing
Ühel kenal päeval, E, 2008-01-14 kell 10:49, kirjutas Markus Schiltknecht: Hi, Jeff Cohen wrote: We did look at allowing general functions for partitioning and this was one concern. The other is that we want to enforce that a row only gets inserted into a single partition, so we

Re: [HACKERS] Declarative partitioning grammar

2008-01-15 Thread Markus Schiltknecht
Hi, Hannu Krosing wrote: I guess it would go to some default partition ? Which doesn't have a name so far, which prevents from addressing that partition. Nor is it pretty well defined, it's just a rest. sure, but this can become really tedious for 1024 partitions, Well, managing 1024

Re: [HACKERS] Declarative partitioning grammar

2008-01-15 Thread Hans-Juergen Schoenig
sure, but this can become really tedious for 1024 partitions, Well, managing 1024 partitions manually is a tedious job, no matter what grammar you take: You'll have to deal with 1024 different partition names. What do you need so many partitions for? imagine a structure which is

<    3   4   5   6   7   8   9   10   11   >