Re: [HACKERS] Declarative partitioning - another take

2016-09-26 Thread Ashutosh Bapat
> > With this patch, the mapping is created *only once* during > RelationBuildPartitionDesc() to assign canonical indexes to individual > list values. The partition OID array will also be rearranged such that > using the new (canonical) index instead of the old > catalog-scan-order-based index wil

Re: [HACKERS] Declarative partitioning - another take

2016-09-26 Thread Amit Langote
On 2016/09/22 19:10, Ashutosh Bapat wrote: > On Thu, Sep 22, 2016 at 1:02 PM, Ashutosh Bapat > wrote: >> For list partitions, the ListInfo stores the index maps for values >> i.e. the index of the partition to which the value belongs. Those >> indexes are same as the indexes in partition OIDs arra

Re: [HACKERS] Declarative partitioning - another take

2016-09-26 Thread Amit Langote
Hi Ashutosh, On 2016/09/22 14:42, Ashutosh Bapat wrote: > Hi Amit, > Following sequence of DDLs gets an error > -- > -- multi-leveled partitions > -- > CREATE TABLE prt1_l (a int, b int, c varchar) PARTITION BY RANGE(a); > CREATE TABLE prt1_l_p1 PARTITION OF prt1_l FOR VALUES START (0) END > (250

Re: [HACKERS] Declarative partitioning - another take

2016-09-22 Thread Ashutosh Bapat
On Thu, Sep 22, 2016 at 1:02 PM, Ashutosh Bapat wrote: > For list partitions, the ListInfo stores the index maps for values > i.e. the index of the partition to which the value belongs. Those > indexes are same as the indexes in partition OIDs array and come from > the catalogs. In case a user cre

Re: [HACKERS] Declarative partitioning - another take

2016-09-22 Thread Ashutosh Bapat
For list partitions, the ListInfo stores the index maps for values i.e. the index of the partition to which the value belongs. Those indexes are same as the indexes in partition OIDs array and come from the catalogs. In case a user creates two partitioned tables with exactly same lists for partitio

Re: [HACKERS] Declarative partitioning - another take

2016-09-21 Thread Ashutosh Bapat
Hi Amit, Following sequence of DDLs gets an error -- -- multi-leveled partitions -- CREATE TABLE prt1_l (a int, b int, c varchar) PARTITION BY RANGE(a); CREATE TABLE prt1_l_p1 PARTITION OF prt1_l FOR VALUES START (0) END (250) PARTITION BY RANGE (b); CREATE TABLE prt1_l_p1_p1 PARTITION OF prt1_l_p1

Re: [HACKERS] Declarative partitioning - another take

2016-09-20 Thread Robert Haas
On Thu, Sep 15, 2016 at 4:53 AM, Amit Langote wrote: > [ new patches ] Re-reviewing 0001. + partexprs + pg_node_tree This documentation doesn't match pg_partition_table.h, which has partexprsrc and partexprbin. I don't understand why it's a good idea to have both, and there seem to b

Re: [HACKERS] Declarative partitioning - another take

2016-09-19 Thread Robert Haas
On Mon, Aug 15, 2016 at 7:21 AM, Amit Langote wrote: >> +if (partexprs) >> +recordDependencyOnSingleRelExpr(&myself, >> +(Node *) partexprs, >> +RelationGetRelid(rel), >> +

Re: [HACKERS] Declarative partitioning - another take

2016-09-18 Thread Amit Langote
On Thu, Sep 15, 2016 at 10:07 PM, Robert Haas wrote: > On Thu, Sep 15, 2016 at 4:53 AM, Amit Langote > wrote: >> Wow, this is bad. What is needed in this case is "canonicalization" of >> the range partition bounds specified in the command. > > I think we shouldn't worry about this. It seems lik

Re: [HACKERS] Declarative partitioning - another take

2016-09-15 Thread Robert Haas
On Thu, Sep 15, 2016 at 4:53 AM, Amit Langote wrote: > Wow, this is bad. What is needed in this case is "canonicalization" of > the range partition bounds specified in the command. I think we shouldn't worry about this. It seems like unnecessary scope creep. All human beings capable of using P

Re: [HACKERS] Declarative partitioning - another take

2016-09-15 Thread Ashutosh Bapat
Hi Amit, It looks like there is some problem while creating paramterized paths for multi-level partitioned tables. Here's a longish testcase CREATE TABLE prt1_l (a int, b int, c varchar) PARTITION BY RANGE(a); CREATE TABLE prt1_l_p1 PARTITION OF prt1_l FOR VALUES START (0) END (250) PARTITION BY

Re: [HACKERS] Declarative partitioning - another take

2016-09-14 Thread Rajkumar Raghuwanshi
I have Continued with testing declarative partitioning with the latest patch. Got some more observation, given below -- Observation 1 : Getting overlap error with START with EXCLUSIVE in range partition. create table test_range_bound ( a int) partition by range(a); --creating a partition to conta

Re: [HACKERS] Declarative partitioning - another take

2016-09-08 Thread Amit Langote
On 2016/09/08 21:38, Rajkumar Raghuwanshi wrote: > On Wed, Sep 7, 2016 at 3:58 PM, Amit Langote wrote: >> On 2016/09/07 17:56, Rajkumar Raghuwanshi wrote: >>> >>> In this case not sure how to create partition table. Do we have something >>> like we have UNBOUNDED for range partition or oracle have

Re: [HACKERS] Declarative partitioning - another take

2016-09-08 Thread Rajkumar Raghuwanshi
On Wed, Sep 7, 2016 at 3:58 PM, Amit Langote wrote: > > Hi, > > On 2016/09/07 17:56, Rajkumar Raghuwanshi wrote: > > Hi, > > > > I have a query regarding list partitioning, > > > > For example if I want to store employee data in a table, with "IT" dept > > employee in emp_p1 partition, "HR" dept

Re: [HACKERS] Declarative partitioning - another take

2016-09-07 Thread Amit Langote
Hi, On 2016/09/07 17:56, Rajkumar Raghuwanshi wrote: > Hi, > > I have a query regarding list partitioning, > > For example if I want to store employee data in a table, with "IT" dept > employee in emp_p1 partition, "HR" dept employee in emp_p2 partition and if > employee belongs to other than t

Re: [HACKERS] Declarative partitioning - another take

2016-09-07 Thread Rajkumar Raghuwanshi
Hi, I have a query regarding list partitioning, For example if I want to store employee data in a table, with "IT" dept employee in emp_p1 partition, "HR" dept employee in emp_p2 partition and if employee belongs to other than these two, should come in emp_p3 partition. In this case not sure how

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Amit Langote
On Tue, Sep 6, 2016 at 9:19 PM, Robert Haas wrote: > On Wed, Aug 31, 2016 at 1:05 PM, Amit Langote > wrote: >>> However, it seems a lot better to make it a property of the parent >>> from a performance point of view. Suppose there are 1000 partitions. >>> Reading one toasted value for pg_class a

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Amit Langote
Hi, On Tue, Sep 6, 2016 at 8:15 PM, Rajkumar Raghuwanshi wrote: > Hi, > > I have applied updated patches given by you, and observe below. > > here in the given example, t6_p3 partition is not allowed to have null, but > I am able to insert it, causing two nulls in the table. > > --create a partit

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Ashutosh Bapat
This patch uses RangeBound structure. There's also a structure defined with the same name in rangetypes.h with some slight differences. Should we rename the one in partition.c as PartRangeBound or something like that to avoid the confusion? -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Robert Haas
On Wed, Aug 31, 2016 at 1:05 PM, Amit Langote wrote: >> However, it seems a lot better to make it a property of the parent >> from a performance point of view. Suppose there are 1000 partitions. >> Reading one toasted value for pg_class and running stringToNode() on >> it is probably a lot faster

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Rajkumar Raghuwanshi
Hi, I have applied updated patches given by you, and observe below. here in the given example, t6_p3 partition is not allowed to have null, but I am able to insert it, causing two nulls in the table. --create a partition table create table t6 (a int, b varchar) partition by list(a); create table

Re: [HACKERS] Declarative partitioning - another take

2016-09-06 Thread Ashutosh Bapat
I found a server crash while running make check in regress folder. with this set of patches. Problem is RelationBuildPartitionKey() partexprsrc may be used uninitialized. Initializing it with NIL fixes the crash. Here's patch to fix it. Came up with the fix after discussion with Amit. On Fri, Aug

Re: [HACKERS] Declarative partitioning - another take

2016-09-02 Thread Amit Langote
On 2016/09/02 15:57, Ashutosh Bapat wrote: > On Fri, Sep 2, 2016 at 12:23 PM, Amit Langote wrote: >> Getting rid of the parent table in the append list by other means may be a >> way to go. We know that the table is empty and safe to just drop. >> > Ok. Does a constraint (1 = 2) or something like

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Ashutosh Bapat
On Fri, Sep 2, 2016 at 12:23 PM, Amit Langote wrote: > On 2016/09/02 15:22, Ashutosh Bapat wrote: > >> > >> > >>> 2. A combination of constraints on the partitions should be applicable > to > >>> the parent. We aren't doing that. > >> > >> How about on seeing that a RELOPT_OTHER_MEMBER_REL is par

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Amit Langote
On 2016/09/02 15:22, Ashutosh Bapat wrote: >> >> >>> 2. A combination of constraints on the partitions should be applicable to >>> the parent. We aren't doing that. >> >> How about on seeing that a RELOPT_OTHER_MEMBER_REL is partitioned parent >> table, we can have get_relation_constraints() includ

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Ashutosh Bapat
> > > > 2. A combination of constraints on the partitions should be applicable to > > the parent. We aren't doing that. > > How about on seeing that a RELOPT_OTHER_MEMBER_REL is partitioned parent > table, we can have get_relation_constraints() include a constant false > clause in the list of const

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Amit Langote
On 2016/09/02 14:38, Ashutosh Bapat wrote: > Here's something I observed with your set of patches posted in June. I have > not checked the latest set of patches. So, if it's something fixed, please > ignore the mail and sorry for me being lazy. > > prt1 is partitioned table and it shows following

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Ashutosh Bapat
Here's something I observed with your set of patches posted in June. I have not checked the latest set of patches. So, if it's something fixed, please ignore the mail and sorry for me being lazy. prt1 is partitioned table and it shows following information with \d+ regression=# \d+ prt1

Re: [HACKERS] Declarative partitioning - another take

2016-09-01 Thread Ashutosh Bapat
> > > I don't think you need to do anything in the path creation code for this. > > As is it flattens all AppendPath hierarchies whether for partitioning or > > inheritance or subqueries. We should leave it as it is. > > I thought it would be convenient for pairwise join code to work with the > hie

Re: [HACKERS] Declarative partitioning - another take

2016-08-31 Thread Amit Langote
On 2016/08/25 16:15, Ashutosh Bapat wrote: > On Thu, Aug 25, 2016 at 12:22 PM, Amit Langote wrote: >> b) >> when accumulating append subpaths, do not flatten a subpath that is itself >> an append when ((AppendPath *) subpath)->path.parent is a RelOptInfo with >> non-NULL partitioning info.Is the la

Re: [HACKERS] Declarative partitioning - another take

2016-08-31 Thread Amit Langote
On 2016/08/31 16:17, Robert Haas wrote: > On Wed, Aug 31, 2016 at 12:37 PM, Amit Langote wrote: >> What I was trying to understand is why this would not be possible >> with a design where partition bound is stored in the catalog as a property >> of individual partitions instead of a design where we

Re: [HACKERS] Declarative partitioning - another take

2016-08-31 Thread Robert Haas
On Wed, Aug 31, 2016 at 12:37 PM, Amit Langote wrote: >>> If we need an AccessExclusiveLock on parent to add/remove a partition >>> (IOW, changing that child table's partitioning information), then do we >>> need to lock the individual partitions when reading partition's >>> information? I mean t

Re: [HACKERS] Declarative partitioning - another take

2016-08-31 Thread Amit Langote
On 2016/08/29 20:53, Robert Haas wrote: > On Fri, Aug 26, 2016 at 1:33 PM, Amit Langote > wrote: >> We do take a lock on the parent because we would be changing its partition >> descriptor (relcache). I changed MergeAttributes() such that an >> AccessExclusiveLock instead of ShareUpdateExclusiveL

Re: [HACKERS] Declarative partitioning - another take

2016-08-29 Thread Robert Haas
On Fri, Aug 26, 2016 at 1:33 PM, Amit Langote wrote: > We do take a lock on the parent because we would be changing its partition > descriptor (relcache). I changed MergeAttributes() such that an > AccessExclusiveLock instead of ShareUpdateExclusiveLock is taken if the > parent is a partitioned t

Re: [HACKERS] Declarative partitioning - another take

2016-08-26 Thread Amit Langote
On 2016/08/18 5:23, Robert Haas wrote: > On Wed, Aug 17, 2016 at 2:21 AM, Amit Langote > wrote: >> I am slightly tempted to eliminate the pg_partition catalog and associated >> syscache altogether and add a column to pg_class as Robert suggested. >> That way, all relid_is_partition() calls will be

Re: [HACKERS] Declarative partitioning - another take

2016-08-25 Thread Ashutosh Bapat
On Thu, Aug 25, 2016 at 12:22 PM, Amit Langote < langote_amit...@lab.ntt.co.jp> wrote: > On 2016/08/22 13:51, Ashutosh Bapat wrote: > > The parent-child relationship of multi-level partitioned tables is not > > retained when creating the AppendRelInfo nodes. We create RelOptInfo > nodes > > for al

Re: [HACKERS] Declarative partitioning - another take

2016-08-24 Thread Amit Langote
On 2016/08/22 13:51, Ashutosh Bapat wrote: > The parent-child relationship of multi-level partitioned tables is not > retained when creating the AppendRelInfo nodes. We create RelOptInfo nodes > for all the leaf and intermediate tables. The AppendRelInfo nodes created > for these RelOptInfos set th

Re: [HACKERS] Declarative partitioning - another take

2016-08-21 Thread Ashutosh Bapat
The parent-child relationship of multi-level partitioned tables is not retained when creating the AppendRelInfo nodes. We create RelOptInfo nodes for all the leaf and intermediate tables. The AppendRelInfo nodes created for these RelOptInfos set the topmost table as the parent of all the leaf and c

Re: [HACKERS] Declarative partitioning - another take

2016-08-17 Thread Robert Haas
On Wed, Aug 17, 2016 at 2:21 AM, Amit Langote wrote: > I am slightly tempted to eliminate the pg_partition catalog and associated > syscache altogether and add a column to pg_class as Robert suggested. > That way, all relid_is_partition() calls will be replaced by > rel->rd_partbound != NULL check

Re: [HACKERS] Declarative partitioning - another take

2016-08-16 Thread Ashutosh Bapat
On Wed, Aug 17, 2016 at 11:51 AM, Amit Langote < langote_amit...@lab.ntt.co.jp> wrote: > On 2016/08/17 14:33, Ashutosh Bapat wrote: > >> +relid_is_partition(Oid relid) > >> +{ > >> + return SearchSysCacheExists1(PARTRELID, > ObjectIdGetDatum(relid)); > >> +} > >> > >> This is used in a lot o

Re: [HACKERS] Declarative partitioning - another take

2016-08-16 Thread Amit Langote
On 2016/08/17 14:33, Ashutosh Bapat wrote: >> +relid_is_partition(Oid relid) >> +{ >> + return SearchSysCacheExists1(PARTRELID, ObjectIdGetDatum(relid)); >> +} >> >> This is used in a lot of places, and the overhead of checking it in >> all of those places is not necessarily nil. Syscache lo

Re: [HACKERS] Declarative partitioning - another take

2016-08-16 Thread Ashutosh Bapat
> +relid_is_partition(Oid relid) > +{ > + return SearchSysCacheExists1(PARTRELID, ObjectIdGetDatum(relid)); > +} > > This is used in a lot of places, and the overhead of checking it in > all of those places is not necessarily nil. Syscache lookups aren't > free. What if we didn't create a n

Re: [HACKERS] Declarative partitioning - another take

2016-08-16 Thread Robert Haas
On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote wrote: > 0003-Catalog-and-DDL-for-partition-bounds.patch > > Partition DDL includes both a way to create new partition and "attach" an > existing table as a partition of parent partitioned table. Attempt to > drop a partition using DROP TABLE causes a

Re: [HACKERS] Declarative partitioning - another take

2016-08-16 Thread Robert Haas
On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote wrote: > 0002-psql-and-pg_dump-support-for-partitioned-tables.patch +if (pset.sversion >= 90600 && tableinfo.relkind == 'P') Version check is redundant, right? +) PARTITION BY RANGE ((a+b)); +\d describe_range_key +Partitioned table "public.desc

Re: [HACKERS] Declarative partitioning - another take

2016-08-16 Thread Robert Eckhardt
On Tue, Aug 16, 2016 at 2:30 AM, Ashutosh Bapat < ashutosh.ba...@enterprisedb.com> wrote: > > > > >> I think it makes sense to keep calling it a table because it has all the >> logical properties of a table even though it will differ from a regular >> table on the basis of physical implementation

Re: [HACKERS] Declarative partitioning - another take

2016-08-15 Thread Ashutosh Bapat
> I think it makes sense to keep calling it a table because it has all the > logical properties of a table even though it will differ from a regular > table on the basis of physical implementation details such as that it does > not own physical storage. Am I missing something? > > > > > + par

Re: [HACKERS] Declarative partitioning - another take

2016-08-15 Thread Amit Langote
Thanks a lot for taking a look at this. On 2016/08/11 3:22, Robert Haas wrote: > On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote > wrote: >> Attached is the latest set of patches to implement declarative >> partitioning. > > Cool. I would encourage you to give some thought to what is the least >

Re: [HACKERS] Declarative partitioning - another take

2016-08-10 Thread Robert Haas
On Wed, Aug 10, 2016 at 7:09 AM, Amit Langote wrote: > Attached is the latest set of patches to implement declarative > partitioning. Cool. I would encourage you to give some thought to what is the least committable subset of these patches, and think about whether it can be reorganized to make t

Re: [HACKERS] Declarative partitioning

2016-08-10 Thread Amit Langote
On 2016/08/10 19:18, Ashutosh Bapat wrote: > FOR VALUE clause of a partition does not allow a constant expression like > (1/5 -1). It gives syntax error > regression=# create table pt1_p1 partition of pt1 for values start (0) end > ((1/5) - 1); > ERROR: syntax error at or near "(" > LINE 1

Re: [HACKERS] Declarative partitioning

2016-08-10 Thread Ashutosh Bapat
FOR VALUE clause of a partition does not allow a constant expression like (1/5 -1). It gives syntax error regression=# create table pt1_p1 partition of pt1 for values start (0) end ((1/5) - 1); ERROR: syntax error at or near "(" LINE 1: ...pt1_p1 partition of pt1 for values start (0) end (

Re: [HACKERS] Declarative partitioning

2016-08-09 Thread Ashutosh Bapat
What strikes me odd about these patches is RelOptInfo has remained unmodified. For a base partitioned table, I would expect it to be marked as partitioned may be indicating the partitioning scheme. Instead of that, I see that the code directly deals with PartitionDesc, PartitionKey which are part o

Re: [HACKERS] Declarative partitioning

2016-08-08 Thread Amit Langote
On 2016/08/09 6:02, Robert Haas wrote: > On Mon, Aug 8, 2016 at 1:40 AM, Amit Langote > wrote: >>> +1, if we could do it. It will need a change in the way Amit's patch stores >>> partitioning scheme in PartitionDesc. >> >> Okay, I will try to implement this in the next version of the patch. >> >>

Re: [HACKERS] Declarative partitioning

2016-08-08 Thread Robert Haas
On Mon, Aug 8, 2016 at 1:40 AM, Amit Langote wrote: >> +1, if we could do it. It will need a change in the way Amit's patch stores >> partitioning scheme in PartitionDesc. > > Okay, I will try to implement this in the next version of the patch. > > One thing that comes to mind is what if a user wa

Re: [HACKERS] Declarative partitioning

2016-08-07 Thread Ashutosh Bapat
> IIUC, this seems like a combination of 2 and 3 above: > > So, we have the following list partitions (as read from the catalog) > > Table 1: p1 {'b', 'f'}, p2 {'c', 'd'}, p3 {'a', 'e'} > Table 2: p1 {'c', 'd'}, p2 {'a', 'e'}, p3 {'b', 'f'} > > By applying the method of 3: > > Table 1: p3 {'a', 'e'

Re: [HACKERS] Declarative partitioning

2016-08-07 Thread Amit Langote
On 2016/08/05 21:38, Ashutosh Bapat wrote: >>> Consider lists ('e', 'i', 'f'), ('h', 'd','m') and ('l', 'b', 'a') for a >>> list partitioned tables. I am suggesting that we arrange them as >>> ('a','b','l'), ('d', 'h', 'm') and ('e', 'f', 'i'). If the given row >> (either >>> for comparison or for

Re: [HACKERS] Declarative partitioning

2016-08-05 Thread Ashutosh Bapat
> > > > > Consider lists ('e', 'i', 'f'), ('h', 'd','m') and ('l', 'b', 'a') for a > > list partitioned tables. I am suggesting that we arrange them as > > ('a','b','l'), ('d', 'h', 'm') and ('e', 'f', 'i'). If the given row > (either > > for comparison or for inserting) has value 'c', we will sear

Re: [HACKERS] Declarative partitioning

2016-08-05 Thread Robert Haas
On Fri, Aug 5, 2016 at 8:10 AM, Ashutosh Bapat wrote: > On Fri, Aug 5, 2016 at 5:21 PM, Robert Haas wrote: >> On Fri, Aug 5, 2016 at 6:53 AM, Ashutosh Bapat >> wrote: >> > The lists for list partitioned tables are stored as they are specified >> > by >> > the user. While searching for a partitio

Re: [HACKERS] Declarative partitioning

2016-08-05 Thread Ashutosh Bapat
On Fri, Aug 5, 2016 at 5:21 PM, Robert Haas wrote: > On Fri, Aug 5, 2016 at 6:53 AM, Ashutosh Bapat > wrote: > > The lists for list partitioned tables are stored as they are specified by > > the user. While searching for a partition to route tuple to, we compare > it > > with every list value of

Re: [HACKERS] Declarative partitioning

2016-08-05 Thread Robert Haas
On Fri, Aug 5, 2016 at 6:53 AM, Ashutosh Bapat wrote: > The lists for list partitioned tables are stored as they are specified by > the user. While searching for a partition to route tuple to, we compare it > with every list value of every partition. We might do something better > similar to what'

Re: [HACKERS] Declarative partitioning

2016-08-05 Thread Ashutosh Bapat
The lists for list partitioned tables are stored as they are specified by the user. While searching for a partition to route tuple to, we compare it with every list value of every partition. We might do something better similar to what's been done to range partitions. The list of values for a given

Re: [HACKERS] Declarative partitioning

2016-07-20 Thread Amit Langote
On 2016/07/19 22:53, Ashutosh Bapat wrote: > I am seeing following warning with this set of patches. > gram.y:4734:24: warning: assignment from incompatible pointer type [enabled > by default] Thanks, will fix. Was a copy-paste error. Thanks, Amit -- Sent via pgsql-hackers mailing list (pg

Re: [HACKERS] Declarative partitioning

2016-07-19 Thread Ashutosh Bapat
I am seeing following warning with this set of patches. gram.y:4734:24: warning: assignment from incompatible pointer type [enabled by default] On Tue, Jul 5, 2016 at 10:18 AM, Amit Langote wrote: > On 2016/07/04 21:31, Ashutosh Bapat wrote: > > Hi Amit, > > I observed that the ChangeVarNodes ca

Re: [HACKERS] Declarative partitioning

2016-07-04 Thread Amit Langote
On 2016/07/04 21:31, Ashutosh Bapat wrote: > Hi Amit, > I observed that the ChangeVarNodes call at line 1229 in > get_relation_constraints() (code below) changes the varno in the cached > copy of partition key expression, which is undesirable. The reason for this > seems to be that the RelationGetP

Re: [HACKERS] Declarative partitioning

2016-07-04 Thread Ashutosh Bapat
Hi Amit, I observed that the ChangeVarNodes call at line 1229 in get_relation_constraints() (code below) changes the varno in the cached copy of partition key expression, which is undesirable. The reason for this seems to be that the RelationGetPartitionCheckQual() returns the copy of partition key

Re: [HACKERS] Declarative partitioning

2016-06-24 Thread Ashutosh Bapat
Hi Amit, I tried creating 2-level partitioned table and tried to create simple table using CTAS from the partitioned table. It gives a cache lookup error. Here's the test CREATE TABLE pt1_l (a int, b varchar, c int) PARTITION BY RANGE(a); CREATE TABLE pt1_l_p1 PARTITION OF pt1_l FOR VALUES START (1

Re: [HACKERS] Declarative partitioning

2016-06-21 Thread Amit Langote
On 2016/06/21 16:49, Ashutosh Bapat wrote: > Hi Amit, > I tried creating partitioned table by range on an expression like > CREATE TABLE pt1_e (a int, b int, c varchar) PARTITION BY RANGE(a + > b); > ERROR: syntax error at or near > "+" > > LINE 1: ...E pt1_e (a int, b int, c varchar) PARTITI

Re: [HACKERS] Declarative partitioning

2016-06-21 Thread Ashutosh Bapat
Hi Amit, I tried creating partitioned table by range on an expression like CREATE TABLE pt1_e (a int, b int, c varchar) PARTITION BY RANGE(a + b); ERROR: syntax error at or near "+" LINE 1: ...E pt1_e (a int, b int, c varchar) PARTITION BY RANGE(a + b); But when I try to create partitioned t

Re: [HACKERS] Declarative partitioning

2016-06-08 Thread Ashutosh Bapat
0003-... patch does not apply cleanly. It has some conflicts in pg_dump.c. I have tried fixing the conflict in attached patch. On Mon, May 23, 2016 at 3:35 PM, Amit Langote wrote: > > Hi Ildar, > > On 2016/05/21 0:29, Ildar Musin wrote: > > On 20.05.2016 11:37, Amit Langote wrote: > >> Moreover,

Re: [HACKERS] Declarative partitioning

2016-05-20 Thread Ildar Musin
Hi Amit, On 20.05.2016 11:37, Amit Langote wrote: Perhaps you're already aware but may I also suggest looking at how clauses are matched to indexes? For example, consider how match_clauses_to_index() in src/backend/optimizer/path/indxpath.c works. Thanks, I'll take a closer look at it. Moreov

Re: [HACKERS] Declarative partitioning

2016-05-19 Thread Amit Langote
On 2016/05/19 2:48, Tom Lane wrote: > Amit Langote writes: >> On 2016/05/18 2:22, Tom Lane wrote: >>> The two ways that we've dealt with this type of hazard are to copy data >>> out of the relcache before using it; or to give the relcache the >>> responsibility of not moving a particular portion o

Re: [HACKERS] Declarative partitioning

2016-05-18 Thread Tom Lane
Amit Langote writes: > On 2016/05/18 2:22, Tom Lane wrote: >> The two ways that we've dealt with this type of hazard are to copy data >> out of the relcache before using it; or to give the relcache the >> responsibility of not moving a particular portion of data if it did not >> change. From memo

Re: [HACKERS] Declarative partitioning

2016-05-18 Thread Ildar Musin
Hi Amit and all, On 18.05.2016 04:26, Amit Langote wrote: On 2016/05/18 2:22, Tom Lane wrote: The two ways that we've dealt with this type of hazard are to copy data out of the relcache before using it; or to give the relcache the responsibility of not moving a particular portion of data if it

Re: [HACKERS] Declarative partitioning

2016-05-17 Thread Amit Langote
On 2016/05/18 2:22, Tom Lane wrote: > Amit Langote writes: >> On 2016/05/16 22:12, Ildar Musin wrote: >>> Could you please tell is >>> it possible that relcache invalidation occurs during SELECT/UPDATE/DELETE >>> query? > >> Hmm, I think invalidation would not occur mid-query since it would have

Re: [HACKERS] Declarative partitioning

2016-05-17 Thread Tom Lane
Amit Langote writes: > On 2016/05/16 22:12, Ildar Musin wrote: >> Could you please tell is >> it possible that relcache invalidation occurs during SELECT/UPDATE/DELETE >> query? > Hmm, I think invalidation would not occur mid-query since it would have > acquired a lock on the table. This is inco

Re: [HACKERS] Declarative partitioning

2016-05-17 Thread Amit Langote
Hi Ildar, On 2016/05/16 22:12, Ildar Musin wrote: > Hi Amit, > > I'm running some experiments based on your infrastructure trying to > optimize SELECT queries. At some point I need to get PartitionDesc for > relation and to do it I'm using RelationGetPartitionDesc() function. > Problem is that t

Re: [HACKERS] Declarative partitioning

2016-05-16 Thread Ildar Musin
Hi Amit, I'm running some experiments based on your infrastructure trying to optimize SELECT queries. At some point I need to get PartitionDesc for relation and to do it I'm using RelationGetPartitionDesc() function. Problem is that this function copies relcache data and it can be quite slow

Re: [HACKERS] Declarative partitioning

2016-05-12 Thread Amit Langote
Hi, On 2016/05/12 17:42, Sameer Thakur-2 wrote: > Hello Amit, > In the example >> create table part201606week4 partition of parted >> for values start (2016, 6, 2) end (2016, 6, 29); > > seems to be a typo Oops, there indeed is. create table part201606week4 partition of parted for values sta

Re: [HACKERS] Declarative partitioning

2016-05-12 Thread Sameer Thakur-2
Hello Amit, In the example >create table part201606week4 partition of parted >for values start (2016, 6, 2) end (2016, 6, 29); seems to be a typo regards Sameer -- View this message in context: http://postgresql.nabble.com/Declarative-partitioning-tp5862462p5903204.html Sent from the Postgr

Re: [HACKERS] Declarative partitioning

2016-05-10 Thread Amit Langote
Hi Ashutosh, On 2016/05/09 20:21, Ashutosh Bapat wrote: > Hi Amit, > I am trying multi-column/expression partitions. Thanks for the tests. > create table t1_multi_col (a int, b int) partition by range (a, b); > create table t1_mc_p1 partition of t1_multi_col for values start (1, 200) > end (100

Re: [HACKERS] Declarative partitioning

2016-05-09 Thread Ashutosh Bapat
Hi Amit, I am trying multi-column/expression partitions. create table t1_multi_col (a int, b int) partition by range (a, b); create table t1_mc_p1 partition of t1_multi_col for values start (1, 200) end (100, 300); create table t1_mc_p2 partition of t1_multi_col for values start (200, 1) end (300,

Re: [HACKERS] Declarative partitioning

2016-04-26 Thread Erik Rijkers
On 2016-04-15 04:35, Amit Langote wrote: A quick test with: 0001-Add-syntax-to-specify-partition-key-v3.patch 0002-Infrastructure-for-creation-of-partitioned-tables-v3.patch 0003-Add-syntax-to-create-partitions-v3.patch 0004-Infrastructure-for-partition-metadata-storage-and-ma-v3.patch 0005-Int

Re: [HACKERS] Declarative partitioning

2016-04-20 Thread Amit Langote
Hi Ildar, On Wed, Apr 20, 2016 at 11:46 PM, Ildar Musin wrote: > Thanks for clarification! I tried the updated patch, now it works correctly. Great, thanks! > I encountered another problem that concerns expressions as partitioning key. > Probably there is still some work in progress. But if it

Re: [HACKERS] Declarative partitioning

2016-04-20 Thread Ildar Musin
Hi Amit, On 20.04.2016 13:28, Amit Langote wrote: On 2016/04/19 23:52, Amit Langote wrote: On Tue, Apr 19, 2016 at 11:26 PM, Alexander Korotkov Another question is that it might be NOT what users expect from that. From the syntax side it very looks like defining something boxes regions for tw

Re: [HACKERS] Declarative partitioning

2016-04-19 Thread Amit Langote
Hi Idlar, Alexander, On Tue, Apr 19, 2016 at 11:26 PM, Alexander Korotkov wrote: > On Tue, Apr 19, 2016 at 4:57 PM, Ildar Musin wrote: >> >> Thanks for your new patch! I've tried it and discovered some strange >> behavior for partitioning by composite key. Here is an example of my setup: >> >> c

Re: [HACKERS] Declarative partitioning

2016-04-19 Thread Alexander Korotkov
On Tue, Apr 19, 2016 at 4:57 PM, Ildar Musin wrote: > On 15.04.2016 07:35, Amit Langote wrote: > >> Thanks a lot for the comments. The patch set changed quite a bit since >> the last version. Once the CF entry was marked returned with feedback on >> March 22, I held off sending the new version at

Re: [HACKERS] Declarative partitioning

2016-04-19 Thread Ildar Musin
Hi Amit, On 15.04.2016 07:35, Amit Langote wrote: Thanks a lot for the comments. The patch set changed quite a bit since the last version. Once the CF entry was marked returned with feedback on March 22, I held off sending the new version at all. Perhaps, it would have been OK. Anyway here it

Re: [HACKERS] Declarative partitioning

2016-04-18 Thread Amit Langote
On 2016/04/18 15:33, Ashutosh Bapat wrote: >> For time being, I will leave this as yet unaddressed (I am thinking about >> what is reasonable to do for this also considering Robert's comment). Is >> that OK? >> > > Right now EXPLAIN of select * from t1, where t1 is partitioned table shows > Appen

Re: [HACKERS] Declarative partitioning

2016-04-18 Thread Ashutosh Bapat
On Mon, Apr 18, 2016 at 1:23 PM, Amit Langote wrote: > On 2016/04/18 15:38, Ashutosh Bapat wrote: > >> There was no KeyTypeCollInfo in early days of the patch and then I found > >> myself doing a lot of: > >> > >> partexprs_item = list_head(key->partexprs); > >> for (attr in key->partattrs) > >>

Re: [HACKERS] Declarative partitioning

2016-04-18 Thread Amit Langote
On 2016/04/18 15:38, Ashutosh Bapat wrote: >> There was no KeyTypeCollInfo in early days of the patch and then I found >> myself doing a lot of: >> >> partexprs_item = list_head(key->partexprs); >> for (attr in key->partattrs) >> { >> if (attr->attnum != 0) >> { >> // simple column

Re: [HACKERS] Declarative partitioning

2016-04-17 Thread Ashutosh Bapat
> On 2016/04/15 18:46, Ashutosh Bapat wrote: > > > > 3. PartitionKeyData contains KeyTypeCollInfo, whose contents can be > > obtained by calling functions exprType, exprTypemod on partexprs. Why do > we > > need to store that information as a separate member? > > There was no KeyTypeCollInfo in ear

Re: [HACKERS] Declarative partitioning

2016-04-17 Thread Ashutosh Bapat
> >> Regarding 6, it seems to me that because Append does not have a > associated > >> relid (like scan nodes have with scanrelid). Maybe we need to either > fix > >> Append or create some enhanced version of Append which would also > support > >> dynamic pruning. > >> > > > > Right, I think, Appe

Re: [HACKERS] Declarative partitioning

2016-04-17 Thread Ashutosh Bapat
On Fri, Apr 15, 2016 at 10:30 PM, Robert Haas wrote: > On Fri, Apr 15, 2016 at 5:46 AM, Ashutosh Bapat > wrote: > > Retaining the partition hierarchy would help to push-down join across > > partition hierarchy effectively. > > -1. You don't get to insert cruft into the final plan for the > conv

Re: [HACKERS] Declarative partitioning

2016-04-17 Thread Amit Langote
Hi, On 2016/04/15 18:46, Ashutosh Bapat wrote: > > 3. PartitionKeyData contains KeyTypeCollInfo, whose contents can be > obtained by calling functions exprType, exprTypemod on partexprs. Why do we > need to store that information as a separate member? There was no KeyTypeCollInfo in early days

Re: [HACKERS] Declarative partitioning

2016-04-17 Thread Amit Langote
Hi Ashutosh, On 2016/04/15 18:48, Ashutosh Bapat wrote: > With the new set of patches, I am getting following warnings but no > compilation failures. Patches apply smoothly. > partition.c:1216:21: warning: variable ‘form’ set but not used > [-Wunused-but-set-variable] > partition.c:1637:20: warni

Re: [HACKERS] Declarative partitioning

2016-04-15 Thread Robert Haas
On Fri, Apr 15, 2016 at 5:46 AM, Ashutosh Bapat wrote: > Retaining the partition hierarchy would help to push-down join across > partition hierarchy effectively. -1. You don't get to insert cruft into the final plan for the convenience of the optimizer. I think the AppendPath needs to be annota

Re: [HACKERS] Declarative partitioning

2016-04-15 Thread Ashutosh Bapat
With the new set of patches, I am getting following warnings but no compilation failures. Patches apply smoothly. partition.c:1216:21: warning: variable ‘form’ set but not used [-Wunused-but-set-variable] partition.c:1637:20: warning: variable ‘form’ set but not used [-Wunused-but-set-variable] Fo

Re: [HACKERS] Declarative partitioning

2016-04-15 Thread Ashutosh Bapat
With the new set of patches, I am getting following warnings but no compilation failures. Patches apply smoothly. partition.c:1216:21: warning: variable ‘form’ set but not used [-Wunused-but-set-variable] partition.c:1637:20: warning: variable ‘form’ set but not used [-Wunused-but-set-variable] Fo

Re: [HACKERS] Declarative partitioning

2016-04-14 Thread Ashutosh Bapat
Hi Amit, Here are some random comments. You said that you were about to post a new patch, so you might have already taken care of those comments. But anyway here they are. 1. Following patches do not apply cleanly for me. 0001 0003 - conflicts at #include for partition.h in rel.h. 0004 -

Re: [HACKERS] Declarative partitioning

2016-03-21 Thread Amit Langote
On 2016/03/22 4:55, Robert Haas wrote: > So, the last patch on this thread was posted on February 17th, and the > CF entry was marked Waiting on Author on March 2nd. Even if we had a > new patch in hand at this point, I don't think there's any real chance > of being able to get this done for 9.6;

Re: [HACKERS] Declarative partitioning

2016-03-21 Thread Simon Riggs
On 21 March 2016 at 19:55, Robert Haas wrote: > On Wed, Mar 16, 2016 at 10:49 AM, Alexander Korotkov > wrote: > >> > I'd like to validate that this development plan doesn't overlaps with > >> > your > >> > plans. If out plans are not overlapping then let's accept this plan > of > >> > work > >>

<    1   2   3   4   5   6   >