Re: [HACKERS] PartitionSchemaData & partcollation (Re: [COMMITTERS] pgsql: Associate partitioning information with each RelOptInfo.)
On 2017/09/28 16:13, Ashutosh Bapat wrote: > On Thu, Sep 28, 2017 at 11:47 AM, Amit Langote wrote: >> On 2017/09/21 12:42, Robert Haas wrote: >>> Associate partitioning information with each RelOptInfo. >>> >>> This is not used for anything yet, but it is necessary infrastructure >>> for partition-wise join and for partition pruning without constraint >>> exclusion. >>> >>> Ashutosh Bapat, reviewed by Amit Langote and with quite a few changes, >>> mostly cosmetic, by me. Additional review and testing of this patch >>> series by Antonin Houska, Amit Khandekar, Rafia Sabih, Rajkumar >>> Raghuwanshi, Thomas Munro, and Dilip Kumar. >> >> I noticed that this commit does not add partcollation field to >> PartitionSchemeData, while it adds parttypcoll. I think it'd be necessary >> to have partcollation too, because partitioning would have used the same, >> not parttypcoll. For, example see the following code in >> partition_rbound_datum_cmp: >> >> cmpval = DatumGetInt32(FunctionCall2Coll(>partsupfunc[i], >> key->partcollation[i], >> rb_datums[i], >> tuple_datums[i])); >> >> So, it would be wrong to use parttypcoll, if we are to use the collation >> to match a clause with the partition bounds when doing partition-pruning. >> Concretely, a clause's inputcollid should match partcollation for the >> corresponding column, not the column's parttypcoll. >> >> Attached is a patch that adds the same. I first thought of including it >> in the partition-pruning patch set [1], but thought we could independently >> fix this. >> > > I think PartitionSchemeData structure will grow as we need more > information about partition key for various things. E.g. partsupfunc > is not part of this structure right now, but it would be required to > compare partition bound datums. Similarly partcollation. Please add > this to partition pruning patchset. May be parttypcoll won't be used > at all and we may want to remove it altogether. Okay, I will post it with the partition-pruning patch set. Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [HACKERS] PartitionSchemaData & partcollation (Re: [COMMITTERS] pgsql: Associate partitioning information with each RelOptInfo.)
Sorry, I meant to say PartitionSchem"e"Data in subject. Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
PartitionSchemaData & partcollation (Re: [COMMITTERS] pgsql: Associate partitioning information with each RelOptInfo.)
On 2017/09/21 12:42, Robert Haas wrote: > Associate partitioning information with each RelOptInfo. > > This is not used for anything yet, but it is necessary infrastructure > for partition-wise join and for partition pruning without constraint > exclusion. > > Ashutosh Bapat, reviewed by Amit Langote and with quite a few changes, > mostly cosmetic, by me. Additional review and testing of this patch > series by Antonin Houska, Amit Khandekar, Rafia Sabih, Rajkumar > Raghuwanshi, Thomas Munro, and Dilip Kumar. I noticed that this commit does not add partcollation field to PartitionSchemeData, while it adds parttypcoll. I think it'd be necessary to have partcollation too, because partitioning would have used the same, not parttypcoll. For, example see the following code in partition_rbound_datum_cmp: cmpval = DatumGetInt32(FunctionCall2Coll(>partsupfunc[i], key->partcollation[i], rb_datums[i], tuple_datums[i])); So, it would be wrong to use parttypcoll, if we are to use the collation to match a clause with the partition bounds when doing partition-pruning. Concretely, a clause's inputcollid should match partcollation for the corresponding column, not the column's parttypcoll. Attached is a patch that adds the same. I first thought of including it in the partition-pruning patch set [1], but thought we could independently fix this. Thoughts? Thanks, Amit [1] https://commitfest.postgresql.org/14/1272/ From 4e25d503a00c5fb42919e73aae037eacb0164af6 Mon Sep 17 00:00:00 2001 From: amit <amitlangot...@gmail.com> Date: Wed, 27 Sep 2017 15:49:50 +0900 Subject: [PATCH 1/5] Add partcollation field to PartitionSchemeData It copies PartitionKeyData.partcollation. We need that in addition to parttypcoll. --- src/backend/optimizer/util/plancat.c | 1 + src/include/nodes/relation.h | 7 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index cac46bedf9..1273f28069 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1897,6 +1897,7 @@ find_partition_scheme(PlannerInfo *root, Relation relation) part_scheme->partopfamily = partkey->partopfamily; part_scheme->partopcintype = partkey->partopcintype; part_scheme->parttypcoll = partkey->parttypcoll; + part_scheme->partcollation = partkey->partcollation; part_scheme->parttyplen = partkey->parttyplen; part_scheme->parttypbyval = partkey->parttypbyval; diff --git a/src/include/nodes/relation.h b/src/include/nodes/relation.h index 48e6012f7f..2adefd0873 100644 --- a/src/include/nodes/relation.h +++ b/src/include/nodes/relation.h @@ -342,6 +342,10 @@ typedef struct PlannerInfo * partition bounds. Since partition key data types and the opclass declared * input data types are expected to be binary compatible (per ResolveOpClass), * both of those should have same byval and length properties. + * + * Since partitioning might be using a collation for a given partition key + * column that is not same as the collation implied by column's type, store + * the same separately. */ typedef struct PartitionSchemeData { @@ -349,7 +353,8 @@ typedef struct PartitionSchemeData int16 partnatts; /* number of partition attributes */ Oid*partopfamily; /* OIDs of operator families */ Oid*partopcintype; /* OIDs of opclass declared input data types */ - Oid*parttypcoll;/* OIDs of collations of partition keys. */ + Oid*parttypcoll;/* OIDs of partition key type collation. */ + Oid*partcollation; /* OIDs of partitioning collation */ /* Cached information about partition key data types. */ int16 *parttyplen; -- 2.11.0 -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Doc: desultory copy-editing for v10 release notes.
On 2017/07/10 12:36, Tom Lane wrote: > Amit Langote <langote_amit...@lab.ntt.co.jp> writes: >> On 2017/07/10 9:11, Tom Lane wrote: >>> Doc: desultory copy-editing for v10 release notes. > >> I see you updated text for the partitioning item: > >> syntax that automatically creates partition constraints and >> - INSERT routing (Amit Langote) >> + handles routing of tuple insertions and updates (Amit Langote) > >> Although I like the new text better, I'm afraid that we don't support >> routing updates yet, only inserts. > > Hm? We correctly handle updates that don't change the partition key > columns, as well as deletes, no? That's true. > The previous text made it sound like *only* the insert case worked properly. So, as long as an UPDATE doesn't change the partition key of a tuple, it's being "routed" correctly, that is, put back into the same partition. To me, the phrase "routing updates" meant "re-routing" a tuple when the partition key change requires it, but I may be wrong. > It might be worth mentioning that you can't move a row into another > partition via UPDATE. Or maybe that's more detail than we need here. > Not sure. Maybe, we can leave that detail out of the release notes. Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Doc: desultory copy-editing for v10 release notes.
On 2017/07/10 9:11, Tom Lane wrote: > Doc: desultory copy-editing for v10 release notes. > > Improve many item descriptions, improve markup, relocate some items > that seemed to be in the wrong section. > > Branch > -- > master > > Details > --- > https://git.postgresql.org/pg/commitdiff/749eceff4a1f9740391b126c81af9fd4bf3b1eaa I see you updated text for the partitioning item: @@ -1574,7 +1553,7 @@ Add table partitioning syntax that automatically creates partition constraints and - INSERT routing (Amit Langote) + handles routing of tuple insertions and updates (Amit Langote) Although I like the new text better, I'm afraid that we don't support routing updates yet, only inserts. create table p (a int) partition by list (a); create table p1 partition of p for values in (1); insert into p values (1); update p set a = a + 1; ERROR: new row for relation "p1" violates partition constraint DETAIL: Failing row contains (2). Routing of updates is being worked on for PG 11 [1]. Attached patch removes "and updates". Thanks, Amit [1] https://commitfest.postgresql.org/14/1023/ diff --git a/doc/src/sgml/release-10.sgml b/doc/src/sgml/release-10.sgml index debaa80099..7ecb66f7af 100644 --- a/doc/src/sgml/release-10.sgml +++ b/doc/src/sgml/release-10.sgml @@ -1553,7 +1553,7 @@ Add table partitioning syntax that automatically creates partition constraints and - handles routing of tuple insertions and updates (Amit Langote) + handles routing of tuple insertions (Amit Langote) -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Apply RLS policies to partitioned tables.
On 2017/06/12 22:46, Tom Lane wrote: > Amit Langote <langote_amit...@lab.ntt.co.jp> writes: >> How about something like the attached? The patch makes >> mtstate->mt_plans[0] to be passed for all invocations of ExecInitQual() on >> WithCheckOption.qual that is being initialized for each partition. > > Well, the question then is whether that behaves correctly. Where it would > matter is if the WCO qual contains a SubPlan, because you'd be adding > multiple instances of the subplan to the PlanState's list. I think it's > probably all right, but it might be wise to add a test case where there > is such a subplan. > > Spacing and comment wording seem a bit random in this, too. Was about to send the updated patch, but noticed that you already committed it after adding the test. Thanks a lot. Regards, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Apply RLS policies to partitioned tables.
On 2017/06/12 11:51, Tom Lane wrote: > Amit Langote <langote_amit...@lab.ntt.co.jp> writes: >> On 2017/06/12 11:09, Joe Conway wrote: >>> On 06/11/2017 05:35 PM, Tom Lane wrote: >>>> First guess is that map_partition_varattnos has forgotten to handle >>>> WithCheckOption.qual. > >>> Drat. I'll take a look, but it would probably be good if someone >>> generally familiar with the partitioned tables patches have a look as well. > >> I am looking too. Commit 587cda35ca3 which added that code did add a test >> in updatable_views.sql, but tests added by this commit have perhaps >> exposed something not previously covered. > > My first guess above is wrong -- the undefined reference is actually > "mtstate->mt_plans[i]". This is because mtstate->mt_num_partitions is > more than mtstate->mt_nplans in the failing case, so that the blithe > assumption that we can use the partition number to index into > mtstate->mt_plans is certainly wrong. Yes, the code should never access mtstate->mt_plans[i] for i > 0, because in this case (the INSERT case), there is only one plan. Unlike UPDATE/DELETE cases, we don't build plans for every partition in the INSERT case. > Don't know how this should > be corrected, offhand, but do we really have more than one plan > for a partitioned query? There can't be, as mentioned, in the INSERT case. How about something like the attached? The patch makes mtstate->mt_plans[0] to be passed for all invocations of ExecInitQual() on WithCheckOption.qual that is being initialized for each partition. Note that we are passing a PlanState that's based on the root parent table's properties to initialize WITH CHECK OPTION quals to be checked against rows that will be inserted into partitions (that is, after tuple-routing). I tried to come up with a test that would cause the existing (unpatched) code to crash, but couldn't. How would one define a policy or WITH CHECK OPTIONS view or something else, such that ExecInitQual() called on WithCheckOptions.qual would try to access parent planstate and would crash once parent becomes an invalid pointer (mtstate->mt_plans[i] for i > 0)? After looking at the code downstream to ExecInitQual(), it seems that the following cases would cases would cause parent planstate to be accessed: * Qual references a whole row var * Qual references an Aggref or a GroupingFunc or a WindowFunc * Qual references a (Alternative) SubPlan Thanks, Amit From 90fdad27c04db70f78c24e4999cd3aef7c06d1f7 Mon Sep 17 00:00:00 2001 From: amit <amitlangot...@gmail.com> Date: Mon, 12 Jun 2017 14:35:05 +0900 Subject: [PATCH] Don't pass invalid PlanState pointer to ExecInitQual Current code in ExecInitModifyTable that initializes WithCheckOption.qual failed to consider that there *isn't* one plan per partition in the ModifyTableState.mt_plans[] array. --- src/backend/executor/nodeModifyTable.c | 16 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index bf26488c51..89991e34ae 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -1841,8 +1841,17 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) if (node->withCheckOptionLists != NIL && mtstate->mt_num_partitions > 0) { List *wcoList; + PlanState *plan; - Assert(operation == CMD_INSERT); + /* +* In case of INSERT on partitioned tables, there is only plan; in the +* planner we don't build one for each partition. Likewise, there is +* only one WITH CHECK OPTIONS list, not one per partition. +*/ + Assert(operation == CMD_INSERT && + list_length(node->withCheckOptionLists) == 1 && + mtstate->mt_nplans == 1); + plan = mtstate->mt_plans[0];/* The only plan */ resultRelInfo = mtstate->mt_partitions; wcoList = linitial(node->withCheckOptionLists); for (i = 0; i < mtstate->mt_num_partitions; i++) @@ -1858,10 +1867,9 @@ ExecInitModifyTable(ModifyTable *node, EState *estate, int eflags) partrel, rel); foreach(ll, mapped_wcoList) { - WithCheckOption *wco = (WithCheckOption *) lfirst(ll); - ExprState *wcoExpr = ExecInitQual((List *) wco->qual, -
Re: [COMMITTERS] pgsql: Apply RLS policies to partitioned tables.
On 2017/06/12 11:09, Joe Conway wrote: > On 06/11/2017 05:35 PM, Tom Lane wrote: >> Joe Conwaywrites: >>> Apply RLS policies to partitioned tables. >> >> Buildfarm member skink has grown a "make check" failure with this commit. >> >> ==28150== VALGRINDERROR-BEGIN >> ==28150== Invalid read of size 8 >> ==28150==at 0x39A355: ExecInitModifyTable (nodeModifyTable.c:1862) >> ==28150==by 0x37F0F8: ExecInitNode (execProcnode.c:168) >> ==28150==by 0x37C219: InitPlan (execMain.c:1044) >> ==28150==by 0x37C3AF: standard_ExecutorStart (execMain.c:256) >> ==28150==by 0x37C4B8: ExecutorStart (execMain.c:151) >> ==28150==by 0x4E: ProcessQuery (pquery.c:157) >> ==28150==by 0x4CCEDF: PortalRunMulti (pquery.c:1287) >> ==28150==by 0x4CDE19: PortalRun (pquery.c:800) >> ==28150==by 0x4C9E85: exec_simple_query (postgres.c:1099) >> ==28150==by 0x4CBF20: PostgresMain (postgres.c:4087) >> ==28150==by 0x44DC04: BackendRun (postmaster.c:4331) >> ==28150==by 0x44FD9B: BackendStartup (postmaster.c:4003) >> ==28150== Address 0xbbdbec8 is 8,008 bytes inside a recently re-allocated >> block of size 8,192 alloc'd >> ==28150==at 0x4C2AB76: malloc (vg_replace_malloc.c:299) >> ==28150==by 0x6002D2: AllocSetAlloc (aset.c:760) >> ==28150==by 0x606EB5: MemoryContextAllocZeroAligned (mcxt.c:791) >> ==28150==by 0x3832B1: CreateExecutorState (execUtils.c:99) >> ==28150==by 0x37C2E6: standard_ExecutorStart (execMain.c:186) >> ==28150==by 0x37C4B8: ExecutorStart (execMain.c:151) >> ==28150==by 0x4E: ProcessQuery (pquery.c:157) >> ==28150==by 0x4CCEDF: PortalRunMulti (pquery.c:1287) >> ==28150==by 0x4CDE19: PortalRun (pquery.c:800) >> ==28150==by 0x4C9E85: exec_simple_query (postgres.c:1099) >> ==28150==by 0x4CBF20: PostgresMain (postgres.c:4087) >> ==28150==by 0x44DC04: BackendRun (postmaster.c:4331) >> ==28150== >> ==28150== VALGRINDERROR-END >> >> The cited line is the ExecInitQual call here: >> >> /* varno = node->nominalRelation */ >> mapped_wcoList = map_partition_varattnos(wcoList, >> node->nominalRelation, >> partrel, rel); >> foreach(ll, mapped_wcoList) >> { >> WithCheckOption *wco = (WithCheckOption *) lfirst(ll); >> ExprState *wcoExpr = ExecInitQual((List *) wco->qual, >>mtstate->mt_plans[i]); >> >> wcoExprs = lappend(wcoExprs, wcoExpr); >> } >> >> First guess is that map_partition_varattnos has forgotten to handle >> WithCheckOption.qual. If so, though, and if that's not resulting >> in visible misbehavior in the regression tests, then we are missing >> a case that the regression tests should be covering. >> >> BTW, it might be advisable to use castNode(WithCheckOption, ...) >> in the line before that. > > Drat. I'll take a look, but it would probably be good if someone > generally familiar with the partitioned tables patches have a look as well. I am looking too. Commit 587cda35ca3 which added that code did add a test in updatable_views.sql, but tests added by this commit have perhaps exposed something not previously covered. Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Logical replication
On Sat, Jan 21, 2017 at 1:19 PM, Amit Kapilawrote: > On Fri, Jan 20, 2017 at 7:36 PM, Peter Eisentraut wrote: >> Logical replication >> >> - Add PUBLICATION catalogs and DDL >> - Add SUBSCRIPTION catalog and DDL >> - Define logical replication protocol and output plugin >> - Add logical replication workers >> >> From: Petr Jelinek >> Reviewed-by: Steve Singer >> Reviewed-by: Andres Freund >> Reviewed-by: Erik Rijkers >> Reviewed-by: Peter Eisentraut >> >> > .. >> 119 files changed, 13354 insertions(+), 95 deletions(-) >> > > > Great work, Congrats Peter Jelinek and Thanks to all involved. > Getting a feature of this magnitude deserves a big round of applause. +1, congrats! Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Fix some more regression test row-order-instability issues.
On Sat, Jan 14, 2017 at 7:32 AM, Tom Lanewrote: > Fix some more regression test row-order-instability issues. > > Commit 0563a3a8b just introduced another instance of the same unsafe > testing methodology that appeared in 2ac3ef7a0, which I corrected in > 257d81572. Robert/Amit, please stop doing that. > > Also look through the rest of f0e44751d's test cases, and correct some > other queries with underdetermined ordering of results from the system > catalogs. These haven't failed in the buildfarm yet, but I don't > have any confidence in that staying true. Sorry about that. I should have taken a cue from the last time you fixed one of these, will be careful henceforth. Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Fix bugs in RelationGetPartitionDispatchInfo.
Hi Alexander, On 2016/12/14 15:18, Alexander Korotkov wrote: > Hmm, Dmitry Ivanov spotted this bug [1]. And he posted this patch with > only difference of variables naming and comments [2]. But he is not > mentioned in commit message. That doesn't look fine. I apologize for my part in failing to repeat in my description of the patch that I sent (which might perhaps be the only thing Robert saw) that Dmitry is the original author of the patch, in addition to being the discoverer of the bug. Thanks again! Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Fix creative, but unportable, spelling of "ptr != NULL".
On 2016/12/13 1:23, Tom Lane wrote: > Fix creative, but unportable, spelling of "ptr != NULL". > > Or at least I suppose that's what was really meant here. But even > aside from the not-per-project-style use of "0" to mean "NULL", > I doubt it's safe to assume that all valid pointers are > NULL. > Per buildfarm member pademelon. Oops, that was definitely unintentional. Thanks for fixing! Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Don't vacuum all-frozen pages.
On 2016/03/11 8:19, Peter Geoghegan wrote: > On Thu, Mar 10, 2016 at 1:22 PM, Andres Freundwrote: >> Yeha! > > Fantastic effort, particularly from Masahiko. Well done. +1! - Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: BRIN: Block Range Indexes
On Sat, Nov 8, 2014 at 4:43 AM, Alvaro Herrera alvhe...@alvh.no-ip.org wrote: src/backend/access/brin/README| 189 A minor typo: +Summarization +- snip + +Wehn VACUUM is run on the table, all unsummarized page ranges are s/Wehn/When/ Thanks, Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Add missing 9.4 release file.
Spotted a typo: + para +worker_spi_launch() in worker_spi shows an example if its use. + /para + /listitem s/example if/example of -- Amit -- Sent via pgsql-committers mailing list (pgsql-committers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-committers
Re: [COMMITTERS] pgsql: Update 9.4 release notes for queryid control
Perhaps a typo/copy-pasto in this commit, ;-) para This allows monitoring tools to only fetch query texts for newly observe entries, as determined by structnamequeryid/. /para s/observe/observed -- Amit