Re: Record a Bitmapset of non-pruned partitions

2021-08-01 Thread Amit Langote
On Sun, Aug 1, 2021 at 9:31 PM David Rowley wrote: > On Fri, 30 Jul 2021 at 19:10, Amit Langote wrote: > > interleaved_parts idea looks clever. I wonder if you decided that > > it's maybe not worth setting that field in the joinrel's > > PartitionBoundInfos? Fo

Re: Record a Bitmapset of non-pruned partitions

2021-07-30 Thread Amit Langote
s [ in terms of partitioning-defined ordering ]." Looks fine otherwise. -- Amit Langote EDB: http://www.enterprisedb.com

Re: a thinko in b676ac443b6

2021-07-27 Thread Amit Langote
On Wed, Jul 28, 2021 at 1:07 AM Tomas Vondra wrote: > On 7/27/21 4:28 AM, Amit Langote wrote: > > I think it can be incorrect to use the same TupleDesc for both the > > slots in ri_Slots (for ready-to-be-inserted tuples) and ri_PlanSlots > > (for subplan output tuples

Re: Allow batched insert during cross-partition updates

2021-07-26 Thread Amit Langote
On Thu, Jul 22, 2021 at 2:18 PM vignesh C wrote: > On Fri, Jul 2, 2021 at 7:35 AM Amit Langote wrote: > > On Fri, Jul 2, 2021 at 1:39 AM Tom Lane wrote: > > > > > > Amit Langote writes: > > > > [ v6-0001-Allow-batching-of-inserts-during-cross-partition-.

a thinko in b676ac443b6

2021-07-26 Thread Amit Langote
x27;s TupleDesc exactly (plus any junk columns), but now it produces only a partial tuple containing the values for the changed columns. So it's better to revert to using planSlot->tts_tupleDescriptor for the slots in ri_PlanSlots. Attached a patch to do so. -- Amit Lango

Re: a misbehavior of partition row movement (?)

2021-07-13 Thread Amit Langote
Hi Ibrar, Sawada-san, On Tue, Jul 13, 2021 at 20:25 Ibrar Ahmed wrote: > > > On Fri, Apr 2, 2021 at 6:09 PM Amit Langote > wrote: > >> On Thu, Apr 1, 2021 at 10:56 AM Masahiko Sawada >> wrote: >> > On Tue, Mar 23, 2021 at 6:27 PM Amit Langote >> wro

Re: ExecRTCheckPerms() and many prunable partitions

2021-07-07 Thread Amit Langote
On Wed, Jul 7, 2021 at 1:41 PM David Rowley wrote: > On Fri, 2 Jul 2021 at 12:41, Amit Langote wrote: > > I'll mark the CF entry as WoA, unless you'd rather I just mark it RwF. > > I've set it to waiting on author. It was still set to needs review. Sorry it slipped

Re: simplifying foreign key/RI checks

2021-07-05 Thread Amit Langote
d patches attached. -- Amit Langote EDB: http://www.enterprisedb.com v9-0001-Export-get_partition_for_tuple.patch Description: Binary data v9-0002-Avoid-using-SPI-for-some-RI-checks.patch Description: Binary data

Re: Allow batched insert during cross-partition updates

2021-07-01 Thread Amit Langote
On Fri, Jul 2, 2021 at 1:39 AM Tom Lane wrote: > > Amit Langote writes: > > [ v6-0001-Allow-batching-of-inserts-during-cross-partition-.patch ] > > Per the cfbot, this isn't applying anymore, so I'm setting it back > to Waiting on Author. Rebased patch att

Re: New committers: Daniel Gustafsson and John Naylor

2021-07-01 Thread Amit Langote
> > > Please join me in wishing them much success and few bugs. > > > > regards, tom lane > > Congrats to you both ! +1, congrats Daniel & John! -- Amit Langote EDB: http://www.enterprisedb.com

Re: ExecRTCheckPerms() and many prunable partitions

2021-07-01 Thread Amit Langote
On Fri, Jul 2, 2021 at 12:45 AM Tom Lane wrote: > Amit Langote writes: > > The problem is that it loops over the entire range table even though > > only one or handful of those entries actually need their permissions > > checked. Most entries, especially those of partition

Re: Record a Bitmapset of non-pruned partitions

2021-06-30 Thread Amit Langote
worthwhile if only for that purpose. He'd suggested that we fix things elsewhere such that that function is not needed in the first place [2], something I keep thinking about in between doing other things, but never sit down to actually write a patch. Given that you're proposing more uses for live_parts, maybe he'd be open to the idea. -- Amit Langote EDB: http://www.enterprisedb.com [1] https://www.postgresql.org/message-id/3f280722-46f2-c2a4-4c19-2cfa28c6c1cd%40lab.ntt.co.jp [2] https://www.postgresql.org/message-id/3529.1554051867%40sss.pgh.pa.us

Re: ExecRTCheckPerms() and many prunable partitions

2021-06-30 Thread Amit Langote
On Wed, Jun 30, 2021 at 23:34 David Rowley wrote: > On Thu, 1 Jul 2021 at 01:34, Amit Langote wrote: > > For now, I have implemented the idea 2 as the attached patch. > > I only just had a fleeting glance at the patch. Aren't you > accidentally missing the 0th RTE

ExecRTCheckPerms() and many prunable partitions

2021-06-30 Thread Amit Langote
ueries. So while still kind of PoC, will add this to July CF for keeping track. -- Amit Langote EDB: http://www.enterprisedb.com [1] https://www.postgresql.org/message-id/ca+hiwqg7zrubmmih3wpsbz4s0h2ehywrnxeducky5hr3fwz...@mail.gmail.com [2] https://w

Re: Deadlock risk while inserting directly into partition?

2021-06-27 Thread Amit Langote
On Mon, Jun 28, 2021 at 12:58 PM Amit Langote wrote: > On Fri, Jun 25, 2021 at 10:26 AM David Rowley wrote: > > On Thu, 24 Jun 2021 at 12:32, David Rowley wrote: > > > The overhead of taking these locks is pretty significant for > > > partitioned tables with lots of

Re: Deadlock risk while inserting directly into partition?

2021-06-27 Thread Amit Langote
/message-id/16db1458-67cf-4add-736e-31b053115e8e%40lab.ntt.co.jp Maybe a good idea to add a line or 2 in 5.11. Table Partitioning? -- Amit Langote EDB: http://www.enterprisedb.com

Re: Deadlock risk while inserting directly into partition?

2021-06-27 Thread Amit Langote
ich we'd have to do anyway, because perhaps the executor proper (mainly InitPlan) should also see the shrunken version of the range table. Not to mention the complexity of getting the "init" pruning itself to run outside a full-blown executor context. Anyway, do you agre

Re: Decoding speculative insert with toast leaks memory

2021-06-17 Thread Amit Langote
Hi Dilip, On Thu, Jun 17, 2021 at 4:45 PM Dilip Kumar wrote: > On Thu, Jun 17, 2021 at 12:52 PM Amit Langote wrote: > > > Oh I missed that the problem report is for the PG13 branch. > > > > How about the attached patch then? > > > Looks good, Thanks for ch

Re: Skip partition tuple routing with constant partition key

2021-06-17 Thread Amit Langote
On Thu, Jun 17, 2021 at 4:18 PM Zhihong Yu wrote: > On Wed, Jun 16, 2021 at 10:37 PM Amit Langote wrote: >> > + if (part_index < 0) >> > + { >> > + bound_offset = >> > partition_range_datum_bsea

Re: Decoding speculative insert with toast leaks memory

2021-06-17 Thread Amit Langote
On Thu, Jun 17, 2021 at 3:42 PM Amit Kapila wrote: > On Thu, Jun 17, 2021 at 10:39 AM Amit Langote wrote: > > > > On Thu, Jun 17, 2021 at 12:56 PM Amit Kapila > > wrote: > > > On Wed, Jun 16, 2021 at 8:18 PM Tom Lane wrote: > > > > >

Re: Skip partition tuple routing with constant partition key

2021-06-16 Thread Amit Langote
Hi, Thanks for reading the patch. On Thu, Jun 17, 2021 at 1:46 PM Zhihong Yu wrote: > On Wed, Jun 16, 2021 at 9:29 PM Amit Langote wrote: >> Attached a slightly revised version of that patch, with a commit >> message this time. > > + int n_tup

Re: Decoding speculative insert with toast leaks memory

2021-06-16 Thread Amit Langote
t;pubactions.pubtruncate = false; entry->publish_as_relid = InvalidOid; entry->map = NULL; /* will be set by maybe_send_schema() if needed */ } Do we need the same statement at the end of the following block? /* Validate the entry */ if (!entry->replicate_valid) { -- Amit Langote EDB: http://www.enterprisedb.com

Re: Skip partition tuple routing with constant partition key

2021-06-16 Thread Amit Langote
On Wed, Jun 16, 2021 at 4:27 PM Amit Langote wrote: > On Mon, Jun 7, 2021 at 8:38 PM houzj.f...@fujitsu.com > wrote: > > The rough idea is to check the average batch number every 1000 rows. > > If the average batch num is greater than 1, then we enable the cache check, > &g

Re: Skip partition tuple routing with constant partition key

2021-06-16 Thread Amit Langote
serted may be more adaptive. Please find attached a revised version of your patch, where I tried to make it a bit easier to follow, hopefully. While doing so, I realized that caching the bound offset across queries makes little sense now, so I decided to keep the changes confined to execPartitio

Re: Multi-Column List Partitioning

2021-06-10 Thread Amit Langote
On Fri, Jun 11, 2021 at 12:37 PM Amit Langote wrote: > I will look at other parts of the patch next week hopefully. For > now, attached is a delta patch that applies on top of your v1, which > does: > > * Simplify partition_list_bsearch() and partition_lbound_datum

Re: Multi-Column List Partitioning

2021-06-10 Thread Amit Langote
for a and b. I will look at other parts of the patch next week hopefully. For now, attached is a delta patch that applies on top of your v1, which does: * Simplify partition_list_bsearch() and partition_lbound_datum_cmp() * Make qsort_partition_list_value_cmp simply call partition_lbound_datum_cmp() instead of having its own logic to compare input bounds * Move partition_lbound_datum_cmp() into partbounds.c as a static function (export seems unnecessary) * Add a comment for PartitionBoundInfo.isnulls and remove that for null_index -- Amit Langote EDB: http://www.enterprisedb.com v1_delta_amit.patch Description: Binary data

Re: Skip partition tuple routing with constant partition key

2021-06-04 Thread Amit Langote
On Fri, Jun 4, 2021 at 6:05 PM Amit Langote wrote: > On Fri, Jun 4, 2021 at 4:38 PM Amit Langote wrote: > > On Thu, Jun 3, 2021 at 8:48 PM Amit Langote wrote: > > > On Tue, Jun 1, 2021 at 5:43 PM houzj.f...@fujitsu.com > > > wrote: > > > > So, If we wan

Re: Skip partition tuple routing with constant partition key

2021-06-04 Thread Amit Langote
On Fri, Jun 4, 2021 at 4:38 PM Amit Langote wrote: > On Thu, Jun 3, 2021 at 8:48 PM Amit Langote wrote: > > On Tue, Jun 1, 2021 at 5:43 PM houzj.f...@fujitsu.com > > wrote: > > > So, If we want to share the cached partition between statements, we seems > > > ca

Re: Skip partition tuple routing with constant partition key

2021-06-04 Thread Amit Langote
On Thu, Jun 3, 2021 at 8:48 PM Amit Langote wrote: > On Tue, Jun 1, 2021 at 5:43 PM houzj.f...@fujitsu.com > wrote: > > So, If we want to share the cached partition between statements, we seems > > cannot > > use ExecPartitionCheck. Instead, I tried directly invoke th

Re: parent foreign tables and row marks

2021-06-03 Thread Amit Langote
On Thu, Jun 3, 2021 at 10:07 AM Amit Langote wrote: > On Thu, Jun 3, 2021 at 3:39 AM Tom Lane wrote: > > > > Amit Langote writes: > > > I noticed that 428b260f87e8 (v12) broke the cases where a parent > > > foreign table has row marks assigned. > > >

Re: Skip partition tuple routing with constant partition key

2021-06-03 Thread Amit Langote
Hou-san, On Tue, Jun 1, 2021 at 5:43 PM houzj.f...@fujitsu.com wrote: > From: Amit Langote > > > > Where are you thinking to cache the partidx list? Inside > > > > PartitionDesc or some executor struct? > > > > > > I was thinking cache the

Re: parent foreign tables and row marks

2021-06-02 Thread Amit Langote
On Thu, Jun 3, 2021 at 3:39 AM Tom Lane wrote: > > Amit Langote writes: > > I noticed that 428b260f87e8 (v12) broke the cases where a parent > > foreign table has row marks assigned. > > Indeed :-(. Fix pushed. I tweaked the comments and test case slightly. Thank yo

Re: join pushdown and issue with foreign update

2021-06-02 Thread Amit Langote
ltype = get_rel_type_id(rte->relid); + if (!OidIsValid(reltype)) + continue; + att->atttypid = reltype; That may be a valid concern. I wonder if it would make sense to also check varattno == 0 here somewhere for good measure. -- Amit Langote EDB: http://www.enterprisedb.com

Re: join pushdown and issue with foreign update

2021-06-02 Thread Amit Langote
relations, I don't think we have other cases that are affected at this point. I have a feeling that your patch will have fixed things enough that the same problem will not occur when we have join pushdown under UPDATE occurring in more cases. -- Amit Langote EDB: http://www.enterprisedb.com

Re: join pushdown and issue with foreign update

2021-06-01 Thread Amit Langote
tables, as we add "wholerow" with type of one child relation > and then > try to use it with another child (of different table type). Perhaps, we can get away with adding the wholerow Var with the target relation's reltype when the target foreign table is not a "child" relation, but the root target relation itself. Maybe like the attached? -- Amit Langote EDB: http://www.enterprisedb.com wholerow-rowid-var-vartype-fix.patch Description: Binary data

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-06-01 Thread Amit Langote
On Tue, Jun 1, 2021 at 6:56 PM Amit Kapila wrote: > On Mon, May 31, 2021 at 8:51 AM Amit Langote wrote: > > On Thu, May 27, 2021 at 3:36 PM Amit Kapila wrote: > > > Why do we need to move send_relation_and_attrs() call? I think it > > > doesn't matter much eith

parent foreign tables and row marks

2021-06-01 Thread Amit Langote
) to add a TID row marking junk column if needed. Attached a patch for that, which also adds the test cases. Actually, I had to make a separate version of the patch to apply to the v12 branch, because EXPLAIN outputs relation aliases a bit differently starting in v13, which is attached too. -- Amit Langot

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-30 Thread Amit Langote
On Thu, May 27, 2021 at 3:36 PM Amit Kapila wrote: > On Fri, May 21, 2021 at 1:12 PM Amit Langote wrote: > > > > Hmm, maybe get_rel_syn_entry() should explicitly set map to NULL when > > first initializing an entry. It's possible that without doing so, the > > ma

Re: Skip partition tuple routing with constant partition key

2021-05-27 Thread Amit Langote
Hou-san, On Thu, May 27, 2021 at 3:56 PM houzj.f...@fujitsu.com wrote: > From: Amit Langote > Sent: Thursday, May 27, 2021 1:54 PM > > On Thu, May 27, 2021 at 11:47 AM houzj.f...@fujitsu.com > > wrote: > > > About teaching relcache about caching the target parti

Re: Skip partition tuple routing with constant partition key

2021-05-26 Thread Amit Langote
What do you think ? Where are you thinking to cache the partidx list? Inside PartitionDesc or some executor struct? -- Amit Langote EDB: http://www.enterprisedb.com

Re: Skip partition tuple routing with constant partition key

2021-05-26 Thread Amit Langote
On Thu, May 27, 2021 at 1:55 PM Zhihong Yu wrote: > For CanUseSavedPartitionForTuple, nit: you can check > !dispatch->savedPartResultInfo at the beginning and return early. > This would save some indentation. Sure, see the attached. -- Amit Langote EDB: http://www.enterprisedb.c

Re: Skip partition tuple routing with constant partition key

2021-05-26 Thread Amit Langote
tuple"). Maybe we don't need this parent-slot-clearing anymore due to code restructuring over the last 3 years, but that will have to be a separate patch. I hope the attached updated patch makes it a bit more clear what's going on. I refactored more of the code in ExecFin

Re: Multi-Column List Partitioning

2021-05-25 Thread Amit Langote
> thoughts about the approaches and please share if you have any better > solution than the above 2. Approach 1 sounds better. It sounds like approach 1 might help us implement support for allowing NULLs in range partition bounds in the future, if at all. For now, it might be

Re: Skip partition tuple routing with constant partition key

2021-05-25 Thread Amit Langote
Hou-san, On Wed, May 26, 2021 at 10:05 AM houzj.f...@fujitsu.com wrote: > From: Amit Langote > Sent: Tuesday, May 25, 2021 10:06 PM > > Though again, I think we can do this without changing the relcache > > interface, > > such as RelationGetPartitionQual(). > >

Re: Skip partition tuple routing with constant partition key

2021-05-25 Thread Amit Langote
Hou-san, On Mon, May 24, 2021 at 10:15 PM houzj.f...@fujitsu.com wrote: > From: Amit Langote > Sent: Monday, May 24, 2021 4:27 PM > > On Mon, May 24, 2021 at 10:31 AM houzj.f...@fujitsu.com > > wrote: > > > Currently for a normal RANGE partition key it will f

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-05-24 Thread Amit Langote
AGS = -DRELCACHE_FORCE_RELEASE -DCATCACHE_FORCE_RELEASE -DCLOBBER_CACHE_ALWAYS -D_GNU_SOURCE -- Amit Langote EDB: http://www.enterprisedb.com

Re: Skip partition tuple routing with constant partition key

2021-05-24 Thread Amit Langote
Hou-san, On Mon, May 24, 2021 at 10:31 AM houzj.f...@fujitsu.com wrote: > From: Amit Langote > Sent: Thursday, May 20, 2021 8:23 PM > > This one seems bit tough. ExecPartitionCheck() uses the generic expression > > evaluation machinery like a black box, which means exe

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-23 Thread Amit Langote
On Mon, May 24, 2021 at 12:16 PM osumi.takami...@fujitsu.com wrote: > On Saturday, May 22, 2021 11:58 AM Amit Langote > wrote: > > On Sat, May 22, 2021 at 11:00 AM osumi.takami...@fujitsu.com > > wrote: > > > I've checked the core file of v3's failure co

Re: Subscription tests fail under CLOBBER_CACHE_ALWAYS

2021-05-22 Thread Amit Langote
ecause > a04daa97a had changed a lot of the APIs here. So I left the APIs > for the xxx_internal() functions alone. Otherwise the patch > pretty much works as-is in v13. That looks reasonable. Thanks. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Subscription tests fail under CLOBBER_CACHE_ALWAYS

2021-05-21 Thread Amit Langote
On Sat, May 22, 2021 at 6:01 AM Tom Lane wrote: > Amit Langote writes: > > IMHO, it would be better to keep the lowest-level > > apply_handle_XXX_internal() out of this, because presumably we're only > > cleaning up the mess in higher-level callers. Somewhat related,

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-21 Thread Amit Langote
On Sat, May 22, 2021 at 11:00 AM osumi.takami...@fujitsu.com wrote: > On Friday, May 21, 2021 9:45 PM I worte: > > On Friday, May 21, 2021 4:43 PM Amit Langote > > wrote: > > > On Fri, May 21, 2021 at 3:55 PM osumi.takami...@fujitsu.com > > > wrote: > &

Re: Multi-Column List Partitioning

2021-05-21 Thread Amit Langote
On Fri, May 21, 2021 at 1:02 PM Amit Langote wrote: > I will now take a look at the patch itself. Some quick observations: * I get a lot of instances of the following 2 warnings when compiling the patched code: Warning #1: partprune.c: In function ‘get_matching_list_bounds’: partprune.c:2

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-21 Thread Amit Langote
On Fri, May 21, 2021 at 3:55 PM osumi.takami...@fujitsu.com wrote: > On Thursday, May 20, 2021 9:59 PM Amit Langote > wrote: > > Here are updated/divided patches. > Thanks for your updates. > > But, I've detected segmentation faults caused by the patch, > which can

Re: Multi-Column List Partitioning

2021-05-20 Thread Amit Langote
o your example would look like this: CREATE TABLE t2_1 PARTITION OF t2 FOR VALUES IN ((1, 2), (1, 5), (2, 2), (2, 10)); IMO, it is not such a bad syntax from a user's PoV. It's not hard to understand from this syntax that the partition constraint is something like (a, b) = (1, 2) OR (a, b) = (1, 5) OR ..., where the = performs row-wise comparison. I will now take a look at the patch itself. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-20 Thread Amit Langote
On Thu, May 20, 2021 at 5:59 PM osumi.takami...@fujitsu.com wrote: > On Tuesday, May 18, 2021 3:30 PM Amit Langote wrote: > > While doing so, it occurred to me (maybe not for the first time) that we are > > *unnecessarily* doing send_relation_and_attrs() for a relation if t

Re: Skip partition tuple routing with constant partition key

2021-05-20 Thread Amit Langote
Hou-san, On Thu, May 20, 2021 at 7:35 PM houzj.f...@fujitsu.com wrote: > From: Amit Langote > Sent: Wednesday, May 19, 2021 9:17 PM > > I guess it would be nice if we could fit in a solution for the use case > > that houjz > > mentioned as a special case. BTW, houjz, c

Re: Skip partition tuple routing with constant partition key

2021-05-20 Thread Amit Langote
On Thu, May 20, 2021 at 9:20 AM tsunakawa.ta...@fujitsu.com wrote: > From: Amit Langote > > On Tue, May 18, 2021 at 11:11 AM houzj.f...@fujitsu.com > > wrote: > > > For some big data scenario, we sometimes transfer data from one table(only > > store not expire

Re: Skip partition tuple routing with constant partition key

2021-05-20 Thread Amit Langote
On Thu, May 20, 2021 at 9:31 AM David Rowley wrote: > On Thu, 20 May 2021 at 01:17, Amit Langote wrote: > > I gave a shot to implementing your idea and ended up with the attached > > PoC patch, which does pass make check-world. > > I only had a quick look at this. >

Re: Subscription tests fail under CLOBBER_CACHE_ALWAYS

2021-05-19 Thread Amit Langote
On Thu, May 20, 2021 at 5:23 AM Tom Lane wrote: > Amit Langote writes: > > IOW, the patch you posted earlier seems like the way to go. > > I really dislike that patch. I think it's doubling down on the messy, > unstructured coding patterns that got us into this situatio

Re: Skip partition tuple routing with constant partition key

2021-05-19 Thread Amit Langote
ate partition not an acceptable solution to address this particular use case? Yeah, I know we should avoid encouraging users to perform DML directly on partitions, but... -- Amit Langote EDB: http://www.enterprisedb.com

Re: Skip partition tuple routing with constant partition key

2021-05-19 Thread Amit Langote
On Tue, May 18, 2021 at 10:28 AM David Rowley wrote: > On Tue, 18 May 2021 at 01:31, Amit Langote wrote: > > Hmm, does this seem common enough for the added complexity to be worthwhile? > > I'd also like to know if there's some genuine use case for this. For > testi

Re: Subscription tests fail under CLOBBER_CACHE_ALWAYS

2021-05-18 Thread Amit Langote
l(), after triggers of the subsequent INSERT can't be processed, instead causing: ERROR: AfterTriggerSaveEvent() called outside of query IOW, the patch you posted earlier seems like the way to go. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Subscription tests fail under CLOBBER_CACHE_ALWAYS

2021-05-18 Thread Amit Langote
re right that this is not a problem of 14~. I can reproduce the > problem on 13 as well, and we have no coverage for tuple routing with > triggers on this branch, so this would never have been stressed in the > buildfarm. There is a good argument to be made here in cherry-picking > 2ecfeda3 to REL_13_STABLE. +1 -- Amit Langote EDB: http://www.enterprisedb.com

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-17 Thread Amit Langote
On Mon, May 17, 2021 at 9:45 PM osumi.takami...@fujitsu.com wrote: > On Monday, May 17, 2021 6:52 PM Amit Langote wrote: > > Both patches are attached. > The patch for PG13 can be applied to it cleanly and the RT succeeded. > > I have few really minor comments on your com

Re: Skip partition tuple routing with constant partition key

2021-05-17 Thread Amit Langote
rthwhile? For an example of what's previously been considered worthwhile for a project like this, see what 0d5f05cde0 did. The cases it addressed are common enough -- a file being loaded into a (time range-) partitioned table using COPY FROM tends to have lines belonging to the same pa

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-17 Thread Amit Langote
On Mon, May 17, 2021 at 6:13 PM Amit Kapila wrote: > On Fri, May 14, 2021 at 12:44 PM Amit Langote wrote: > > On Thu, May 13, 2021 at 7:43 PM Amit Kapila wrote: > > > > > Also, don't we need to free the > > > entire map as suggested by me? > > >

Re: making update/delete of inheritance trees scale better

2021-05-16 Thread Amit Langote
partitions to share + * In an inherited UPDATE/DELETE it's important for child tables to share While at it, I also noticed that the comment refers to the row_identity_vars, but it can be unclear which variable it is referring to. So fixed that too. -- Amit Langote EDB: http://www.enterprisedb.com RowIdentityVarInfo-comment.patch Description: Binary data

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-14 Thread Amit Langote
Takamichi-san, On Fri, May 14, 2021 at 11:19 AM osumi.takami...@fujitsu.com wrote: > On Thursday, May 13, 2021 7:43 PM Amit Kapila wrote: > > On Tue, Apr 20, 2021 at 8:36 AM Amit Langote > > wrote: > > > Back in: > > https://www.postgresql.org/message-id/CA%2BHi

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-14 Thread Amit Langote
On Thu, May 13, 2021 at 7:43 PM Amit Kapila wrote: > On Tue, Apr 20, 2021 at 8:36 AM Amit Langote wrote: > > On Sat, Apr 17, 2021 at 1:30 PM Amit Kapila wrote: > > > On Fri, Apr 16, 2021 at 11:24 PM Andres Freund > > > wrote:> > This made me take a brie

Re: PG 14 release notes, first draft

2021-05-13 Thread Amit Langote
On Thu, May 13, 2021 at 11:59 PM Bruce Momjian wrote: > On Thu, May 13, 2021 at 02:46:58PM +0900, Amit Langote wrote: > > How about writing the 2nd line instead as: > > > > Updates/deletes on partitioned tables can now use execution-time > > partition pruning. >

Re: Inherited UPDATE/DELETE vs async execution

2021-05-13 Thread Amit Langote
On Thu, May 13, 2021 at 8:10 PM Etsuro Fujita wrote: > On Thu, May 13, 2021 at 5:00 PM Etsuro Fujita wrote: > > On Thu, May 13, 2021 at 3:32 PM Amit Langote > > wrote: > > > On Wed, May 12, 2021 at 6:45 PM Etsuro Fujita > > > wrote: > > > >

Re: Inherited UPDATE/DELETE vs async execution

2021-05-12 Thread Amit Langote
On Wed, May 12, 2021 at 6:45 PM Etsuro Fujita wrote: > On Tue, May 11, 2021 at 9:53 PM Amit Langote wrote: > > Okay, so I take it that making these ForeignScan nodes (that only > > fetch the data) asynchronous doesn't interfere with update/delete > > subsequently being

Re: PG 14 release notes, first draft

2021-05-12 Thread Amit Langote
On Thu, May 13, 2021 at 2:39 AM Bruce Momjian wrote: > On Tue, May 11, 2021 at 11:57:10AM +0900, Amit Langote wrote: > > On Mon, May 10, 2021 at 11:40 PM Justin Pryzby wrote: > > > 86dc90056d Rework planning and execution of UPDATE and DELETE. > > > a1115fa078 Post

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-05-12 Thread Amit Langote
On Fri, May 7, 2021 at 2:13 AM Álvaro Herrera wrote: > On 2021-Apr-30, Amit Langote wrote: > > > The case I was looking at is when a partition detach appears as > > in-progress to a serializable transaction. > > Yeah, I was exceedingly sloppy on my reasoning about this,

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-12 Thread Amit Langote
Takamichi-san, On Tue, Apr 27, 2021 at 9:37 PM osumi.takami...@fujitsu.com wrote: > On Tuesday, April 20, 2021 12:07 PM Amit Langote > wrote: > > On Sat, Apr 17, 2021 at 1:30 PM Amit Kapila > > wrote: > > > On Fri, Apr 16, 2021 at 11:24 PM Andres Freund > >

Re: Feedback on table expansion hook (including patch)

2021-05-12 Thread Amit Langote
o give partitioned > tables a special access method, and then shove most of the planner > and executor's hard-wired partitioning logic into access method > callbacks. That would make it a lot more feasible for extensions > to implement custom partitioning-like behavior ... or so I guess. Interesting proposition... -- Amit Langote EDB: http://www.enterprisedb.com

Re: Query regarding RANGE Partitioning

2021-05-12 Thread Amit Langote
ies insert any values for column 'a' between '0' > to '100', > either it should go to default partition if exists. Otherwise it should fail > saying, no partition > found. I feel there should be some way to create partitions in these > scenarios. Well, you simply need to come up with bound values for r3 that don't overlap with existing partitions' ranges; the following will work for example: create table r3 partition of r for values from (0,100) to (100,0); -- Amit Langote EDB: http://www.enterprisedb.com

Re: PG 14 release notes, first draft

2021-05-11 Thread Amit Langote
odifyTable on top to do the actual update/delete -- and because Append/MergeAppend nodes carry run-time pruning info, update/delete now get that ability. -- Amit Langote EDB: http://www.enterprisedb.com

Re: PG 14 release notes, first draft

2021-05-11 Thread Amit Langote
On Wed, May 12, 2021 at 6:23 AM Bruce Momjian wrote: > On Tue, May 11, 2021 at 11:57:10AM +0900, Amit Langote wrote: > > On Mon, May 10, 2021 at 11:40 PM Justin Pryzby wrote: > > > Same as the last couple years, I checked for missing items in the release > > > notes,

Re: Inherited UPDATE/DELETE vs async execution

2021-05-11 Thread Amit Langote
Fujita-san, On Tue, May 11, 2021 at 5:56 PM Etsuro Fujita wrote: > On Mon, May 10, 2021 at 9:21 PM Amit Langote wrote: > > On Sat, May 8, 2021 at 1:21 AM Etsuro Fujita > > wrote: > > > I noticed this while working on the > > > EXPLAIN-ANALYZE-for-async-capable

Re: update/delete and execution-time partition pruning

2021-05-10 Thread Amit Langote
On Tue, May 11, 2021 at 12:57 PM David Rowley wrote: > > On Tue, 11 May 2021 at 15:46, David Rowley wrote: > > I'll take care of this. > > Pushed. Thank you. -- Amit Langote EDB: http://www.enterprisedb.com

update/delete and execution-time partition pruning

2021-05-10 Thread Amit Langote
to be changed in a future release of PostgreSQL. Now because we include the Append/MergeAppend node even if the query is update/delete, this note is no longer useful or valid. Attached patch removes that note. -- Amit Langote EDB: http://www.enterprisedb.com update-delete-exec

Re: PG 14 release notes, first draft

2021-05-10 Thread Amit Langote
://www.postgresql.org/message-id/CA%2BHiwqEcawatEaUh1uTbZMEZTJeLzbroRTz9_X9Z5CFjTWJkhw%40mail.gmail.com Thank you. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Inherited UPDATE/DELETE vs async execution

2021-05-10 Thread Amit Langote
. Attached is a patch for > that. Maybe I am missing something, though. I see that your patch is to disable asynchronous execution in ForeignScan nodes responsible for direct update/delete, but why not do the same for other ForeignScan nodes too? Or the other way around -- is it because fixing th

Re: Allow batched insert during cross-partition updates

2021-05-09 Thread Amit Langote
mstate->aux_fmstate; > > It might be better to write like " if (fmstate != NULL && > fmstate->aux_fmstate != NULL)". Sure, done. Actually, there's a if (fmstate) statement right below the code being added, which I fixed to match the style used by the new code. -- Amit Langote EDB: http://www.enterprisedb.com v6-0001-Allow-batching-of-inserts-during-cross-partition-.patch Description: Binary data

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-05-05 Thread Amit Langote
> https://www.postgresql.org/docs/devel/ddl-partitioning.html#DDL-PARTITIONING-USING-INHERITANCE That makes sense, thanks for noticing. How about the attached? -- Amit Langote EDB: http://www.enterprisedb.com ddl-partitioning-note-detach-concurrenltly.patch Description: Binary data

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-04-30 Thread Amit Langote
(Thanks for committing the fix.) On Thu, Apr 29, 2021 at 1:11 AM Álvaro Herrera wrote: > On Wed, Apr 28, 2021, at 10:21 AM, Amit Langote wrote: > I noticed that rd_partdesc_nodetached_xmin can sometimes end up with > value 0. While you seem to be already aware of that, because otherw

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-04-28 Thread Amit Langote
e readable to just duplicate this stanza in the blocks that assign to rd_partdesc_nodetached and rd_partdesc, respectively? That's not much code to duplicate and it'd be easier to see which context is for which partdesc. + TransactionId rd_partdesc_nodetached_xmin; /* xmin for the above */ Could you please expand this description a bit? -- Amit Langote EDB: http://www.enterprisedb.com

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-04-27 Thread Amit Langote
On Tue, Apr 27, 2021 at 4:34 PM Amit Langote wrote: > Thanks for the updated patch. I've been reading it, but I noticed a > bug in 8aba9322511f, which I thought you'd want to know to make a note > of when committing this one. > > So we decided in 8aba9322511f that it i

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-04-27 Thread Amit Langote
etached is no longer transient, so the problem doesn't exist. I will write more about the updated patch in a bit... -- Amit Langote EDB: http://www.enterprisedb.com

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-04-26 Thread Amit Langote
ecessarily sane with asserts off, since you'll get an arbitrary > detach-Xmin assigned to the partdesc, depending on catalog scan order. Maybe this is an ignorant question but is the plan to add an elog() in this code path or a check (and an ereport()) somewhere in ATExecDetachPartition() to p

Re: Table refer leak in logical replication

2021-04-26 Thread Amit Langote
On Mon, Apr 26, 2021 at 3:27 PM Michael Paquier wrote: > On Fri, Apr 23, 2021 at 09:38:01PM +0900, Amit Langote wrote: > > On Thu, Apr 22, 2021 at 1:45 PM Michael Paquier wrote: > >> On the other hand, the tests for partitions have much more value IMO, > >> but look

Re: Does rewriteTargetListIU still need to add UPDATE tlist entries?

2021-04-25 Thread Amit Langote
0.00..55.20 rows=2260 width=42) Output: ((base.a + 1) - 44), ROW((base.a + 1), (base.b + 2)), base.ctid (7 rows) I didn't however study closely why REPLACEVARS_CHANGE_VARNO does the correct thing, so am not sure if there might be cases that would be broken. > So, as far as I can tell, this is an oversight in 86dc90056 and we > ought to clean it up as attached. Thanks for noticing this and the patch. If you are confident that REPLACEVARS_CHANGE_VARNO covers all imaginable cases, I suppose it makes sense to apply it. -- Amit Langote EDB: http://www.enterprisedb.com

Re: Table refer leak in logical replication

2021-04-23 Thread Amit Langote
On Thu, Apr 22, 2021 at 1:45 PM Michael Paquier wrote: > On Wed, Apr 21, 2021 at 09:58:10PM +0900, Amit Langote wrote: > > Okay, done. > > So, I have been working on that today, and tried to apply the full set > before realizing when writing the commit message that this was

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-04-23 Thread Amit Langote
On Fri, Apr 23, 2021 at 4:26 AM Alvaro Herrera wrote: > On 2021-Apr-22, Amit Langote wrote: > > -* The reason for this check is that we want to avoid seeing the > > +* The reason for this hack is that we want to avoid seeing the > > * partition a

Re: problem with RETURNING and update row movement

2021-04-22 Thread Amit Langote
On Fri, Apr 23, 2021 at 12:49 AM Tom Lane wrote: > Amit Langote writes: > > I think we should also add slot != srcSlot to this condition. > > Good idea, should save useless comparisons of identical tupdescs. > Done. > > >> (I've not looked at porting this t

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-04-22 Thread Amit Langote
l find a way to write it more succinctly. BTW, I do feel a bit alarmed by the potential performance impact of this. If detached_exist of a cached partdesc is true, then RI queries invoked during a bulk DML operation would have to rebuild one for every tuple to be checked, right? I haven't tried an actual example yet though. -- Amit Langote EDB: http://www.enterprisedb.com

Re: problem with RETURNING and update row movement

2021-04-21 Thread Amit Langote
On Thu, Apr 22, 2021 at 9:37 AM Tom Lane wrote: > Amit Langote writes: > > FWIW, I think we should go ahead and apply the patches for the bug > > reported here. Anyone who tries to project an updated tuple's system > > columns using RETURNING are likely to face probl

Re: Table refer leak in logical replication

2021-04-21 Thread Amit Langote
On Wed, Apr 21, 2021 at 7:38 PM Michael Paquier wrote: > On Wed, Apr 21, 2021 at 04:21:52PM +0900, Amit Langote wrote: > > So I had started last night by adding some tests for this in > > 003_constraints.pl because there are already some replica BR trigger > > tests there. I

Re: Table refer leak in logical replication

2021-04-21 Thread Amit Langote
On Wed, Apr 21, 2021 at 11:13 AM Amit Langote wrote: > On Wed, Apr 21, 2021 at 9:31 AM Michael Paquier wrote: > > On Tue, Apr 20, 2021 at 06:20:03PM +0530, Amit Kapila wrote: > > > +1. I think it makes sense to add a test case especially because we > > > don't hav

<    1   2   3   4   5   6   7   8   9   10   >