Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-04-08 Thread Etsuro Fujita
On Fri, Apr 9, 2021 at 9:49 AM Justin Pryzby wrote: > I rebased this patch to resolve a trivial 1 line conflict from c5b7ba4e6. Thanks for rebasing! Actually, I've started reviewing this, but I couldn't finish my review. My apologies for not having much time on this. I'll continue to work on

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-04-08 Thread Etsuro Fujita
On Fri, Apr 9, 2021 at 9:49 AM Justin Pryzby wrote: > > I rebased this patch to resolve a trivial 1 line conflict from c5b7ba4e6. > > -- > Justin

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-04-08 Thread Zhihong Yu
On Thu, Apr 8, 2021 at 5:49 PM Justin Pryzby wrote: > I rebased this patch to resolve a trivial 1 line conflict from c5b7ba4e6. > > -- > Justin > Hi, In src/backend/commands/copyfrom.c : + if (resultRelInfo->ri_RelationDesc->rd_rel->relkind == RELKIND_FOREIGN_TABLE) There are a few steps of

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-04-08 Thread Justin Pryzby
I rebased this patch to resolve a trivial 1 line conflict from c5b7ba4e6. -- Justin >From 0987ca4f62fb8c9b43a3fe142d955d8a9cb6f36f Mon Sep 17 00:00:00 2001 From: Takayuki Tsunakawa Date: Tue, 9 Feb 2021 12:50:00 +0900 Subject: [PATCH] Fast COPY FROM into the foreign or sharded table. This

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > On Mon, Mar 22, 2021 at 08:18:56PM -0700, Zhihong Yu wrote: > > with data_dest_cb callback. It is used for send text representation of a > > tuple to a custom destination. > > > > send text -> sending text > > I would say "It is used to send the text representation ..." I

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread Justin Pryzby
On Mon, Mar 22, 2021 at 08:18:56PM -0700, Zhihong Yu wrote: > with data_dest_cb callback. It is used for send text representation of a > tuple to a custom destination. > > send text -> sending text I would say "It is used to send the text representation ..." > struct PgFdwModifyState

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread Zhihong Yu
Hi, In the description: with data_dest_cb callback. It is used for send text representation of a tuple to a custom destination. send text -> sending text struct PgFdwModifyState *aux_fmstate; /* foreign-insert state, if * created */ +

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Andrey Lepikhov > Macros _() at the postgresExecForeignCopy routine: > if (PQputCopyEnd(conn, OK ? NULL : _("canceled by server")) <= 0) > > uses gettext. Under linux it is compiled ok, because (as i understood) > uses standard implementation of gettext: > objdump -t

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread Andrey Lepikhov
On 5/3/21 21:54, tsunakawa.ta...@fujitsu.com wrote: I've managed to rebased it, although it took unexpectedly long. The patch is attached. It passes make check against core and postgres_fdw. I'll turn the CF status back to ready for committer shortly. Macros _() at the

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > Could you rebase again and send an updated patch ? > I could do it if you want. Rebased and attached. Fortunately, there was no rebase conflict this time. make check passed for PG core and postgres_fdw. Regards Takayuki Tsunakawa

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-05 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > I think this change to the regression tests is suspicous: > > -CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES > ($1, $2) > -COPY rem2, line 1: "-1 xyzzy" > +CONTEXT: COPY loc2, line 1: "-1 xyzzy" > +remote SQL command: COPY public.loc2(f1, f2)

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-04 Thread Justin Pryzby
I think this change to the regression tests is suspicous: -CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2) -COPY rem2, line 1: "-1 xyzzy" +CONTEXT: COPY loc2, line 1: "-1 xyzzy" +remote SQL command: COPY public.loc2(f1, f2) FROM STDIN +COPY rem2, line 2 I

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-04 Thread Ibrar Ahmed
On Thu, Mar 4, 2021 at 12:40 PM tsunakawa.ta...@fujitsu.com < tsunakawa.ta...@fujitsu.com> wrote: > From: Zhihong Yu > > This feature enables bulk COPY into foreign table in the case of > > multi inserts is possible > > > > 'is possible' -> 'if possible' > > > > FDWAPI was extended by next

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-03 Thread tsunakawa.ta...@fujitsu.com
From: Zhihong Yu > This feature enables bulk COPY into foreign table in the case of > multi inserts is possible > > 'is possible' -> 'if possible' > > FDWAPI was extended by next routines: > > next routines -> the following routines Thank you, fixed slightly differently. (I feel the need

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-03 Thread Zhihong Yu
Hi, This feature enables bulk COPY into foreign table in the case of multi inserts is possible 'is possible' -> 'if possible' FDWAPI was extended by next routines: next routines -> the following routines For postgresExecForeignCopy(): + if ((!OK && PQresultStatus(res) !=

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-03 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > Find attached some language fixes. Thanks a lot! (I wish there will be some tool like "pgEnglish" that corrects English in code comments and docs.) > |/* Do this to ensure we've pumped libpq back to idle state */ > > I don't know why you mean by "pumped"? I

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-03 Thread Justin Pryzby
Find attached some language fixes. |/* Do this to ensure we've pumped libpq back to idle state */ I don't know why you mean by "pumped"? The CopySendEndOfRow "case COPY_CALLBACK:" should have a "break;" This touches some of the same parts as my "bulk insert" patch:

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-16 Thread Andrey V. Lepikhov
On 2/15/21 1:31 PM, Amit Langote wrote: Tsunakawa-san, Andrey, +static void +postgresBeginForeignCopy(ModifyTableState *mtstate, + ResultRelInfo *resultRelInfo) +{ ... + if (resultRelInfo->ri_RangeTableIndex == 0) + { + ResultRelInfo *rootResultRelInfo =

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-15 Thread tsunakawa.ta...@fujitsu.com
From: Amit Langote > Think I have mentioned upthread that this looks better as: > > if (rootResultRelInfo->ri_usesMultiInsert) > leaf_part_rri->ri_usesMultiInsert = ExecMultiInsertAllowed(leaf_part_rri); > > This keeps the logic confined to ExecInitPartitionInfo() where it > belongs. No

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-15 Thread Amit Langote
Tsunakawa-san, Andrey, On Mon, Feb 15, 2021 at 1:54 PM tsunakawa.ta...@fujitsu.com wrote: > From: Justin Pryzby > > This is crashing during fdw check. > > http://cfbot.cputube.org/andrey-lepikhov.html > > > > Maybe it's related to this patch: > > |commit 6214e2b2280462cbc3aa1986e350e167651b3905

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-14 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > This is crashing during fdw check. > http://cfbot.cputube.org/andrey-lepikhov.html > > Maybe it's related to this patch: > |commit 6214e2b2280462cbc3aa1986e350e167651b3905 > |Fix permission checks on constraint violation errors on partitions. > |Security:

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-14 Thread Justin Pryzby
On Tue, Feb 09, 2021 at 04:35:03AM +, tsunakawa.ta...@fujitsu.com wrote: > Rebased to HEAD with the following modifications. It passes make check in > the top directory and contrib/postgres_fdw. > That said, with the reviews from some people and good performance results, I > think this can

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-11 Thread tsunakawa.ta...@fujitsu.com
From: Andrey V. Lepikhov > On 2/9/21 12:47 PM, tsunakawa.ta...@fujitsu.com wrote: > > As Tang-san and you showed, the basic part already demonstrated > impressive improvement. If there's no objection, I'd like to make this ready > for > committer in a few days. > Good. I've marked this as

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-09 Thread tsunakawa.ta...@fujitsu.com
From: Andrey V. Lepikhov > I tried to use 1E4 - 1E8 rows in a tuple buffer. But the results weren't > impressive. I guess that's because the 64 KB threshold came first. > We can use one more GUC instead of a precompiled constant. Yes, agreed. > > Why don't we focus on committing the basic

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-09 Thread Andrey V. Lepikhov
On 2/9/21 12:47 PM, tsunakawa.ta...@fujitsu.com wrote: From: Andrey V. Lepikhov I guess you used many hash partitions. Sadly, The current COPY implementation only accumulates either 1,000 rows or 64 KB of input data (very small!) before flushing all CopyMultiInsertBuffers. One

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-08 Thread tsunakawa.ta...@fujitsu.com
From: Andrey V. Lepikhov > On 2/9/21 9:35 AM, tsunakawa.ta...@fujitsu.com wrote: > > * Why is a separate FDW connection established for each COPY? To avoid > using the same FDW connection for multiple foreign table partitions in a > single > COPY run? > With separate connection you can init a

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-08 Thread Andrey V. Lepikhov
On 2/9/21 9:35 AM, tsunakawa.ta...@fujitsu.com wrote: From: tsunakawa.ta...@fujitsu.com From: Andrey Lepikhov Also, I might defer working on the extended part (v9 0003 and 0004) and further separate them in a different thread, if it seems to take longer. I reviewed them but haven't

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-08 Thread tsunakawa.ta...@fujitsu.com
From: tsunakawa.ta...@fujitsu.com > From: Andrey Lepikhov > > Of course, you can rebase it. > > Thank you. I might modify the basic part to incorporate my past proposal > about improving the layering or modularity related to ri_useMultiInsert. > (But I > may end up giving up due to

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-02 Thread tsunakawa.ta...@fujitsu.com
From: Andrey Lepikhov > Of course, you can rebase it. Thank you. I might modify the basic part to incorporate my past proposal about improving the layering or modularity related to ri_useMultiInsert. (But I may end up giving up due to lack of energy.) Also, I might defer working on the

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-02 Thread Andrey Lepikhov
On 2/2/21 11:57, tsunakawa.ta...@fujitsu.com wrote: Hello, Andrey-san, From: Tang, Haiying Sometimes before i suggested additional optimization [1] which can additionally speed up COPY by 2-4 times. Maybe you can perform the benchmark for this solution too? ... But the patch no longer

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-01 Thread tsunakawa.ta...@fujitsu.com
Hello, Andrey-san, From: Tang, Haiying > > Sometimes before i suggested additional optimization [1] which can > > additionally speed up COPY by 2-4 times. Maybe you can perform the > > benchmark for this solution too? ... > But the patch no longer applies, I tried to apply on

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-01-27 Thread Tang, Haiying
Hi Andrey, > Sometimes before i suggested additional optimization [1] which can > additionally speed up COPY by 2-4 times. Maybe you can perform the > benchmark for this solution too? Sorry for the late reply, I just have time to take this test now. But the patch no longer applies, I tried to

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-01-11 Thread Andrey V. Lepikhov
On 1/11/21 11:16 PM, Tomas Vondra wrote: Hi Andrey, Unfortunately, this no longer applies :-( I tried to apply this on top of c532d15ddd (from 2020/12/30) but even that has non-trivial conflicts. Can you send a rebased version? regards Applied on 044aa9e70e. -- regards, Andrey Lepikhov

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-01-11 Thread Andrey V. Lepikhov
On 1/11/21 4:59 PM, Tang, Haiying wrote: Hi Andrey, I had a general look at this extension feature, I think it's beneficial for some application scenarios of PostgreSQL. So I did 7 performance cases test on your patch(v13). The results are really good. As you can see below we can get 7-10

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-01-11 Thread Tomas Vondra
Hi Andrey, Unfortunately, this no longer applies :-( I tried to apply this on top of c532d15ddd (from 2020/12/30) but even that has non-trivial conflicts. Can you send a rebased version? regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-01-11 Thread Tang, Haiying
Hi Andrey, I had a general look at this extension feature, I think it's beneficial for some application scenarios of PostgreSQL. So I did 7 performance cases test on your patch(v13). The results are really good. As you can see below we can get 7-10 times improvement with this patch. PSA

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-12-29 Thread Andrey Lepikhov
On 29.12.2020 16:20, Hou, Zhijie wrote: see new version in attachment. I took a look into the patch, and have some comments. 1. + PG_FINALLY(); + { + copy_fmstate = NULL; /* Detect problems */ I don't quite understand this comment, does it means we want to detect

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-12-29 Thread Hou, Zhijie
Hi > see new version in attachment. I took a look into the patch, and have some comments. 1. + PG_FINALLY(); + { + copy_fmstate = NULL; /* Detect problems */ I don't quite understand this comment, does it means we want to detect something like Null reference ? 2. +

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-12-23 Thread Andrey V. Lepikhov
On 12/22/20 12:04 PM, Tang, Haiying wrote: Hi Andrey, There is an error report in your patch as follows. Please take a check. https://travis-ci.org/github/postgresql-cfbot/postgresql/jobs/750682857#L1519 copyfrom.c:374:21: error: ‘save_cur_lineno’ is used uninitialized in this function

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-12-21 Thread Tang, Haiying
Hi Andrey, There is an error report in your patch as follows. Please take a check. https://travis-ci.org/github/postgresql-cfbot/postgresql/jobs/750682857#L1519 >copyfrom.c:374:21: error: ‘save_cur_lineno’ is used uninitialized in this >function [-Werror=uninitialized] Regards, Tang

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-12-14 Thread Andrey V. Lepikhov
On 12/1/20 2:02 PM, Amit Langote wrote: On Tue, Dec 1, 2020 at 2:40 PM tsunakawa.ta...@fujitsu.com wrote: From: Amit Langote >> The code appears to require both BeginForeignCopy and EndForeignCopy, >> while the following documentation says they are optional. Which is >> correct? (I suppose

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-12-01 Thread Amit Langote
On Tue, Dec 1, 2020 at 2:40 PM tsunakawa.ta...@fujitsu.com wrote: > From: Amit Langote > > Andrey's original patch had the flag to, as I understand it, make the > > partitioning case work correctly. When inserting into a > > non-partitioned table, there's only one relation to care about. In >

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-30 Thread tsunakawa.ta...@fujitsu.com
From: Amit Langote > Andrey's original patch had the flag to, as I understand it, make the > partitioning case work correctly. When inserting into a > non-partitioned table, there's only one relation to care about. In > that case, CopyFrom() can use either the new COPY interface or the > INSERT

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-30 Thread Amit Langote
On Thu, Nov 26, 2020 at 11:42 AM tsunakawa.ta...@fujitsu.com wrote: > From: Amit Langote > > Anyway, one thing we could do is rename > > ExecRelationAllowsMultiInsert() to ExecSetRelationUsesMultiInsert(), > > that is, to make it actually set ri_usesMultiInsert and have places > > like

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-25 Thread tsunakawa.ta...@fujitsu.com
From: Amit Langote > Second is whether the interface for setting ri_usesMultiInsert > encourages situations where different modules could possibly engage in > conflicting behaviors. I can't think of a real-life example of that > with the current implementation, but maybe the interface provided

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-25 Thread Amit Langote
Hi, On Tue, Oct 20, 2020 at 11:31 AM tsunakawa.ta...@fujitsu.com wrote: > > > (2) > > > + Assert(rri->ri_usesMultiInsert == false); > > > > > > As the above assertion represents, I'm afraid the semantics of > > ExecRelationAllowsMultiInsert() and ResultRelInfo->ri_usesMultiInsert are > >

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-23 Thread Andrey Lepikhov
On 11/24/20 9:27 AM, tsunakawa.ta...@fujitsu.com wrote: Andrey-san, Fujita-san, From: Etsuro Fujita On Mon, Nov 23, 2020 at 5:39 PM Andrey Lepikhov wrote: On 11/23/20 7:49 AM, tsunakawa.ta...@fujitsu.com wrote: Could I or my colleague continue this patch in a few days? It looks it's

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-23 Thread tsunakawa.ta...@fujitsu.com
Andrey-san, Fujita-san, From: Etsuro Fujita > On Mon, Nov 23, 2020 at 5:39 PM Andrey Lepikhov > wrote: > > On 11/23/20 7:49 AM, tsunakawa.ta...@fujitsu.com wrote: > > > Could I or my colleague continue this patch in a few days? It looks it's > stalled over one month. > > > > I don't found any

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-23 Thread Etsuro Fujita
On Mon, Nov 23, 2020 at 5:39 PM Andrey Lepikhov wrote: > On 11/23/20 7:49 AM, tsunakawa.ta...@fujitsu.com wrote: > > Could I or my colleague continue this patch in a few days? It looks it's > > stalled over one month. > > I don't found any problems with this patch that needed to be corrected. >

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-23 Thread Andrey Lepikhov
On 11/23/20 7:49 AM, tsunakawa.ta...@fujitsu.com wrote: Hi Andrey-san, From: Tomas Vondra I needed to look at this patch while working on something related, and I found it got broken by 6973533650c a couple days ago. So here's a fixed version, to keep cfbot happy. I haven't done any

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-22 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey-san, From: Tomas Vondra > I needed to look at this patch while working on something related, and I > found it > got broken by 6973533650c a couple days ago. So here's a fixed version, to > keep > cfbot happy. I haven't done any serious review yet. Could I or my colleague continue

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-10 Thread Tomas Vondra
Hi, I needed to look at this patch while working on something related, and I found it got broken by 6973533650c a couple days ago. So here's a fixed version, to keep cfbot happy. I haven't done any serious review yet. regards -- Tomas Vondra EnterpriseDB: http://www.enterprisedb.com The

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-10-19 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey-san, Thanks for the revision. The patch looks good except for the following two items. (18) + if (target_resultRelInfo->ri_FdwRoutine != NULL) + { + if (target_resultRelInfo->ri_usesMultiInsert) + { +

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-10-19 Thread Andrey Lepikhov
19.10.2020 09:12, tsunakawa.ta...@fujitsu.com пишет: Hello Andrey-san, Thank you for challenging an interesting feature. Below are my review comments. (1) - /* for use by copy.c when performing multi-inserts */ + /* +* The following fields are currently only relevant

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-10-18 Thread tsunakawa.ta...@fujitsu.com
Hello Andrey-san, Thank you for challenging an interesting feature. Below are my review comments. (1) - /* for use by copy.c when performing multi-inserts */ + /* +* The following fields are currently only relevant to copy.c. +* +* True if okay to use

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-21 Thread Andrey Lepikhov
This patch currently looks very ready for use. And I'm taking a close look at the error reporting. Here we have difference in behavior of local and foreign table: regression test in postgres_fdw.sql: copy rem2 from stdin; -1 xyzzy \. reports error (1): = ERROR: new row

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-20 Thread Andrey Lepikhov
16.09.2020 12:10, Amit Langote пишет: On Thu, Sep 10, 2020 at 6:57 PM Andrey V. Lepikhov wrote: On 9/9/20 5:51 PM, Amit Langote wrote: Ok. I rewrited the patch 0001 with the Alexey suggestion. Thank you. Some mostly cosmetic suggestions on that: +bool +checkMultiInsertMode(const

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-16 Thread Amit Langote
On Thu, Sep 10, 2020 at 6:57 PM Andrey V. Lepikhov wrote: > On 9/9/20 5:51 PM, Amit Langote wrote: > > On Wed, Sep 9, 2020 at 6:42 PM Alexey Kondratov > > wrote: > >> And InitResultRelInfo() may set ri_usesMultiInsert to false by default, > >> since it's used only by COPY now. Then you won't

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-10 Thread Andrey V. Lepikhov
On 9/9/20 5:51 PM, Amit Langote wrote: On Wed, Sep 9, 2020 at 6:42 PM Alexey Kondratov wrote: On 2020-09-09 11:45, Andrey V. Lepikhov wrote: This does not seem very convenient and will lead to errors in the future. So, I agree with Amit. And InitResultRelInfo() may set ri_usesMultiInsert to

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-09 Thread Amit Langote
On Wed, Sep 9, 2020 at 6:42 PM Alexey Kondratov wrote: > On 2020-09-09 11:45, Andrey V. Lepikhov wrote: > > This does not seem very convenient and will lead to errors in the > > future. So, I agree with Amit. > > And InitResultRelInfo() may set ri_usesMultiInsert to false by default, > since it's

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-09 Thread Alexey Kondratov
On 2020-09-09 11:45, Andrey V. Lepikhov wrote: On 9/8/20 8:34 PM, Alexey Kondratov wrote: On 2020-09-08 17:00, Amit Langote wrote: wrote: On 2020-09-08 10:34, Amit Langote wrote: Another ambiguous part of the refactoring was in changing InitResultRelInfo() arguments: @@ -1278,6 +1280,7 @@

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-09 Thread Andrey V. Lepikhov
Version 8 split into two patches (in accordance with Amit suggestion). Also I eliminate naming inconsistency (thanks to Alexey). Based on master, f481d28232. -- regards, Andrey Lepikhov Postgres Professional >From 21b11f4ec0bec71bc7226014ef15c58dee9002da Mon Sep 17 00:00:00 2001 From: amitlan

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-09 Thread Andrey V. Lepikhov
On 9/8/20 8:34 PM, Alexey Kondratov wrote: On 2020-09-08 17:00, Amit Langote wrote: wrote: On 2020-09-08 10:34, Amit Langote wrote: Another ambiguous part of the refactoring was in changing InitResultRelInfo() arguments: @@ -1278,6 +1280,7 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo,   

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-08 Thread Alexey Kondratov
On 2020-09-08 17:00, Amit Langote wrote: Hi Alexey, On Tue, Sep 8, 2020 at 6:29 PM Alexey Kondratov wrote: On 2020-09-08 10:34, Amit Langote wrote: > Any thoughts on the taking out the refactoring changes out of the main > patch as I suggested? > +1 for splitting the patch. It was rather

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-08 Thread Amit Langote
Hi Alexey, On Tue, Sep 8, 2020 at 6:29 PM Alexey Kondratov wrote: > On 2020-09-08 10:34, Amit Langote wrote: > > Any thoughts on the taking out the refactoring changes out of the main > > patch as I suggested? > > > > +1 for splitting the patch. It was rather difficult for me to > distinguish

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-08 Thread Andrey V. Lepikhov
On 9/8/20 12:34 PM, Amit Langote wrote: Hi Andrey, On Mon, Sep 7, 2020 at 7:31 PM Andrey V. Lepikhov wrote: On 9/7/20 12:26 PM, Michael Paquier wrote: While on it, the CF bot is telling that the documentation of the patch fails to compile. This needs to be fixed. -- Michael v.7 (in

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-08 Thread Alexey Kondratov
Hi, I've started doing a review of v7 yesterday. On 2020-09-08 10:34, Amit Langote wrote: On Mon, Sep 7, 2020 at 7:31 PM Andrey V. Lepikhov wrote: > v.7 (in attachment) fixes this problem. I also accepted Amit's suggestion to rename all fdwapi routines such as ForeignCopyIn to

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-08 Thread Amit Langote
Hi Andrey, On Mon, Sep 7, 2020 at 7:31 PM Andrey V. Lepikhov wrote: > On 9/7/20 12:26 PM, Michael Paquier wrote: > > While on it, the CF bot is telling that the documentation of the patch > > fails to compile. This needs to be fixed. > > -- > > Michael > > > v.7 (in attachment) fixes this

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-07 Thread Andrey V. Lepikhov
On 9/7/20 12:26 PM, Michael Paquier wrote: On Mon, Aug 24, 2020 at 06:19:28PM +0900, Amit Langote wrote: On Mon, Aug 24, 2020 at 4:18 PM Amit Langote wrote: I would Oops, thought I'd continue writing, but hit send before actually doing that. Please ignore. I have some comments on v6,

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-07 Thread Michael Paquier
On Mon, Aug 24, 2020 at 06:19:28PM +0900, Amit Langote wrote: > On Mon, Aug 24, 2020 at 4:18 PM Amit Langote wrote: > > I would > > Oops, thought I'd continue writing, but hit send before actually doing > that. Please ignore. > > I have some comments on v6, which I will share later this week.

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-08-24 Thread Amit Langote
On Mon, Aug 24, 2020 at 4:18 PM Amit Langote wrote: > I would Oops, thought I'd continue writing, but hit send before actually doing that. Please ignore. I have some comments on v6, which I will share later this week. -- Amit Langote EnterpriseDB: http://www.enterprisedb.com

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-08-24 Thread Amit Langote
Hi Andrey, On Fri, Aug 21, 2020 at 9:19 PM Andrey Lepikhov wrote: > On 8/7/20 2:14 PM, Amit Langote wrote: > > I was playing around with v5 and I noticed an assertion failure which > > I concluded is due to improper setting of ri_usesBulkModify. You can > > reproduce it with these steps. > > >

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-08-21 Thread Andrey Lepikhov
On 8/7/20 2:14 PM, Amit Langote wrote: I was playing around with v5 and I noticed an assertion failure which I concluded is due to improper setting of ri_usesBulkModify. You can reproduce it with these steps. create extension postgres_fdw; create server lb foreign data wrapper postgres_fdw ;

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-08-07 Thread Amit Langote
On Mon, Aug 3, 2020 at 8:38 PM Amit Langote wrote: > On Wed, Jul 29, 2020 at 5:36 PM Andrey V. Lepikhov > wrote: > > > Will send more comments after reading the v5 patch. > > > > > Ok. I'll be waiting for the end of your review. > > Sorry about the late reply. > > If you'd like to send a new

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-08-03 Thread Amit Langote
Hi Andrey, On Wed, Jul 29, 2020 at 5:36 PM Andrey V. Lepikhov wrote: > > Will send more comments after reading the v5 patch. > > > Ok. I'll be waiting for the end of your review. Sorry about the late reply. If you'd like to send a new version for other reviewers, please feel free. I haven't

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-29 Thread Andrey V. Lepikhov
On 7/29/20 1:03 PM, Amit Langote wrote: Hi Andrey, Thanks for updating the patch. I will try to take a look later. On Wed, Jul 22, 2020 at 6:09 PM Andrey V. Lepikhov wrote: On 7/16/20 2:14 PM, Amit Langote wrote: * Why the "In" in these API names? + /* COPY a bulk of tuples into a

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-29 Thread Amit Langote
Hi Andrey, Thanks for updating the patch. I will try to take a look later. On Wed, Jul 22, 2020 at 6:09 PM Andrey V. Lepikhov wrote: > On 7/16/20 2:14 PM, Amit Langote wrote: > > * Why the "In" in these API names? > > > > + /* COPY a bulk of tuples into a foreign relation */ > > +

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-28 Thread Alexey Kondratov
On 2020-07-28 03:33, Andrey Lepikhov wrote: 27.07.2020 21:34, Alexey Kondratov пишет: Hi Andrey, On 2020-07-23 09:23, Andrey V. Lepikhov wrote: On 7/16/20 2:14 PM, Amit Langote wrote: Amit Langote EnterpriseDB: http://www.enterprisedb.com Version 5 of the patch. With changes caused by

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-27 Thread Andrey Lepikhov
27.07.2020 21:34, Alexey Kondratov пишет: Hi Andrey, On 2020-07-23 09:23, Andrey V. Lepikhov wrote: On 7/16/20 2:14 PM, Amit Langote wrote: Amit Langote EnterpriseDB: http://www.enterprisedb.com Version 5 of the patch. With changes caused by Amit's comments. Just got a segfault with

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-27 Thread Alexey Kondratov
Hi Andrey, On 2020-07-23 09:23, Andrey V. Lepikhov wrote: On 7/16/20 2:14 PM, Amit Langote wrote: Amit Langote EnterpriseDB: http://www.enterprisedb.com Version 5 of the patch. With changes caused by Amit's comments. Just got a segfault with your v5 patch by deleting from a foreign table.

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-23 Thread Andrey V. Lepikhov
On 7/16/20 2:14 PM, Amit Langote wrote: Amit Langote EnterpriseDB: http://www.enterprisedb.com Version 5 of the patch. With changes caused by Amit's comments. -- regards, Andrey Lepikhov Postgres Professional >From 24465d61d6f0ec6a45578d252bda1690ac045543 Mon Sep 17 00:00:00 2001 From:

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-22 Thread Andrey V. Lepikhov
On 7/16/20 2:14 PM, Amit Langote wrote: Hi Andrey, Thanks for this work. I have been reading through your patch and here's a what I understand it does and how: The patch aims to fix the restriction that COPYing into a foreign table can't use multi-insert buffer mechanism effectively. That's

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-16 Thread Amit Langote
Hi Andrey, Thanks for this work. I have been reading through your patch and here's a what I understand it does and how: The patch aims to fix the restriction that COPYing into a foreign table can't use multi-insert buffer mechanism effectively. That's because copy.c currently uses the

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-12 Thread Andrey Lepikhov
22.06.2020 17:11, Ashutosh Bapat пишет: On Wed, 17 Jun 2020 at 11:54, Andrey V. Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: On 6/15/20 10:26 AM, Ashutosh Bapat wrote: > Thanks Andrey for the patch. I am glad that the patch has taken care > of some corner cases already

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-02 Thread Andrey V. Lepikhov
On 6/22/20 5:11 PM, Ashutosh Bapat wrote: mailto:a.lepik...@postgrespro.ru>> wrote: It looks like we call BeginForeignInsert and EndForeignInsert even though actual copy is performed using BeginForeignCopy, ExecForeignCopy and EndForeignCopy. BeginForeignInsert constructs the INSERT query

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-06-22 Thread Ashutosh Bapat
On Wed, 17 Jun 2020 at 11:54, Andrey V. Lepikhov wrote: > On 6/15/20 10:26 AM, Ashutosh Bapat wrote: > > Thanks Andrey for the patch. I am glad that the patch has taken care > > of some corner cases already but there exist still more. > > > > COPY command constructed doesn't take care of dropped

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-06-22 Thread Andrey Lepikhov
19.06.2020 19:58, Etsuro Fujita пишет: On Tue, Jun 2, 2020 at 2:51 PM Andrey Lepikhov wrote: Hiding the COPY code under the buffers management machinery allows us to generalize buffers machinery, execute one COPY operation on each buffer and simplify error handling. I'm not sure that it's

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-06-19 Thread Etsuro Fujita
On Tue, Jun 2, 2020 at 2:51 PM Andrey Lepikhov wrote: > 02.06.2020 05:02, Etsuro Fujita пишет: > > I think I also thought something similar to this before [1]. Will take a > > look. I'm still reviewing the patch, but let me comment on it. > > [1] > >

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-06-17 Thread Andrey V. Lepikhov
On 6/15/20 10:26 AM, Ashutosh Bapat wrote: Thanks Andrey for the patch. I am glad that the patch has taken care of some corner cases already but there exist still more. COPY command constructed doesn't take care of dropped columns. There is code in deparseAnalyzeSql which constructs list of

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-06-14 Thread Ashutosh Bapat
Thanks Andrey for the patch. I am glad that the patch has taken care of some corner cases already but there exist still more. COPY command constructed doesn't take care of dropped columns. There is code in deparseAnalyzeSql which constructs list of columns for a given foreign relation. 0002 patch

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-06-01 Thread Andrey Lepikhov
Thank you for the answer, 02.06.2020 05:02, Etsuro Fujita пишет: I think I also thought something similar to this before [1]. Will take a look. [1] https://www.postgresql.org/message-id/23990375-45a6-5823-b0aa-a6a7a6a957f0%40lab.ntt.co.jp I have looked into the thread. My first version

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-06-01 Thread Etsuro Fujita
Hi Andrey, On Mon, Jun 1, 2020 at 6:29 PM Andrey Lepikhov wrote: > Currently i see, COPY FROM insertion into the partitioned table with > foreign partitions is not optimal: even if table constraints allows can > do multi insert copy, we will flush the buffers and prepare new INSERT > query for