Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2023-04-06 Thread Etsuro Fujita
On Tue, Apr 4, 2023 at 7:28 PM Etsuro Fujita wrote: > The parallel-abort patch received a review from David, and I addressed > his comments. Also, he tested with the patch, and showed that it > reduces time taken to abort remote transactions. So, if there are no > objections, I will commit the

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2023-04-04 Thread Etsuro Fujita
On Wed, Jan 18, 2023 at 8:06 PM Etsuro Fujita wrote: > I rebased the patch. Attached is an updated patch. The parallel-abort patch received a review from David, and I addressed his comments. Also, he tested with the patch, and showed that it reduces time taken to abort remote transactions.

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2023-01-18 Thread Etsuro Fujita
Hi Vignesh, On Wed, Jan 4, 2023 at 9:19 PM vignesh C wrote: > On Tue, 1 Nov 2022 at 15:54, Etsuro Fujita wrote: > > Attached is a rebased version of the patch. > > The patch does not apply on top of HEAD as in [1], please post a rebased > patch: I rebased the patch. Attached is an updated

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2023-01-04 Thread vignesh C
On Tue, 1 Nov 2022 at 15:54, Etsuro Fujita wrote: > > Hi David, > > On Sat, Oct 1, 2022 at 5:54 AM David Zhang wrote: > > After rebase the file `postgres_fdw.out` and applied to master branch, > > make and make check are all ok for postgres_fdw. > > Thanks for testing! Attached is a rebased

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-11-01 Thread Etsuro Fujita
Hi David, On Sat, Oct 1, 2022 at 5:54 AM David Zhang wrote: > After rebase the file `postgres_fdw.out` and applied to master branch, > make and make check are all ok for postgres_fdw. Thanks for testing! Attached is a rebased version of the patch. Best regards, Etsuro Fujita

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-09-30 Thread David Zhang
Hi Etsuro, The patch needs rebase due to commits 4036bcbbb, 8c8d307f8 and 82699edbf, so I updated the patch. Attached is a new version, in which I also tweaked comments a little bit. After rebase the file `postgres_fdw.out` and applied to master branch, make and make check are all ok for

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-07-07 Thread Etsuro Fujita
Hi Jacob, On Fri, Jul 1, 2022 at 3:50 AM Jacob Champion wrote: > On 5/12/22 01:46, Etsuro Fujita wrote: > > On Wed, May 11, 2022 at 7:39 PM Etsuro Fujita > > wrote: > >> I’m planning to commit this as a follow-up patch for commit 04e706d42. > > > > Done. > > FYI, I think cfbot is confused

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-06-30 Thread Jacob Champion
On 5/12/22 01:46, Etsuro Fujita wrote: > On Wed, May 11, 2022 at 7:39 PM Etsuro Fujita wrote: >> I’m planning to commit this as a follow-up patch for commit 04e706d42. > > Done. FYI, I think cfbot is confused about the patch under review here. (When I first opened the thread I thought the patch

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-05-12 Thread Etsuro Fujita
On Wed, May 11, 2022 at 7:39 PM Etsuro Fujita wrote: > On Fri, May 6, 2022 at 7:08 PM Etsuro Fujita wrote: > > > > On Wed, Apr 20, 2022 at 4:55 AM David Zhang > > > > wrote: > > > >> + * remote server in parallel at (sub)transaction end. > > > I noticed that the comment failed to mention

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-05-11 Thread Etsuro Fujita
On Fri, May 6, 2022 at 7:08 PM Etsuro Fujita wrote: > > > On Wed, Apr 20, 2022 at 4:55 AM David Zhang wrote: > > >> + * remote server in parallel at (sub)transaction end. > I noticed that the comment failed to mention that the > parallel_commit option is disabled by default. Also, I

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-05-06 Thread Etsuro Fujita
On Thu, May 5, 2022 at 6:39 AM David Zhang wrote: > On 2022-05-02 1:25 a.m., Etsuro Fujita wrote: > > On Wed, Apr 20, 2022 at 4:55 AM David Zhang wrote: > >> I tried to apply the patch to master and plan to run some tests, but got > >> below errors due to other commits. > > I rebased the patch

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-05-04 Thread David Zhang
Thanks a lot for the patch update. On 2022-05-02 1:25 a.m., Etsuro Fujita wrote: Hi, On Wed, Apr 20, 2022 at 4:55 AM David Zhang wrote: I tried to apply the patch to master and plan to run some tests, but got below errors due to other commits. I rebased the patch against HEAD. Attached is

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-05-02 Thread Etsuro Fujita
Hi, On Wed, Apr 20, 2022 at 4:55 AM David Zhang wrote: > I tried to apply the patch to master and plan to run some tests, but got > below errors due to other commits. I rebased the patch against HEAD. Attached is an updated patch. > + * remote server in parallel at (sub)transaction end. >

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-04-19 Thread David Zhang
I tried to apply the patch to master and plan to run some tests, but got below errors due to other commits. $ git apply --check v7-0003-postgres-fdw-Add-support-for-parallel-abort.patch error: patch failed: doc/src/sgml/postgres-fdw.sgml:479 error: doc/src/sgml/postgres-fdw.sgml: patch does

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-03-25 Thread Etsuro Fujita
On Thu, Mar 24, 2022 at 1:34 PM Etsuro Fujita wrote: > Attached is a new patch set. The new version of 0002 is just a > cleanup patch (see the commit message in 0002), and I think it's > committable, so I'm planning to commit it, if no objections. Done. Attached is the 0003 patch, which is the

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-03-23 Thread Etsuro Fujita
On Sat, Mar 5, 2022 at 7:32 PM Etsuro Fujita wrote: > Attached is an updated version. In the 0002 patch I introduced a macro for building an abort command in preparation for the parallel abort patch (0003), but I moved it to 0003. Attached is a new patch set. The new version of 0002 is just a

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-03-13 Thread Etsuro Fujita
On Sat, Mar 12, 2022 at 10:02 AM David Zhang wrote: > Applied patches v6-0002 and v6-0003 to master branch, and the `make check` > test is ok. > > Here is my test result in 10 times average on 3 virtual machines: > before the patches: > > abort.1 = 2.5473 ms > > abort.2 = 4.1572 ms > > after the

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-03-11 Thread David Zhang
Applied patches v6-0002 and v6-0003 to master branch, and the `make check` test is ok. Here is my test result in 10 times average on 3 virtual machines: before the patches: abort.1 = 2.5473 ms abort.2 = 4.1572 ms after the patches with OPTIONS (ADD parallel_abort 'true'): abort.1 =

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-03-05 Thread Etsuro Fujita
On Mon, Feb 28, 2022 at 6:53 PM Etsuro Fujita wrote: > Here is an updated version. I added to the 0003 patch a macro for > defining the milliseconds to wait, as proposed by David upthread. I modified the 0003 patch further: 1) I added to pgfdw_cancel_query_end/pgfdw_exec_cleanup_query_end the

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-28 Thread Etsuro Fujita
On Thu, Feb 24, 2022 at 2:49 PM Etsuro Fujita wrote: > I think the 0003 patch needs rebase. > I'll update the patch. Here is an updated version. I added to the 0003 patch a macro for defining the milliseconds to wait, as proposed by David upthread. Best regards, Etsuro Fujita

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-23 Thread Etsuro Fujita
On Wed, Feb 23, 2022 at 3:30 PM Etsuro Fujita wrote: > Attached is an updated patch. > Barring objections, I’ll commit > the patch. I have committed the patch. I think the 0003 patch needs rebase. I'll update the patch. Best regards, Etsuro Fujita

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-22 Thread Etsuro Fujita
On Tue, Feb 22, 2022 at 1:03 AM Fujii Masao wrote: > On 2022/02/21 14:45, Etsuro Fujita wrote: > > On Fri, Feb 18, 2022 at 1:46 AM Fujii Masao > > wrote: > >> I reviewed 0001 patch. It looks good to me except the following minor > >> things. If these are addressed, I think that the 001 patch

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-21 Thread Fujii Masao
On 2022/02/21 14:45, Etsuro Fujita wrote: On Fri, Feb 18, 2022 at 1:46 AM Fujii Masao wrote: I reviewed 0001 patch. It looks good to me except the following minor things. If these are addressed, I think that the 001 patch can be marked as ready for committer. OK +* Also

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-20 Thread Etsuro Fujita
On Sat, Feb 19, 2022 at 6:55 AM David Zhang wrote: > Tried to apply the patches to master branch, no warning found and > regression test passed. Thanks for testing! > Now, we have many places (5) calling the same function with a constant > number 3. Is this a good time to consider redefine

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-20 Thread Etsuro Fujita
On Fri, Feb 18, 2022 at 1:46 AM Fujii Masao wrote: > I reviewed 0001 patch. It looks good to me except the following minor things. > If these are addressed, I think that the 001 patch can be marked as ready for > committer. OK > +* Also determine to commit (sub)transactions opened on

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-18 Thread David Zhang
Thanks a lot for updating the patch. Tried to apply the patches to master branch, no warning found and regression test passed. Now, we have many places (5) calling the same function with a constant number 3. Is this a good time to consider redefine this number a macro somewhere? Thank

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-17 Thread Fujii Masao
On 2022/02/11 21:59, Etsuro Fujita wrote: I tweaked comments/docs a little bit as well. Thanks for updating the patches! I reviewed 0001 patch. It looks good to me except the following minor things. If these are addressed, I think that the 001 patch can be marked as ready for committer.

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-11 Thread Etsuro Fujita
On Tue, Feb 8, 2022 at 3:49 AM Fujii Masao wrote: > Here are the review comments for 0001 patch. > > I got the following compiler warning. > > [16:58:07.120] connection.c: In function ‘pgfdw_finish_pre_commit_cleanup’: > [16:58:07.120] connection.c:1726:4: error: ISO C90 forbids mixed

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-07 Thread Fujii Masao
On 2022/02/07 14:35, Etsuro Fujita wrote: 0001 patch failed to be applied. Could you rebase the patch? Done. Attached is an updated version of the patch set. Thanks for updating the patch! Here are the review comments for 0001 patch. I got the following compiler warning. [16:58:07.120]

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-06 Thread Etsuro Fujita
Hi Julien, On Sun, Jan 16, 2022 at 1:07 PM Julien Rouhaud wrote: > On Thu, Jan 06, 2022 at 05:29:23PM +0900, Etsuro Fujita wrote: > > Done. Attached is a new version. > I also see that Fuji-san raised some questions, so for now I will simply > change > the patch status to Waiting on Author.

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-02-06 Thread Etsuro Fujita
On Thu, Jan 13, 2022 at 11:54 AM Fujii Masao wrote: > At first I'm reading the 0001 patch. Here are the comments for the patch. Thanks for reviewing! > 0001 patch failed to be applied. Could you rebase the patch? Done. Attached is an updated version of the patch set. > +

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-01-15 Thread Julien Rouhaud
Hi, On Thu, Jan 06, 2022 at 05:29:23PM +0900, Etsuro Fujita wrote: > > Done. Attached is a new version. The patchset doesn't apply anymore: http://cfbot.cputube.org/patch_36_3392.log === Applying patches on top of PostgreSQL commit ID 43c2175121c829c8591fc5117b725f1f22bfb670 === === applying

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-01-12 Thread Fujii Masao
On 2022/01/06 17:29, Etsuro Fujita wrote: On Fri, Dec 3, 2021 at 6:07 PM Fujii Masao wrote: On 2021/11/16 18:55, Etsuro Fujita wrote: I changed my mind; I’ll update the patch to ignore the error as before, because 1) as far as I know, there are no reports from the field concerning that we

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-01-06 Thread Etsuro Fujita
On Fri, Dec 3, 2021 at 6:07 PM Fujii Masao wrote: > On 2021/11/16 18:55, Etsuro Fujita wrote: > > I changed my mind; I’ll update the patch to ignore the error as > > before, because 1) as far as I know, there are no reports from the > > field concerning that we ignore all kinds of errors in

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-12-03 Thread Fujii Masao
On 2021/11/16 18:55, Etsuro Fujita wrote: I changed my mind; I’ll update the patch to ignore the error as before, because 1) as far as I know, there are no reports from the field concerning that we ignore all kinds of errors in cleaning up the prepared statements, so maybe we don’t need to

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-18 Thread Etsuro Fujita
On Thu, Nov 18, 2021 at 1:09 PM Fujii Masao wrote: > On 2021/11/16 18:55, Etsuro Fujita wrote: > > Sorry, my explanation was not enough, but I don’t think this is always > > true. Let me explain using an example: > > > > create server loopback foreign data wrapper postgres_fdw options > >

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-17 Thread Fujii Masao
On 2021/11/16 18:55, Etsuro Fujita wrote: Sorry, my explanation was not enough, but I don’t think this is always true. Let me explain using an example: create server loopback foreign data wrapper postgres_fdw options (dbname 'postgres', parallel_commit 'true'); create user mapping for

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-16 Thread Etsuro Fujita
Kuroda-san, On Thu, Nov 11, 2021 at 11:27 AM kuroda.hay...@fujitsu.com wrote: > I love your proposal because it will remove a bottleneck > for PostgreSQL build-in sharding. > > I read your patch briefly and I think basically it's good. Great! Thanks for reviewing! > Currently I have only one

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-16 Thread Etsuro Fujita
On Mon, Nov 8, 2021 at 1:13 PM Fujii Masao wrote: > On 2021/11/07 18:06, Etsuro Fujita wrote: > > On Mon, Nov 1, 2021 at 3:22 PM Fujii Masao > > wrote: > >> Could you tell me why the parameter is necessary? > >> Can't we always enable the feature? > > I think it might be OK to enable it by > >

RE: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-10 Thread kuroda.hay...@fujitsu.com
Dear Fujita-san, I love your proposal because it will remove a bottleneck for PostgreSQL build-in sharding. I read your patch briefly and I think basically it's good. Currently I have only one comment. In your patch, postgres_fdw sends a COMMIT command to all entries in the hash table and

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-07 Thread Fujii Masao
On 2021/11/07 18:06, Etsuro Fujita wrote: On Mon, Nov 1, 2021 at 3:22 PM Fujii Masao wrote: On 2021/10/31 18:05, Etsuro Fujita wrote: The patch is pretty simple: if a server option added by the patch “parallel_commit” is enabled, Could you tell me why the parameter is necessary? Can't we

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-07 Thread Etsuro Fujita
On Tue, Nov 2, 2021 at 7:47 AM David Zhang wrote: > Followed your instructions, I performed some basic tests to compare the > performance between before and after. In my testing environment (two > foreign servers on the same local machine), the performance varies, > sometimes the time spent on

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-07 Thread Etsuro Fujita
On Mon, Nov 1, 2021 at 3:22 PM Fujii Masao wrote: > On 2021/10/31 18:05, Etsuro Fujita wrote: > > The patch is pretty simple: if a server option added > > by the patch “parallel_commit” is enabled, > > Could you tell me why the parameter is necessary? > Can't we always enable the feature? I

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-01 Thread David Zhang
I evaluated the effectiveness of the patch using a simple multi-statement transaction: BEGIN; SAVEPOINT s; INSERT INTO ft1 VALUES (10, 10); INSERT INTO ft2 VALUES (20, 20); RELEASE SAVEPOINT s; COMMIT; where ft1 and ft2 are foreign tables created on different foreign servers hosted on different

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-11-01 Thread Fujii Masao
On 2021/10/31 18:05, Etsuro Fujita wrote: Hi, As I said before [1], I’m working on $SUBJECT. Attached is a WIP patch for that. Thanks for the patch! The patch is pretty simple: if a server option added by the patch “parallel_commit” is enabled, Could you tell me why the parameter is

postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2021-10-31 Thread Etsuro Fujita
Hi, As I said before [1], I’m working on $SUBJECT. Attached is a WIP patch for that. The patch is pretty simple: if a server option added by the patch “parallel_commit” is enabled, 1) asynchronously send COMMIT TRANSACTION (RELEASE SAVEPOINT) to all remote servers involved in a local