Re: Transactions involving multiple postgres foreign servers, take 2

Fujii-san, On Thu, Oct 7, 2021 at 11:37 PM Fujii Masao wrote: > On 2021/10/07 19:47, Etsuro Fujita wrote: > > On Thu, Oct 7, 2021 at 1:29 PM k.jami...@fujitsu.com > > wrote: > >> and prove that commit performance can be improved, > >> e.g. if we can commit not in serial but also possible in

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/10/07 19:47, Etsuro Fujita wrote: Hi, On Thu, Oct 7, 2021 at 1:29 PM k.jami...@fujitsu.com wrote: That said, if we're going to initially support it on postgres_fdw, which is simpler than the latest patches, we need to ensure that abnormalities and errors are properly handled

Re: Transactions involving multiple postgres foreign servers, take 2

Hi, On Thu, Oct 7, 2021 at 1:29 PM k.jami...@fujitsu.com wrote: > That said, if we're going to initially support it on postgres_fdw, which is > simpler > than the latest patches, we need to ensure that abnormalities and errors > are properly handled and prove that commit performance can be

RE: Transactions involving multiple postgres foreign servers, take 2

Hi Fujii-san and Sawada-san, Thank you very much for your replies. > >> I noticed that this thread and its set of patches have been marked with > "Returned with Feedback" by yourself. > >> I find the feature (atomic commit for foreign transactions) very > >> useful and it will pave the road for

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/10/05 10:38, Masahiko Sawada wrote: Hi, On Tue, Oct 5, 2021 at 9:56 AM k.jami...@fujitsu.com wrote: Hi Sawada-san, I noticed that this thread and its set of patches have been marked with "Returned with Feedback" by yourself. I find the feature (atomic commit for foreign

Re: Transactions involving multiple postgres foreign servers, take 2

Hi, On Tue, Oct 5, 2021 at 9:56 AM k.jami...@fujitsu.com wrote: > > Hi Sawada-san, > > I noticed that this thread and its set of patches have been marked with > "Returned with Feedback" by yourself. > I find the feature (atomic commit for foreign transactions) very useful > and it will pave the

RE: Transactions involving multiple postgres foreign servers, take 2

Hi Sawada-san, I noticed that this thread and its set of patches have been marked with "Returned with Feedback" by yourself. I find the feature (atomic commit for foreign transactions) very useful and it will pave the road for having a distributed transaction management in Postgres. Although we

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/07/09 22:44, Masahiko Sawada wrote: On Fri, Jul 9, 2021 at 3:26 PM Fujii Masao wrote: As far as I read the code, keep using old API for foreign subtransaction doesn't cause any actual bug. But it's just strange and half-baked to manage top and sub transaction in the differenet layer

RE: Transactions involving multiple postgres foreign servers, take 2

Hi Sawada-san, Thank you for your reply. > BTW did you test on the local? That is, the foreign servers are > located on the same machine? Yes, I tested on the local since I cannot prepare the good network now. > I guess it would be better to start a new thread for this improvement. Thank

Re: Transactions involving multiple postgres foreign servers, take 2

On Tue, Jul 13, 2021 at 1:14 PM r.takahash...@fujitsu.com wrote: > > Hi Sawada-san, > > > Thank you for your reply. > > > Not sure but it might be possible to keep holding an xlogreader for > > reading PREPARE WAL records even after the transaction commit. But I > > wonder how much open() for wal

RE: Transactions involving multiple postgres foreign servers, take 2

Hi, > Wouldn't it be better to explicitly initialize the pointer with NULL? Thank you for your advice. You are correct. Anyway, I fixed it and re-run the performance test, it of course does not affect tps. Regards, Ryohei Takahashi

Re: Transactions involving multiple postgres foreign servers, take 2

Em ter., 13 de jul. de 2021 às 01:14, r.takahash...@fujitsu.com < r.takahash...@fujitsu.com> escreveu: > Hi Sawada-san, > > > Thank you for your reply. > > > Not sure but it might be possible to keep holding an xlogreader for > > reading PREPARE WAL records even after the transaction commit. But

RE: Transactions involving multiple postgres foreign servers, take 2

Hi Sawada-san, Thank you for your reply. > Not sure but it might be possible to keep holding an xlogreader for > reading PREPARE WAL records even after the transaction commit. But I > wonder how much open() for wal segment file accounts for the total > execution time of 2PC. 2PC requires 2

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Jul 9, 2021 at 3:26 PM Fujii Masao wrote: > > > > On 2021/06/30 10:05, Masahiko Sawada wrote: > > I've attached the new version patch that incorporates the comments > > from Fujii-san and Ikeda-san I got so far. > > Thanks for updating the patches! > > I'm now reading 0001 and 0002

Re: Transactions involving multiple postgres foreign servers, take 2

Sorry for the late reply. On Tue, Jul 6, 2021 at 3:15 PM r.takahash...@fujitsu.com wrote: > > Hi, > > > I'm interested in this patch and I also run the same test with Ikeda-san's > fxact_update.pgbench. Thank you for testing! > In my environment (poor spec VM), the result is following. > > *

Re: Transactions involving multiple postgres foreign servers, take 2

Sorry for the late reply. On Mon, Jul 5, 2021 at 3:29 PM Masahiro Ikeda wrote: > > > > On 2021/06/30 10:05, Masahiko Sawada wrote: > > On Fri, Jun 25, 2021 at 9:53 AM Masahiro Ikeda > > wrote: > >> > >> Hi Jamison-san, sawada-san, > >> > >> Thanks for testing! > >> > >> FWIF, I tested using

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/06/30 10:05, Masahiko Sawada wrote: I've attached the new version patch that incorporates the comments from Fujii-san and Ikeda-san I got so far. Thanks for updating the patches! I'm now reading 0001 and 0002 patches and wondering if we can commit them at first because they just

RE: Transactions involving multiple postgres foreign servers, take 2

On Wed, June 30, 2021 10:06 (GMT+9), Masahiko Sawada wrote: > I've attached the new version patch that incorporates the comments from > Fujii-san and Ikeda-san I got so far. We launch a resolver process per foreign > server, committing prepared foreign transactions on foreign servers in >

RE: Transactions involving multiple postgres foreign servers, take 2

Hi, I'm interested in this patch and I also run the same test with Ikeda-san's fxact_update.pgbench. In my environment (poor spec VM), the result is following. * foreign_twophase_commit = disabled 363tps * foreign_twophase_commit = required (It is necessary to set -R ${RATE} as Ikeda-san

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/06/30 10:05, Masahiko Sawada wrote: > On Fri, Jun 25, 2021 at 9:53 AM Masahiro Ikeda > wrote: >> >> Hi Jamison-san, sawada-san, >> >> Thanks for testing! >> >> FWIF, I tested using pgbench with "--rate=" option to know the server >> can execute transactions with stable throughput. As

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/06/24 22:11, Masahiko Sawada wrote: > On Sat, Jun 12, 2021 at 1:25 AM Fujii Masao > wrote: >> On 2021/05/11 13:37, Masahiko Sawada wrote: >> So currently FdwXactInsertEntry() calls XLogInsert() and XLogFlush() for >> XLOG_FDWXACT_INSERT WAL record. Additionally we should also wait

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/06/24 22:27, Masahiko Sawada wrote: > On Thu, Jun 24, 2021 at 9:46 PM k.jami...@fujitsu.com > wrote: >> >> Hi Sawada-san, >> >> I also tried to play a bit with the latest patches similar to Ikeda-san, >> and with foreign 2PC parameter enabled/required. > > Thank you for testing the

Re: Transactions involving multiple postgres foreign servers, take 2

On Thu, Jun 24, 2021 at 10:11 PM Masahiko Sawada wrote: > > On Sat, Jun 12, 2021 at 1:25 AM Fujii Masao > wrote: > > > > > > > > (5) > > +2. Pre-Commit phase (1st phase of two-phase commit) > > +we record the corresponding WAL indicating that the foreign server is > > involved > > +with the

Re: Transactions involving multiple postgres foreign servers, take 2

On Thu, Jun 24, 2021 at 9:46 PM k.jami...@fujitsu.com wrote: > > Hi Sawada-san, > > I also tried to play a bit with the latest patches similar to Ikeda-san, > and with foreign 2PC parameter enabled/required. Thank you for testing the patch! > > > > >> b. about performance bottleneck (just share

Re: Transactions involving multiple postgres foreign servers, take 2

On Sat, Jun 12, 2021 at 1:25 AM Fujii Masao wrote: > > > > On 2021/05/11 13:37, Masahiko Sawada wrote: > > I've attached the updated patches that incorporated comments from > > Zhihong and Ikeda-san. > > Thanks for updating the patches! > > I'm still reading these patches, but I'd like to share

RE: Transactions involving multiple postgres foreign servers, take 2

Hi Sawada-san, I also tried to play a bit with the latest patches similar to Ikeda-san, and with foreign 2PC parameter enabled/required. > > >> b. about performance bottleneck (just share my simple benchmark > > >> results) > > >> > > >> The resolver process can be performance bottleneck easily

RE: Transactions involving multiple postgres foreign servers, take 2

From: Robert Haas > On Tue, Jun 15, 2021 at 5:51 AM tsunakawa.ta...@fujitsu.com > wrote: > > Postgres can do that, but other implementations can not necessaily do it, > > I'm > afraid. But before that, the FDW interface documentation doesn't describe > anything about how to handle

Re: Transactions involving multiple postgres foreign servers, take 2

On Tue, Jun 15, 2021 at 5:51 AM tsunakawa.ta...@fujitsu.com wrote: > Postgres can do that, but other implementations can not necessaily do it, I'm > afraid. But before that, the FDW interface documentation doesn't describe > anything about how to handle interrupts. Actually, odbc_fdw and

RE: Transactions involving multiple postgres foreign servers, take 2

From: Robert Haas > Well, we're talking about running this commit routine from within > CommitTransaction(), right? So I think it is in fact running in the > server. And if that's so, then you have to worry about how to make it > respond to interrupts. You can't just call some functions >

Re: Transactions involving multiple postgres foreign servers, take 2

On Sun, Jun 13, 2021 at 10:04 PM tsunakawa.ta...@fujitsu.com wrote: > I know sending a commit request may get an error from various underlying > functions, but we're talking about the client side, not the Postgres's server > side that could unexpectedly ereport(ERROR) somewhere. So, the new

RE: Transactions involving multiple postgres foreign servers, take 2

From: Robert Haas > On Thu, Jun 10, 2021 at 9:58 PM tsunakawa.ta...@fujitsu.com > wrote: > > The question I have been asking is how. With that said, we should only have > two options; one is the return value of the FDW commit routine, and the other > is > via ereport(ERROR). I suggested the

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/05/11 13:37, Masahiko Sawada wrote: I've attached the updated patches that incorporated comments from Zhihong and Ikeda-san. Thanks for updating the patches! I'm still reading these patches, but I'd like to share some review comments that I found so far. (1) +/* Remove the foreign

Re: Transactions involving multiple postgres foreign servers, take 2

On Thu, Jun 10, 2021 at 9:58 PM tsunakawa.ta...@fujitsu.com wrote: > I understand that. As I cited yesterday and possibly before, that's why > xa_commit() returns various return codes. So, I have never suggested that > FDWs should not report an error and always report success for the commit

RE: Transactions involving multiple postgres foreign servers, take 2

From: Robert Haas > That is completely unrealistic. As Sawada-san has pointed out > repeatedly, there are tons of things that can go wrong even after the > remote side has prepared the transaction. Preparing a transaction only > promises that the remote side will let you commit the transaction

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Jun 4, 2021 at 4:04 AM tsunakawa.ta...@fujitsu.com wrote: > Why does the user have to get an error? Once the local transaction has been > prepared, which means all remote ones also have been prepared, the whole > transaction is determined to commit. So, the user doesn't have to

RE: Transactions involving multiple postgres foreign servers, take 2

From: Kyotaro Horiguchi > If we accept each elementary-commit (via FDW connection) to fail, the > parent(?) there's no way the root 2pc-commit can succeed. How can we > ignore the fdw-error in that case? No, we don't ignore the error during FDW commit. As mentioned at the end of this mail,

Re: Transactions involving multiple postgres foreign servers, take 2

At Tue, 8 Jun 2021 08:45:24 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Kyotaro Horiguchi > > I think the discussion is based the behavior that any process that is > > responsible for finishing the 2pc-commit continue retrying remote > > commits until all of the remote-commits

RE: Transactions involving multiple postgres foreign servers, take 2

From: Masahiko Sawada > Maybe it's better to start a new thread to discuss this topic. If your > idea is good, we can lower all error that happened after writing the > commit record to warning, reducing the cases where the client gets > confusion by receiving an error after the commit. No. It's

Re: Transactions involving multiple postgres foreign servers, take 2

On Wed, Jun 9, 2021 at 4:10 PM tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > On Tue, Jun 8, 2021 at 5:28 PM tsunakawa.ta...@fujitsu.com > > wrote: > > > Then, in what kind of scenario are we talking about the difficulty, and > > > how is > > it difficult to handle, when we

RE: Transactions involving multiple postgres foreign servers, take 2

From: Masahiko Sawada > On Tue, Jun 8, 2021 at 5:28 PM tsunakawa.ta...@fujitsu.com > wrote: > > Then, in what kind of scenario are we talking about the difficulty, and how > > is > it difficult to handle, when we adopt either the method 1 or 2? (I'd just > like to > have the same clear

Re: Transactions involving multiple postgres foreign servers, take 2

On Tue, Jun 8, 2021 at 5:28 PM tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > On Tue, Jun 8, 2021 at 9:47 AM tsunakawa.ta...@fujitsu.com > > wrote: > > > Why does the client have to know the error on a remote server, whereas the > > global transaction itself is destined to

RE: Transactions involving multiple postgres foreign servers, take 2

From: Kyotaro Horiguchi > I think the discussion is based the behavior that any process that is > responsible for finishing the 2pc-commit continue retrying remote > commits until all of the remote-commits succeed. Thank you for coming back. We're talking about the first attempt to prepare and

RE: Transactions involving multiple postgres foreign servers, take 2

From: Masahiko Sawada > On Tue, Jun 8, 2021 at 9:47 AM tsunakawa.ta...@fujitsu.com > wrote: > > Why does the client have to know the error on a remote server, whereas the > global transaction itself is destined to commit? > > It's not necessarily on a remote server. It could be a problem with >

Re: Transactions involving multiple postgres foreign servers, take 2

At Tue, 8 Jun 2021 16:32:14 +0900, Masahiko Sawada wrote in > On Tue, Jun 8, 2021 at 9:47 AM tsunakawa.ta...@fujitsu.com > wrote: > > > > From: Masahiko Sawada > > > I think we should not reinterpret the severity of the error and lower > > > it. Especially, in this case, any kind of errors

Re: Transactions involving multiple postgres foreign servers, take 2

(I have caught up here. Sorry in advance for possible pointless discussion by me..) At Tue, 8 Jun 2021 00:47:08 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Masahiko Sawada > > I think we should not reinterpret the severity of the error and lower > > it. Especially, in this case, any

Re: Transactions involving multiple postgres foreign servers, take 2

On Tue, Jun 8, 2021 at 9:47 AM tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > I think we should not reinterpret the severity of the error and lower > > it. Especially, in this case, any kind of errors can be thrown. It > > could be such a serious error that FDW developer wants

RE: Transactions involving multiple postgres foreign servers, take 2

From: Masahiko Sawada > I think we should not reinterpret the severity of the error and lower > it. Especially, in this case, any kind of errors can be thrown. It > could be such a serious error that FDW developer wants to report to > the client. Do we lower even PANIC to a lower severity such as

Re: Transactions involving multiple postgres foreign servers, take 2

> 2021/06/04 21:38、Masahiko Sawada のメール: > > On Fri, Jun 4, 2021 at 5:16 PM Masahiko Sawada wrote: >> >> On Fri, Jun 4, 2021 at 3:58 PM ikeda...@oss.nttdata.com >> wrote: >>> >>> >>> >>> 2021/06/04 12:28、Masahiko Sawada のメール: >>> >>> >>> Thank you for pointing it out. This idea has

Re: Transactions involving multiple postgres foreign servers, take 2

> 2021/06/04 17:16、Masahiko Sawada のメール: > > On Fri, Jun 4, 2021 at 3:58 PM ikeda...@oss.nttdata.com > wrote: >> >> >> >> 2021/06/04 12:28、Masahiko Sawada のメール: >> >> >> Thank you for pointing it out. This idea has been proposed several >> times and there were discussions. I'd like to

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Jun 4, 2021 at 5:16 PM Masahiko Sawada wrote: > > On Fri, Jun 4, 2021 at 3:58 PM ikeda...@oss.nttdata.com > wrote: > > > > > > > > 2021/06/04 12:28、Masahiko Sawada のメール: > > > > > > Thank you for pointing it out. This idea has been proposed several > > times and there were discussions.

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Jun 4, 2021 at 5:59 PM tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > On Fri, Jun 4, 2021 at 5:04 PM tsunakawa.ta...@fujitsu.com > > wrote: > > > Why does the user have to get an error? Once the local transaction has > > > been > > prepared, which means all remote

RE: Transactions involving multiple postgres foreign servers, take 2

From: Masahiko Sawada > On Fri, Jun 4, 2021 at 5:04 PM tsunakawa.ta...@fujitsu.com > wrote: > > Why does the user have to get an error? Once the local transaction has been > prepared, which means all remote ones also have been prepared, the whole > transaction is determined to commit. So, the

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Jun 4, 2021 at 5:04 PM tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > 1. the backend continues attempting to commit all prepared foreign > > transactions until all of them are committed. > > 2. the backend attempts to commit all prepared foreign transactions > > once. If

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Jun 4, 2021 at 3:58 PM ikeda...@oss.nttdata.com wrote: > > > > 2021/06/04 12:28、Masahiko Sawada のメール: > > > Thank you for pointing it out. This idea has been proposed several > times and there were discussions. I'd like to summarize the proposed > ideas and those pros and cons before

RE: Transactions involving multiple postgres foreign servers, take 2

From: Masahiko Sawada 1. the backend continues attempting to commit all prepared foreign > transactions until all of them are committed. > 2. the backend attempts to commit all prepared foreign transactions > once. If an error happens, leave them for the resolver. > 3. the backend asks the

Re: Transactions involving multiple postgres foreign servers, take 2

> 2021/06/04 12:28、Masahiko Sawada のメール: > > On Thu, Jun 3, 2021 at 1:56 PM Masahiro Ikeda > wrote: >> >> >> >> On 2021/05/25 21:59, Masahiko Sawada wrote: >>> On Fri, May 21, 2021 at 5:48 PM Masahiro Ikeda >>> wrote: On 2021/05/21 13:45,

Re: Transactions involving multiple postgres foreign servers, take 2

On Thu, Jun 3, 2021 at 1:56 PM Masahiro Ikeda wrote: > > > > On 2021/05/25 21:59, Masahiko Sawada wrote: > > On Fri, May 21, 2021 at 5:48 PM Masahiro Ikeda > > wrote: > >> > >> On 2021/05/21 13:45, Masahiko Sawada wrote: > >>> > >>> Yes. We also might need to be careful about the order of

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/05/25 21:59, Masahiko Sawada wrote: > On Fri, May 21, 2021 at 5:48 PM Masahiro Ikeda > wrote: >> >> On 2021/05/21 13:45, Masahiko Sawada wrote: >>> >>> Yes. We also might need to be careful about the order of foreign >>> transaction resolution. I think we need to resolve foreign>

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, May 21, 2021 at 5:48 PM Masahiro Ikeda wrote: > > > > On 2021/05/21 13:45, Masahiko Sawada wrote: > > > > Yes. We also might need to be careful about the order of foreign > > transaction resolution. I think we need to resolve foreign> transactions in > > arrival order at least within a

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/05/21 13:45, Masahiko Sawada wrote: > On Fri, May 21, 2021 at 12:45 PM Masahiro Ikeda > wrote: >> >> >> >> On 2021/05/21 10:39, Masahiko Sawada wrote: >>> On Thu, May 20, 2021 at 1:26 PM Masahiro Ikeda >>> wrote: On 2021/05/11 13:37, Masahiko Sawada wrote: > I've

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, May 21, 2021 at 12:45 PM Masahiro Ikeda wrote: > > > > On 2021/05/21 10:39, Masahiko Sawada wrote: > > On Thu, May 20, 2021 at 1:26 PM Masahiro Ikeda > > wrote: > >> > >> > >> On 2021/05/11 13:37, Masahiko Sawada wrote: > >>> I've attached the updated patches that incorporated comments

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/05/21 10:39, Masahiko Sawada wrote: > On Thu, May 20, 2021 at 1:26 PM Masahiro Ikeda > wrote: >> >> >> On 2021/05/11 13:37, Masahiko Sawada wrote: >>> I've attached the updated patches that incorporated comments from >>> Zhihong and Ikeda-san. >> >> Thanks for updating the patches! >>

Re: Transactions involving multiple postgres foreign servers, take 2

On Thu, May 20, 2021 at 1:26 PM Masahiro Ikeda wrote: > > > On 2021/05/11 13:37, Masahiko Sawada wrote: > > I've attached the updated patches that incorporated comments from > > Zhihong and Ikeda-san. > > Thanks for updating the patches! > > > I have other comments including trivial things. > > >

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/05/11 13:37, Masahiko Sawada wrote: > I've attached the updated patches that incorporated comments from > Zhihong and Ikeda-san. Thanks for updating the patches! I have other comments including trivial things. a. about "foreign_transaction_resolver_timeout" parameter Now, the

Re: Transactions involving multiple postgres foreign servers, take 2

On Mon, May 10, 2021 at 9:38 PM Masahiko Sawada wrote: > On Mon, May 3, 2021 at 11:11 PM Zhihong Yu wrote: > > > > > > > > On Mon, May 3, 2021 at 5:25 AM Masahiko Sawada > wrote: > >> > >> On Sun, May 2, 2021 at 1:23 AM Zhihong Yu wrote: > >> > > >> > > >> > > >> > On Fri, Apr 30, 2021 at

Re: Transactions involving multiple postgres foreign servers, take 2

On Mon, May 3, 2021 at 5:25 AM Masahiko Sawada wrote: > On Sun, May 2, 2021 at 1:23 AM Zhihong Yu wrote: > > > > > > > > On Fri, Apr 30, 2021 at 9:09 PM Masahiko Sawada > wrote: > >> > >> On Wed, Mar 17, 2021 at 6:03 PM Zhihong Yu wrote: > >> > > >> > Hi, > >> > For

Re: Transactions involving multiple postgres foreign servers, take 2

On Sun, May 2, 2021 at 1:23 AM Zhihong Yu wrote: > > > > On Fri, Apr 30, 2021 at 9:09 PM Masahiko Sawada wrote: >> >> On Wed, Mar 17, 2021 at 6:03 PM Zhihong Yu wrote: >> > >> > Hi, >> > For v35-0007-Prepare-foreign-transactions-at-commit-time.patch : >> >> Thank you for reviewing the patch! >>

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Apr 30, 2021 at 9:09 PM Masahiko Sawada wrote: > On Wed, Mar 17, 2021 at 6:03 PM Zhihong Yu wrote: > > > > Hi, > > For v35-0007-Prepare-foreign-transactions-at-commit-time.patch : > > Thank you for reviewing the patch! > > > > > With this commit, the foreign server modified within the

Re: Transactions involving multiple postgres foreign servers, take 2

On Wed, Mar 17, 2021 at 6:03 PM Zhihong Yu wrote: > > Hi, > For v35-0007-Prepare-foreign-transactions-at-commit-time.patch : Thank you for reviewing the patch! > > With this commit, the foreign server modified within the transaction marked > as 'modified'. > > transaction marked -> transaction

Re: Transactions involving multiple postgres foreign servers, take 2

On Tue, Apr 27, 2021 at 10:03 AM Masahiro Ikeda wrote: > > > > On 2021/03/17 12:03, Masahiko Sawada wrote: > > I've attached the updated version patch set. > > Thanks for updating the patches! I'm now restarting to review of 2PC because > I'd like to use this feature in PG15. Thank you for

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/03/17 12:03, Masahiko Sawada wrote: > I've attached the updated version patch set. Thanks for updating the patches! I'm now restarting to review of 2PC because I'd like to use this feature in PG15. I think the following logic of resolving and removing the fdwxact entries by the

Re: Transactions involving multiple postgres foreign servers, take 2

Hi, For v35-0007-Prepare-foreign-transactions-at-commit-time.patch : With this commit, the foreign server modified within the transaction marked as 'modified'. transaction marked -> transaction is marked +#define IsForeignTwophaseCommitRequested() \ +(foreign_twophase_commit >

Re: Transactions involving multiple postgres foreign servers, take 2

On Thu, Feb 11, 2021 at 6:25 PM Masahiko Sawada wrote: > On Fri, Feb 5, 2021 at 2:45 PM Masahiko Sawada > wrote: > > > > On Tue, Feb 2, 2021 at 5:18 PM Fujii Masao > wrote: > > > > > > > > > > > > On 2021/01/27 14:08, Masahiko Sawada wrote: > > > > On Wed, Jan 27, 2021 at 10:29 AM Fujii Masao

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Feb 5, 2021 at 2:45 PM Masahiko Sawada wrote: > > On Tue, Feb 2, 2021 at 5:18 PM Fujii Masao > wrote: > > > > > > > > On 2021/01/27 14:08, Masahiko Sawada wrote: > > > On Wed, Jan 27, 2021 at 10:29 AM Fujii Masao > > > wrote: > > >> > > >> > > >> You fixed some issues. But maybe you

Re: Transactions involving multiple postgres foreign servers, take 2

On Tue, Feb 2, 2021 at 5:18 PM Fujii Masao wrote: > > > > On 2021/01/27 14:08, Masahiko Sawada wrote: > > On Wed, Jan 27, 2021 at 10:29 AM Fujii Masao > > wrote: > >> > >> > >> You fixed some issues. But maybe you forgot to attach the latest patches? > > > > Yes, I've attached the updated

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/01/27 14:08, Masahiko Sawada wrote: On Wed, Jan 27, 2021 at 10:29 AM Fujii Masao wrote: You fixed some issues. But maybe you forgot to attach the latest patches? Yes, I've attached the updated patches. Thanks for updating the patch! I tried to review 0001 and 0002 as the

Re: Transactions involving multiple postgres foreign servers, take 2

On Sat, Jan 16, 2021 at 1:39 AM Zhihong Yu wrote: > > Hi, Thank you for reviewing the patch! > For v32-0004-Add-PrepareForeignTransaction-API.patch : > > + * Whenever a foreign transaction is processed, the corresponding FdwXact > + * entry is update. To avoid holding the lock during

Re: Transactions involving multiple postgres foreign servers, take 2

On 2021/01/18 14:54, Masahiko Sawada wrote: On Fri, Jan 15, 2021 at 7:45 AM Zhihong Yu wrote: For v32-0002-postgres_fdw-supports-commit-and-rollback-APIs.patch : + entry->changing_xact_state = true; ... + entry->changing_xact_state = abort_cleanup_failure; I don't see return

Re: Transactions involving multiple postgres foreign servers, take 2

Hi, Masahiko-san: bq. How about FdwXactRequestToLaunchResolver()? Sounds good to me. bq. But there is already a function named FdwXactExists() Then we can leave the function name as it is. Cheers On Sun, Jan 17, 2021 at 9:55 PM Masahiko Sawada wrote: > On Fri, Jan 15, 2021 at 7:45 AM

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Jan 15, 2021 at 7:45 AM Zhihong Yu wrote: > > For v32-0002-postgres_fdw-supports-commit-and-rollback-APIs.patch : > > + entry->changing_xact_state = true; > ... > + entry->changing_xact_state = abort_cleanup_failure; > > I don't see return statement in between the two assignments. I

Re: Transactions involving multiple postgres foreign servers, take 2

Hi, For v32-0004-Add-PrepareForeignTransaction-API.patch : + * Whenever a foreign transaction is processed, the corresponding FdwXact + * entry is update. To avoid holding the lock during transaction processing + * which may take an unpredicatable time the in-memory data of foreign entry is

Re: Transactions involving multiple postgres foreign servers, take 2

On Fri, Jan 15, 2021 at 4:03 AM Zhihong Yu wrote: > > Hi, > For v32-0008-Prepare-foreign-transactions-at-commit-time.patch : Thank you for reviewing the patch! > > + boolhave_notwophase = false; > > Maybe name the variable have_no_twophase so that it is easier to read. Fixed. > > +

Re: Transactions involving multiple postgres foreign servers, take 2

For v32-0002-postgres_fdw-supports-commit-and-rollback-APIs.patch : + entry->changing_xact_state = true; ... + entry->changing_xact_state = abort_cleanup_failure; I don't see return statement in between the two assignments. I wonder why entry->changing_xact_state is set to true, and later

Re: Transactions involving multiple postgres foreign servers, take 2

Hi, For v32-0008-Prepare-foreign-transactions-at-commit-time.patch : + boolhave_notwophase = false; Maybe name the variable have_no_twophase so that it is easier to read. +* Two-phase commit is not required if the number of servers performed performed -> performing +

Re: Transactions involving multiple postgres foreign servers, take 2

Hi, For pg-foreign/v31-0004-Add-PrepareForeignTransaction-API.patch : However these functions are not neither committed nor aborted at I think the double negation was not intentional. Should be 'are neither ...' For FdwXactShmemSize(), is another MAXALIGN(size) needed prior to the return

Re: Transactions involving multiple postgres foreign servers, take 2

On Wed, 21 Oct 2020 at 18:33, tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > So what's your opinion? > > My opinion is simple and has not changed. Let's clarify and refine the > design first in the following areas (others may have pointed out something > else too, but I

Re: Transactions involving multiple postgres foreign servers, take 2

On Wed, Oct 21, 2020 at 3:03 PM tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > So what's your opinion? > > * Global visibility > This is what Amit-san suggested some times -- "design it before reviewing the > current patch." I'm a bit optimistic about this and think this FDW

RE: Transactions involving multiple postgres foreign servers, take 2

From: Masahiko Sawada > So what's your opinion? My opinion is simple and has not changed. Let's clarify and refine the design first in the following areas (others may have pointed out something else too, but I don't remember), before going deeper into the code review. * FDW interface New

RE: Transactions involving multiple postgres foreign servers, take 2

From: Kyotaro Horiguchi > > if (PQstatus(entry->conn) != CONNECTION_OK || > > PQtransactionStatus(entry->conn) != PQTRANS_IDLE || > > entry->changing_xact_state) > > { > > elog(DEBUG3, "discarding connection %p", entry->conn); > >

Re: Transactions involving multiple postgres foreign servers, take 2

At Tue, 20 Oct 2020 21:22:31 +0900, Masahiko Sawada wrote in > On Tue, 20 Oct 2020 at 17:56, tsunakawa.ta...@fujitsu.com > wrote: > > > > From: Kyotaro Horiguchi > > > It seems to respond to a statement-cancel signal immediately while > > > waiting for a coming byte. However, seems to wait

Re: Transactions involving multiple postgres foreign servers, take 2

On Tue, 20 Oct 2020 at 17:56, tsunakawa.ta...@fujitsu.com wrote: > > From: Kyotaro Horiguchi > > It seems to respond to a statement-cancel signal immediately while > > waiting for a coming byte. However, seems to wait forever while > > waiting a space in send-buffer. (Is that mean the session

Re: Transactions involving multiple postgres foreign servers, take 2

On Tue, 20 Oct 2020 at 16:54, tsunakawa.ta...@fujitsu.com wrote: > > From: Kyotaro Horiguchi > > At Tue, 20 Oct 2020 15:53:29 +0900, Masahiko Sawada > > wrote in > > > I think it doesn't matter whether in FDW framework or not. The user > > > normally doesn't care which backend processes

RE: Transactions involving multiple postgres foreign servers, take 2

From: Kyotaro Horiguchi > It seems to respond to a statement-cancel signal immediately while > waiting for a coming byte. However, seems to wait forever while > waiting a space in send-buffer. (Is that mean the session will be > stuck if it sends a large chunk of bytes while the network is

RE: Transactions involving multiple postgres foreign servers, take 2

From: Kyotaro Horiguchi > At Tue, 20 Oct 2020 15:53:29 +0900, Masahiko Sawada > wrote in > > I think it doesn't matter whether in FDW framework or not. The user > > normally doesn't care which backend processes connecting to foreign > > servers. They will attempt to cancel the query like always

Re: Transactions involving multiple postgres foreign servers, take 2

At Tue, 20 Oct 2020 04:23:12 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Kyotaro Horiguchi > > > Furthermore, FDW is not cancellable in general. So, I don't see a point > > > in > > trying hard to make only commit be cancelable. > > > > I think that it is quite important that

Re: Transactions involving multiple postgres foreign servers, take 2

At Tue, 20 Oct 2020 15:53:29 +0900, Masahiko Sawada wrote in > On Tue, 20 Oct 2020 at 13:23, tsunakawa.ta...@fujitsu.com > wrote: > > > > From: Kyotaro Horiguchi > > > I don't think the inability to cancel all session at once cannot be a > > > reason not to not to allow operators to cancel a

Re: Transactions involving multiple postgres foreign servers, take 2

On Tue, 20 Oct 2020 at 13:23, tsunakawa.ta...@fujitsu.com wrote: > > From: Kyotaro Horiguchi > > I don't think the inability to cancel all session at once cannot be a > > reason not to not to allow operators to cancel a stuck session. > > Yeah, I didn't mean to discount the ability to cancel

RE: Transactions involving multiple postgres foreign servers, take 2

From: Kyotaro Horiguchi > I don't think the inability to cancel all session at once cannot be a > reason not to not to allow operators to cancel a stuck session. Yeah, I didn't mean to discount the ability to cancel queries. I just want to confirm how the user can use the cancellation in

Re: Transactions involving multiple postgres foreign servers, take 2

At Tue, 20 Oct 2020 02:44:09 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Ashutosh Bapat > > Using pg_cancel_backend() and pg_terminate_backend() a DBA can cancel > > running query from any backend or terminate a backend. For either to > > work the backend needs to be interruptible.

RE: Transactions involving multiple postgres foreign servers, take 2

From: Ashutosh Bapat > Using pg_cancel_backend() and pg_terminate_backend() a DBA can cancel > running query from any backend or terminate a backend. For either to > work the backend needs to be interruptible. IIRC, Robert had made an > effort to make postgres_fdw interruptible few years back.

  1   2   3   >