Re: [HACKERS] logical decoding of two-phase transactions

2022-01-04 Thread Amit Kapila
On Tue, Jan 4, 2022 at 9:00 AM Masahiko Sawada wrote: > > According to the doc, the two_phase field has: > > True if the slot is enabled for decoding prepared transactions. Always > false for physical slots. > > It's unnatural a bit to me that replication slots have such a property > since the

Re: [HACKERS] logical decoding of two-phase transactions

2022-01-03 Thread Masahiko Sawada
Hi, On Mon, Aug 9, 2021 at 12:00 PM Amit Kapila wrote: > > On Wed, Aug 4, 2021 at 4:14 PM Amit Kapila wrote: > > > > I have pushed this last patch in the series. > > > > I have closed this CF entry. Thanks to everyone involved in this work! > I have a questoin about two_phase column of

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-08 Thread Amit Kapila
On Wed, Aug 4, 2021 at 4:14 PM Amit Kapila wrote: > > I have pushed this last patch in the series. > I have closed this CF entry. Thanks to everyone involved in this work! -- With Regards, Amit Kapila.

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-04 Thread Amit Kapila
On Wed, Aug 4, 2021 at 6:51 AM tanghy.f...@fujitsu.com wrote: > > On Tuesday, August 3, 2021 6:03 PM vignesh C wrote: > > > > On Tue, Aug 3, 2021 at 12:32 PM Amit Kapila wrote: > > > > > > On Tue, Aug 3, 2021 at 6:17 AM Peter Smith wrote: > > > > > > > > Please find attached the latest patch

RE: [HACKERS] logical decoding of two-phase transactions

2021-08-03 Thread tanghy.f...@fujitsu.com
On Tuesday, August 3, 2021 6:03 PM vignesh C wrote: > > On Tue, Aug 3, 2021 at 12:32 PM Amit Kapila wrote: > > > > On Tue, Aug 3, 2021 at 6:17 AM Peter Smith wrote: > > > > > > Please find attached the latest patch set v102* > > > > > > > I have made minor modifications in the comments and

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-03 Thread vignesh C
On Tue, Aug 3, 2021 at 12:32 PM Amit Kapila wrote: > > On Tue, Aug 3, 2021 at 6:17 AM Peter Smith wrote: > > > > Please find attached the latest patch set v102* > > > > I have made minor modifications in the comments and docs, please see > attached. Can you please check whether the names of

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-03 Thread Peter Smith
On Tue, Aug 3, 2021 at 5:02 PM Amit Kapila wrote: > > On Tue, Aug 3, 2021 at 6:17 AM Peter Smith wrote: > > > > Please find attached the latest patch set v102* > > > > I have made minor modifications in the comments and docs, please see > attached. Can you please check whether the names of

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-03 Thread Amit Kapila
On Tue, Aug 3, 2021 at 6:17 AM Peter Smith wrote: > > Please find attached the latest patch set v102* > I have made minor modifications in the comments and docs, please see attached. Can you please check whether the names of contributors in the commit message are correct or do we need to change

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Mon, May 10, 2021 at 1:31 PM vignesh C wrote: > ... > > 2) I felt we can change lsn data type from Int64 to XLogRecPtr > + > +Int64 > + > +The LSN of the prepare. > + > + > + > + > +Int64 > + > +The end LSN of the transaction. > + > + > > 3) I felt we can change

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
Please find attached the latest patch set v102* Differences: * Rebased to HEAD @ today. * This is a documentation change only. A recent commit [1] has changed the documentation style for the message formats slightly to annotate the data types. For consistency, the same style change needs to be

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Amit Kapila
On Sun, Aug 1, 2021 at 3:51 PM Peter Smith wrote: > > On Sun, Aug 1, 2021 at 3:05 AM vignesh C wrote: > > > > On Sat, Jul 31, 2021 at 11:12 AM Ajin Cherian wrote: > > > > > > On Sat, Jul 31, 2021 at 2:39 PM Amit Kapila > > > wrote: > > > > > > > Here, the test is expecting 2 prepared

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Fri, Jul 30, 2021 at 4:33 PM Greg Nancarrow wrote: > > On Fri, Jul 30, 2021 at 2:02 PM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > v99-0002 --> v100-0001 > > > > A few minor comments: > > (1) doc/src/sgml/protocol.sgml > > In the following description,

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Fri, Jul 30, 2021 at 3:18 PM vignesh C wrote: > > On Fri, Jul 30, 2021 at 9:32 AM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > v99-0002 --> v100-0001 > > > > Differences: > > > > * Rebased to HEAD @ today (needed because some recent commits [1][2] broke

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Fri, Jul 30, 2021 at 6:25 PM tanghy.f...@fujitsu.com wrote: > > On Friday, July 30, 2021 12:02 PM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > v99-0002 --> v100-0001 > > > > Thanks for your patch. A few comments on the test file: > > 1.

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
On Sat, Jul 31, 2021 at 9:36 PM Amit Kapila wrote: > > On Fri, Jul 30, 2021 at 9:32 AM Peter Smith wrote: > > > > Please find attached the latest patch set v100* > > > > Few minor comments: > 1. > CREATE SUBSCRIPTION regress_testsub CONNECTION > 'dbname=regress_doesnotexist' PUBLICATION testpub

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-02 Thread Peter Smith
Please find attached the latest patch v101 Differences: * Rebased to HEAD @ today. * Addresses all v100 review comments from Vignesh [1], Greg [2], Tang [3], and Amit [2]. [1] https://www.postgresql.org/message-id/CALDaNm2N3qgSv3XyHW%2Bop_SJcLmz1s%3D0jJc-taxUmeEBXW5EPw%40mail.gmail.com

Re: [HACKERS] logical decoding of two-phase transactions

2021-08-01 Thread Peter Smith
On Sun, Aug 1, 2021 at 3:05 AM vignesh C wrote: > > On Sat, Jul 31, 2021 at 11:12 AM Ajin Cherian wrote: > > > > On Sat, Jul 31, 2021 at 2:39 PM Amit Kapila wrote: > > > > > Here, the test is expecting 2 prepared transactions corresponding to > > > two subscriptions but it waits for just one

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-31 Thread vignesh C
On Sat, Jul 31, 2021 at 11:12 AM Ajin Cherian wrote: > > On Sat, Jul 31, 2021 at 2:39 PM Amit Kapila wrote: > > > Here, the test is expecting 2 prepared transactions corresponding to > > two subscriptions but it waits for just one subscription via > > appname_copy. It should wait for the second

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-31 Thread Amit Kapila
On Fri, Jul 30, 2021 at 9:32 AM Peter Smith wrote: > > Please find attached the latest patch set v100* > Few minor comments: 1. CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, two_phase = true); \dRs+ + --fail - alter of

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-31 Thread Amit Kapila
On Sat, Jul 31, 2021 at 11:12 AM Ajin Cherian wrote: > > On Sat, Jul 31, 2021 at 2:39 PM Amit Kapila wrote: > > > Here, the test is expecting 2 prepared transactions corresponding to > > two subscriptions but it waits for just one subscription via > > appname_copy. It should wait for the second

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-30 Thread Ajin Cherian
On Sat, Jul 31, 2021 at 2:39 PM Amit Kapila wrote: > Here, the test is expecting 2 prepared transactions corresponding to > two subscriptions but it waits for just one subscription via > appname_copy. It should wait for the second subscription using > $appname as well. > > What do you think? I

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-30 Thread Amit Kapila
On Wed, Jul 14, 2021 at 11:52 AM Amit Kapila wrote: > > On Mon, Jul 12, 2021 at 9:14 AM Peter Smith wrote: > > Pushed. > As reported by Michael [1], there is one test failure related to this commit. The failure is as below: # Failed test 'transaction is prepared on subscriber' # at

RE: [HACKERS] logical decoding of two-phase transactions

2021-07-30 Thread tanghy.f...@fujitsu.com
On Friday, July 30, 2021 12:02 PM Peter Smith wrote: > > Please find attached the latest patch set v100* > > v99-0002 --> v100-0001 > Thanks for your patch. A few comments on the test file: 1. src/test/subscription/t/022_twophase_cascade.pl 1.1 I saw your test cases for "PREPARE / COMMIT

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-30 Thread Greg Nancarrow
On Fri, Jul 30, 2021 at 2:02 PM Peter Smith wrote: > > Please find attached the latest patch set v100* > > v99-0002 --> v100-0001 > A few minor comments: (1) doc/src/sgml/protocol.sgml In the following description, is the word "large" really needed? Also "the message ... for a ... message"

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-29 Thread vignesh C
On Fri, Jul 30, 2021 at 9:32 AM Peter Smith wrote: > > Please find attached the latest patch set v100* > > v99-0002 --> v100-0001 > > Differences: > > * Rebased to HEAD @ today (needed because some recent commits [1][2] broke > v99) > The patch applies neatly, tests passes and documentation

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-29 Thread Peter Smith
On Thu, Jul 29, 2021 at 9:56 PM Amit Kapila wrote: > > On Tue, Jul 27, 2021 at 11:41 AM Peter Smith wrote: > > > > Please find attached the latest patch set v99* > > > > v98-0001 --> split into v99-0001 + v99-0002 > > > > Pushed the first refactoring patch after making few modifications as

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-29 Thread Peter Smith
Please find attached the latest patch set v100* v99-0002 --> v100-0001 Differences: * Rebased to HEAD @ today (needed because some recent commits [1][2] broke v99) * Addresses v99 review comments from Amit [1]. [1]

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-29 Thread Amit Kapila
On Tue, Jul 27, 2021 at 11:41 AM Peter Smith wrote: > > Please find attached the latest patch set v99* > > v98-0001 --> split into v99-0001 + v99-0002 > Pushed the first refactoring patch after making few modifications as below. 1. - /* open the spool file for the committed transaction */ + /*

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-27 Thread Peter Smith
Please find attached the latest patch set v99* v98-0001 --> split into v99-0001 + v99-0002 Differences: * Rebased to HEAD @ yesterday. * Addresses review comments from Amit [1] and split the v98 patch as requested. [1]

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-27 Thread Peter Smith
On Fri, Jul 23, 2021 at 8:08 PM Amit Kapila wrote: > > On Tue, Jul 20, 2021 at 9:24 AM Peter Smith wrote: > > > > Please find attached the latest patch set v98* > > > > Review comments: > All the following review comments are addressed in v99* patch set. > 1. > /* > - * Handle

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-26 Thread Peter Smith
On Fri, Jul 23, 2021 at 8:08 PM Amit Kapila wrote: > > On Tue, Jul 20, 2021 at 9:24 AM Peter Smith wrote: > > > > Please find attached the latest patch set v98* > > > > Review comments: > [...] > With Regards, > Amit Kapila. Thanks for your review comments. I having been

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-23 Thread Amit Kapila
On Tue, Jul 20, 2021 at 9:24 AM Peter Smith wrote: > > Please find attached the latest patch set v98* > Review comments: 1. /* - * Handle STREAM COMMIT message. + * Common spoolfile processing. + * Returns how many changes were applied. */ -static void

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-19 Thread Peter Smith
On Fri, Jul 16, 2021 at 4:08 PM vignesh C wrote: > [...] > Thanks for the updated patch, the patch applies cleanly and test passes: > I had couple of comments: > 1) Should we include "stream_prepare_cb" here in > logicaldecoding-streaming section of logicaldecoding.sgml > documentation: > To

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-19 Thread Peter Smith
On Mon, Jul 19, 2021 at 3:28 PM Greg Nancarrow wrote: > > On Wed, Jul 14, 2021 at 6:33 PM Peter Smith wrote: > > > > Please find attached the latest patch set v97* > > > > I couldn't spot spot any significant issues in the v97-0001 patch, but > do have the following trivial feedback comments: >

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-19 Thread Peter Smith
Please find attached the latest patch set v98* Patches: v97-0001 --> v98-0001 Differences: * Rebased to HEAD @ yesterday. * Code/Docs changes: 1. Fixed the same strcpy problem as reported by Tom Lane [1] for the previous 2PC patch. 2. Addressed all feedback suggestions given by Greg [2].

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-19 Thread Amit Kapila
On Mon, Jul 19, 2021 at 1:00 PM Peter Smith wrote: > > On Mon, Jul 19, 2021 at 4:41 PM Amit Kapila wrote: > > > > > > > > OK. I have implemented this reported [1] potential buffer overrun > > > using the constraining strlcpy, because the GID limitation of 200 > > > bytes is already mentioned in

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-19 Thread Peter Smith
On Mon, Jul 19, 2021 at 4:41 PM Amit Kapila wrote: > > On Mon, Jul 19, 2021 at 9:19 AM Peter Smith wrote: > > > > On Mon, Jul 19, 2021 at 12:43 PM Amit Kapila > > wrote: > > > > > > On Mon, Jul 19, 2021 at 1:55 AM Tom Lane wrote: > > > > > > > > Amit Kapila writes: > > > > > Pushed. > > > >

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-19 Thread Amit Kapila
On Mon, Jul 19, 2021 at 9:19 AM Peter Smith wrote: > > On Mon, Jul 19, 2021 at 12:43 PM Amit Kapila wrote: > > > > On Mon, Jul 19, 2021 at 1:55 AM Tom Lane wrote: > > > > > > Amit Kapila writes: > > > > Pushed. > > > > > > I think you'd be way better off making the gid fields be "char *" > > >

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-18 Thread Greg Nancarrow
On Wed, Jul 14, 2021 at 6:33 PM Peter Smith wrote: > > Please find attached the latest patch set v97* > I couldn't spot spot any significant issues in the v97-0001 patch, but do have the following trivial feedback comments: (1) doc/src/sgml/protocol.sgml Suggestion: BEFORE: + contains a

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-18 Thread Peter Smith
On Mon, Jul 19, 2021 at 12:43 PM Amit Kapila wrote: > > On Mon, Jul 19, 2021 at 1:55 AM Tom Lane wrote: > > > > Amit Kapila writes: > > > Pushed. > > > > I think you'd be way better off making the gid fields be "char *" > > and pstrdup'ing the result of pq_getmsgstring. Another possibility > >

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-18 Thread Amit Kapila
On Mon, Jul 19, 2021 at 1:55 AM Tom Lane wrote: > > Amit Kapila writes: > > Pushed. > > I think you'd be way better off making the gid fields be "char *" > and pstrdup'ing the result of pq_getmsgstring. Another possibility > perhaps is to use strlcpy, but I'd only go that way if it's important

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-18 Thread Tom Lane
Amit Kapila writes: > Pushed. Coverity thinks this has security issues, and I agree. /srv/coverity/git/pgsql-git/postgresql/src/backend/replication/logical/proto.c: 144 in logicalrep_read_begin_prepare() 143 /* read gid (copy it into a pre-allocated buffer) */ >>> CID 1487517:

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-16 Thread vignesh C
On Wed, Jul 14, 2021 at 2:03 PM Peter Smith wrote: > > On Wed, Jul 14, 2021 at 4:23 PM Amit Kapila wrote: > > > > On Mon, Jul 12, 2021 at 9:14 AM Peter Smith wrote: > > > > > > On Sun, Jul 11, 2021 at 8:20 PM Amit Kapila > > > wrote: > > > > > > > > On Fri, Jul 9, 2021 at 4:43 AM Peter Smith

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-14 Thread Peter Smith
On Wed, Jul 14, 2021 at 4:23 PM Amit Kapila wrote: > > On Mon, Jul 12, 2021 at 9:14 AM Peter Smith wrote: > > > > On Sun, Jul 11, 2021 at 8:20 PM Amit Kapila wrote: > > > > > > On Fri, Jul 9, 2021 at 4:43 AM Peter Smith wrote: > > > > > > > > > The patch looks good to me, I don't have any

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-14 Thread Amit Kapila
On Mon, Jul 12, 2021 at 9:14 AM Peter Smith wrote: > > On Sun, Jul 11, 2021 at 8:20 PM Amit Kapila wrote: > > > > On Fri, Jul 9, 2021 at 4:43 AM Peter Smith wrote: > > > > > > > The patch looks good to me, I don't have any comments. > > > > > > I tried the v95-0001 patch. > > > > > > - The

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-11 Thread Peter Smith
On Sun, Jul 11, 2021 at 8:20 PM Amit Kapila wrote: > > On Fri, Jul 9, 2021 at 4:43 AM Peter Smith wrote: > > > > > The patch looks good to me, I don't have any comments. > > > > I tried the v95-0001 patch. > > > > - The patch applied cleanly and all build / testing was OK. > > - The

RE: [HACKERS] logical decoding of two-phase transactions

2021-07-09 Thread tanghy.f...@fujitsu.com
On Friday, July 9, 2021 2:56 PM Ajin Cherian wrote: > > On Fri, Jul 9, 2021 at 9:13 AM Peter Smith wrote: > > > I tried the v95-0001 patch. > > > > - The patch applied cleanly and all build / testing was OK. > > - The documentation also builds OK. > > - I checked all v95-0001 / v93-0001

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-09 Thread Ajin Cherian
On Fri, Jul 9, 2021 at 9:13 AM Peter Smith wrote: > I tried the v95-0001 patch. > > - The patch applied cleanly and all build / testing was OK. > - The documentation also builds OK. > - I checked all v95-0001 / v93-0001 differences and found no problems. > - Furthermore, I noted that v95-0001

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-08 Thread Peter Smith
On Thu, Jul 8, 2021 at 10:08 PM vignesh C wrote: > > On Thu, Jul 8, 2021 at 11:37 AM Amit Kapila wrote: > > > > On Tue, Jul 6, 2021 at 9:58 AM Peter Smith wrote: > > > > > > Please find attached the latest patch set v93* > > > > > > > Thanks, I have gone through the 0001 patch and made a number

Re: [HACKERS] logical decoding of two-phase transactions

2021-07-08 Thread vignesh C
On Thu, Jul 8, 2021 at 11:37 AM Amit Kapila wrote: > > On Tue, Jul 6, 2021 at 9:58 AM Peter Smith wrote: > > > > Please find attached the latest patch set v93* > > > > Thanks, I have gone through the 0001 patch and made a number of > changes. (a) Removed some of the code which was leftover from

RE: [HACKERS] logical decoding of two-phase transactions

2021-07-06 Thread tanghy.f...@fujitsu.com
On Tuesday, July 6, 2021 7:18 PM Ajin Cherian > > thanks for the test! > I hadn't updated the case where sending schema across was the first > change of the transaction as part of the decoding of the > truncate command. In this test case, the schema was sent across > without the stream start,

RE: [HACKERS] logical decoding of two-phase transactions

2021-07-02 Thread tanghy.f...@fujitsu.com
On Thursday, July 1, 2021 11:48 AM Ajin Cherian > > Adding a new patch (0004) to this patch-set that handles skipping of > empty streamed transactions. patch-0003 did not > handle empty streamed transactions. To support this, added a new flag > "sent_stream_start" to PGOutputTxnData. > Also

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-30 Thread Amit Kapila
On Tue, Jun 29, 2021 at 5:31 PM vignesh C wrote: > > On Tue, Jun 29, 2021 at 12:26 PM Amit Kapila wrote: > > > > On Wed, Jun 23, 2021 at 4:10 PM Ajin Cherian wrote: > > > > > > > The first two patches look mostly good to me. I have combined them > > into one and made some minor changes. (a)

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-29 Thread vignesh C
On Tue, Jun 29, 2021 at 12:26 PM Amit Kapila wrote: > > On Wed, Jun 23, 2021 at 4:10 PM Ajin Cherian wrote: > > > > The first two patches look mostly good to me. I have combined them > into one and made some minor changes. (a) Removed opt_two_phase and > related code from repl_gram.y as that is

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-29 Thread Ajin Cherian
On Tue, Jun 29, 2021 at 4:56 PM Amit Kapila wrote: > > On Wed, Jun 23, 2021 at 4:10 PM Ajin Cherian wrote: > > > > The first two patches look mostly good to me. I have combined them > into one and made some minor changes. (a) Removed opt_two_phase and > related code from repl_gram.y as that is

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-29 Thread Amit Kapila
On Wed, Jun 23, 2021 at 4:10 PM Ajin Cherian wrote: > The first two patches look mostly good to me. I have combined them into one and made some minor changes. (a) Removed opt_two_phase and related code from repl_gram.y as that is not required for this version of the patch. (b) made some changes

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-22 Thread vignesh C
On Wed, Jun 23, 2021 at 9:10 AM Ajin Cherian wrote: > > On Tue, Jun 22, 2021 at 3:36 PM Greg Nancarrow wrote: > > > Some minor comments: > > > > (1) > > v88-0002 > > > > doc/src/sgml/logicaldecoding.sgml > > > > "examples shows" is not correct. > > I think there is only ONE example being

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-21 Thread Greg Nancarrow
On Mon, Jun 21, 2021 at 4:37 PM Peter Smith wrote: > > Please find attached the latest patch set v88* > Some minor comments: (1) v88-0002 doc/src/sgml/logicaldecoding.sgml "examples shows" is not correct. I think there is only ONE example being referred to. BEFORE: +The following

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-17 Thread Amit Kapila
On Fri, Jun 18, 2021 at 7:43 AM Peter Smith wrote: > > On Thu, Jun 17, 2021 at 6:22 PM Greg Nancarrow wrote: > > > > For example, in the v86-0001 patch: > > > > +/* > > + * Handle PREPARE message. > > + */ > > +static void > > +apply_handle_prepare(StringInfo s) > > +{ > > +

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-17 Thread vignesh C
On Thu, Jun 17, 2021 at 7:40 PM Ajin Cherian wrote: > > On Wed, Jun 16, 2021 at 9:08 AM Peter Smith wrote: > > > > On Fri, Jun 11, 2021 at 6:34 PM Peter Smith wrote: > > > > > KNOWN ISSUES: This v85 patch was built and tested using yesterday's > > > master, but due to lots of recent activity in

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-17 Thread Peter Smith
On Thu, Jun 17, 2021 at 6:22 PM Greg Nancarrow wrote: > > On Wed, Jun 16, 2021 at 9:08 AM Peter Smith wrote: > > > > > > Please find attached the latest patch set v86* > > > > A couple of comments: > > (1) I think one of my suggested changes was missed (or was that > intentional?): > > BEFORE:

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-17 Thread Greg Nancarrow
On Wed, Jun 16, 2021 at 9:08 AM Peter Smith wrote: > > > Please find attached the latest patch set v86* > A couple of comments: (1) I think one of my suggested changes was missed (or was that intentional?): BEFORE: +The LSN of the commit prepared. AFTER: +The

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-08 Thread Amit Kapila
On Wed, Jun 9, 2021 at 9:58 AM Greg Nancarrow wrote: > > (5) src/backend/access/transam/twophase.c > > Question: > > Is: > > + * do this optimization if we encounter many collisions in GID > > meant to be: > > + * do this optimization if we encounter any collisions in GID > No, it should be fine

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-08 Thread Amit Kapila
On Wed, Jun 9, 2021 at 10:34 AM Ajin Cherian wrote: > > On Tue, Jun 8, 2021 at 4:19 PM Peter Smith wrote: > > > > 3. > > > @@ -432,10 +432,19 @@ CreateInitDecodingContext(const char *plugin, > > > MemoryContextSwitchTo(old_context); > > > > > > /* > > > - * We allow decoding of prepared

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-08 Thread Greg Nancarrow
On Tue, Jun 8, 2021 at 4:12 PM Peter Smith wrote: > > Please find attached the latest patch set v83* > Some feedback for the v83 patch set: v83-0001: (1) doc/src/sgml/protocol.sgml (i) Remove extra space: BEFORE: + The transaction will be decoded and transmitted at AFTER: +

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-08 Thread Peter Smith
On Thu, Jun 3, 2021 at 7:56 PM Amit Kapila wrote: > > On Wed, Jun 2, 2021 at 4:34 AM Peter Smith wrote: > > > > Please find attached the latest patch set v82* > > > > Few comments on 0001: > > 1. > + /* > + * BeginTransactionBlock is necessary to balance the

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-07 Thread Greg Nancarrow
On Wed, Jun 2, 2021 at 9:04 AM Peter Smith wrote: > > Please find attached the latest patch set v82* > Some suggested changes to the 0001 patch comments (and note also the typo "doumentation"): diff of before and after follows: 8c8 < built-in logical replication, we need to do the below

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-03 Thread Amit Kapila
On Wed, Jun 2, 2021 at 4:34 AM Peter Smith wrote: > > Please find attached the latest patch set v82* > Few comments on 0001: 1. + /* + * BeginTransactionBlock is necessary to balance the EndTransactionBlock + * called within the PrepareTransactionBlock below. + */ +

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-01 Thread Peter Smith
On Mon, May 31, 2021 at 9:16 PM Amit Kapila wrote: > > On Fri, May 28, 2021 at 11:55 AM Amit Kapila wrote: > > > > One minor comment for 0001. > > * Special case: if when tables were specified but copy_data is > > + * false then it is safe to enable two_phase up-front because > > + * those

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-31 Thread Amit Kapila
On Fri, May 28, 2021 at 11:55 AM Amit Kapila wrote: > > One minor comment for 0001. > * Special case: if when tables were specified but copy_data is > + * false then it is safe to enable two_phase up-front because > + * those tables are already initially READY state. Note, if > + * the

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-28 Thread Amit Kapila
On Thu, May 27, 2021 at 8:05 AM Ajin Cherian wrote: > > Thanks for confirmation. The problem seemed to be as you reported a > table not closed when a transaction was committed. > This seems to be because the function UpdateTwoPhaseState was > committing a transaction inside the function when the

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-28 Thread vignesh C
On Fri, May 28, 2021 at 9:14 AM Ajin Cherian wrote: > > On Wed, May 26, 2021 at 6:53 PM vignesh C wrote: > > > > On Tue, May 25, 2021 at 8:54 AM Ajin Cherian wrote: > > > > > > On Fri, May 21, 2021 at 6:43 PM Peter Smith wrote: > > > > > > > Fixed in v77-0001, v77-0002 > > > > > > Attaching a

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-26 Thread Ajin Cherian
On Thu, May 27, 2021 at 11:20 AM tanghy.f...@fujitsu.com wrote: > > On Wed, May 26, 2021 10:13 PM Ajin Cherian wrote: > > > > I've attached a patch that fixes this issue. Do test and confirm. > > > > Thanks for your patch. > I have tested and confirmed that the issue I reported has been fixed.

RE: [HACKERS] logical decoding of two-phase transactions

2021-05-26 Thread tanghy.f...@fujitsu.com
On Wed, May 26, 2021 10:13 PM Ajin Cherian wrote: > > I've attached a patch that fixes this issue. Do test and confirm. > Thanks for your patch. I have tested and confirmed that the issue I reported has been fixed. Regards Tang

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-26 Thread vignesh C
On Tue, May 25, 2021 at 8:54 AM Ajin Cherian wrote: > > On Fri, May 21, 2021 at 6:43 PM Peter Smith wrote: > > > Fixed in v77-0001, v77-0002 > > Attaching a new patch-set that rebases the patch, addresses review > comments from Peter as well as a test failure reported by Tang. I've > also added

RE: [HACKERS] logical decoding of two-phase transactions

2021-05-25 Thread tanghy.f...@fujitsu.com
> > 13. > > @@ -507,7 +558,16 @@ CreateSubscription(CreateSubscriptionStmt *stmt, > > bool isTopLevel) > > { > > Assert(slotname); > > > > - walrcv_create_slot(wrconn, slotname, false, > > + /* > > + * Even if two_phase is set, don't create the slot with > > + * two-phase enabled. Will enable

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-21 Thread Peter Smith
On Tue, May 18, 2021 at 9:32 PM Amit Kapila wrote: > > On Thu, May 13, 2021 at 3:20 PM Peter Smith wrote: > > > > Please find attached the latest patch set v75* > > > > Review comments for v75-0001-Add-support-for-prepared-transactions-to-built-i: >

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-18 Thread Amit Kapila
On Thu, May 13, 2021 at 3:20 PM Peter Smith wrote: > > Please find attached the latest patch set v75* > Review comments for v75-0001-Add-support-for-prepared-transactions-to-built-i: === 1. - CREATE_REPLICATION_SLOT

RE: [HACKERS] logical decoding of two-phase transactions

2021-05-18 Thread tanghy.f...@fujitsu.com
Hi Ajin >The above patch had some changes missing which resulted in some tap >tests failing. Sending an updated patchset. Keeping the patchset >version the same. Thanks for your patch. I see a problem about Segmentation fault when using it. Please take a look at this. The steps to reproduce

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-18 Thread Peter Smith
On Sun, May 16, 2021 at 12:07 AM Ajin Cherian wrote: > > On Thu, May 13, 2021 at 7:50 PM Peter Smith wrote: > > > > Please find attached the latest patch set v75* > > > > Differences from v74* are: > > > > * Rebased to HEAD @ today. > > > > * v75 also addresses some of the feedback comments from

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-17 Thread vignesh C
On Mon, May 17, 2021 at 6:10 PM Ajin Cherian wrote: > > The above patch had some changes missing which resulted in some tap > tests failing. Sending an updated patchset. Keeping the patchset > version the same. Thanks for the updated patch, the updated patch fixes the tap test failures.

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-13 Thread Peter Smith
On Mon, May 10, 2021 at 1:31 PM vignesh C wrote: > > On Thu, Apr 29, 2021 at 2:23 PM Peter Smith wrote: > > > > Please find attached the latest patch set v74* > > > > Differences from v73* are: > > > > * Rebased to HEAD @ 2 days ago. > > > > * v74 addresses most of the feedback comments from

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-10 Thread vignesh C
On Mon, May 10, 2021 at 10:51 AM Peter Smith wrote: > > On Mon, May 10, 2021 at 1:31 PM vignesh C wrote: > > > > > 4) Should we change this to "The end LSN of the prepared transaction" > > just to avoid any confusion of it meaning commit/rollback. > > + > > +Int64 > > + > > +The

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-09 Thread Peter Smith
On Mon, May 10, 2021 at 1:31 PM vignesh C wrote: > > 4) Should we change this to "The end LSN of the prepared transaction" > just to avoid any confusion of it meaning commit/rollback. > + > +Int64 > + > +The end LSN of the transaction. > + > + > Can you please provide more

Re: [HACKERS] logical decoding of two-phase transactions

2021-05-09 Thread vignesh C
On Thu, Apr 29, 2021 at 2:23 PM Peter Smith wrote: > > Please find attached the latest patch set v74* > > Differences from v73* are: > > * Rebased to HEAD @ 2 days ago. > > * v74 addresses most of the feedback comments from Vignesh posts [1][2][3]. > Thanks for the updated patch. Few comments:

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-29 Thread Peter Smith
On Tue, Apr 27, 2021 at 6:17 PM vignesh C wrote: > > On Wed, Apr 21, 2021 at 12:13 PM Peter Smith wrote: > > > > On Tue, Apr 20, 2021 at 3:45 PM Peter Smith wrote: > > > > > > Please find attached the latest patch set v73`* > > > > > > Differences from v72* are: > > > > > > * Rebased to HEAD @

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-29 Thread Peter Smith
On Tue, Apr 27, 2021 at 1:41 PM vignesh C wrote: > > On Wed, Apr 21, 2021 at 12:13 PM Peter Smith wrote: > > > > On Tue, Apr 20, 2021 at 3:45 PM Peter Smith wrote: > > > > > > Please find attached the latest patch set v73`* > > > > > > Differences from v72* are: > > > > > > * Rebased to HEAD @

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-29 Thread Peter Smith
On Mon, Apr 26, 2021 at 9:22 PM vignesh C wrote: > > On Wed, Apr 21, 2021 at 12:13 PM Peter Smith wrote: > > > > On Tue, Apr 20, 2021 at 3:45 PM Peter Smith wrote: > > > > > > Please find attached the latest patch set v73`* > > > > > > Differences from v72* are: > > > > > > * Rebased to HEAD @

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-28 Thread Ajin Cherian
Modified pgbench's "tpcb-like" builtin query as below to do two-phase commits and then ran a 4 cascade replication setup. "BEGIN;\n" "UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;\n" "SELECT abalance FROM pgbench_accounts WHERE aid = :aid;\n"

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-27 Thread vignesh C
On Wed, Apr 21, 2021 at 12:13 PM Peter Smith wrote: > > On Tue, Apr 20, 2021 at 3:45 PM Peter Smith wrote: > > > > Please find attached the latest patch set v73`* > > > > Differences from v72* are: > > > > * Rebased to HEAD @ today (required because v72-0001 no longer applied > > cleanly) > > >

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-26 Thread vignesh C
On Wed, Apr 21, 2021 at 12:13 PM Peter Smith wrote: > > On Tue, Apr 20, 2021 at 3:45 PM Peter Smith wrote: > > > > Please find attached the latest patch set v73`* > > > > Differences from v72* are: > > > > * Rebased to HEAD @ today (required because v72-0001 no longer applied > > cleanly) > > >

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-26 Thread vignesh C
On Wed, Apr 21, 2021 at 12:13 PM Peter Smith wrote: > > On Tue, Apr 20, 2021 at 3:45 PM Peter Smith wrote: > > > > Please find attached the latest patch set v73`* > > > > Differences from v72* are: > > > > * Rebased to HEAD @ today (required because v72-0001 no longer applied > > cleanly) > > >

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-09 Thread Peter Smith
On Fri, Apr 9, 2021 at 6:40 PM Amit Kapila wrote: > > On Fri, Apr 9, 2021 at 12:33 PM Peter Smith wrote: > > > > On Mon, Dec 14, 2020 at 8:27 PM Amit Kapila wrote: > > > > > > 2. > > > + /* > > > + * Flags are determined from the state of the transaction. We know we > > > + * always get PREPARE

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-09 Thread Amit Kapila
On Fri, Apr 9, 2021 at 12:33 PM Peter Smith wrote: > > On Mon, Dec 14, 2020 at 8:27 PM Amit Kapila wrote: > > > > 2. > > + /* > > + * Flags are determined from the state of the transaction. We know we > > + * always get PREPARE first and then [COMMIT|ROLLBACK] PREPARED, so if > > + * it's

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-09 Thread Peter Smith
On Mon, Dec 14, 2020 at 8:27 PM Amit Kapila wrote: > > 2. > + /* > + * Flags are determined from the state of the transaction. We know we > + * always get PREPARE first and then [COMMIT|ROLLBACK] PREPARED, so if > + * it's already marked as committed then it has to be COMMIT PREPARED (and > + *

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-01 Thread Peter Smith
On Thu, Apr 1, 2021 at 4:58 PM Ajin Cherian wrote: > > > > On Thu, Apr 1, 2021 at 2:29 PM Amit Kapila wrote: >> >> On Tue, Mar 30, 2021 at 5:34 AM Peter Smith wrote: >> > >> > Please find attached the latest patch set v68* >> > >> >> I think this patch is in much better shape than it was few

Re: [HACKERS] logical decoding of two-phase transactions

2021-04-01 Thread vignesh C
On Thu, Apr 1, 2021 at 8:59 AM Amit Kapila wrote: > > On Tue, Mar 30, 2021 at 5:34 AM Peter Smith wrote: > > > > Please find attached the latest patch set v68* > > > > I think this patch is in much better shape than it was few versions > earlier but I feel still some more work and testing is

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-31 Thread Ajin Cherian
On Thu, Apr 1, 2021 at 2:29 PM Amit Kapila wrote: > On Tue, Mar 30, 2021 at 5:34 AM Peter Smith wrote: > > > > Please find attached the latest patch set v68* > > > > I think this patch is in much better shape than it was few versions > earlier but I feel still some more work and testing is

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-31 Thread Amit Kapila
On Tue, Mar 30, 2021 at 5:34 AM Peter Smith wrote: > > Please find attached the latest patch set v68* > I think this patch is in much better shape than it was few versions earlier but I feel still some more work and testing is required. We can try to make it work with the streaming option and do

Re: [HACKERS] logical decoding of two-phase transactions

2021-03-29 Thread vignesh C
On Tue, Mar 30, 2021 at 5:34 AM Peter Smith wrote: > > Please find attached the latest patch set v68* > > Differences from v67* are: > > * Rebased to HEAD @ today. > > * v68 fixes an issue reported by Vignesh [1] where a scenario was > found which still was able to cause a generated GID clash.

  1   2   3   4   5   >