Re: [HACKERS] logical decoding of two-phase transactions

2017-10-29 Thread Craig Ringer
On 28 October 2017 at 03:53, Sokolov Yura wrote: > On 2017-10-26 22:01, Sokolov Yura wrote: > Small improvement compared to v7: > - twophase_gid is written with alignment padding in the XactLogCommitRecord > and XactLogAbortRecord. I think Nikhils has done some significant work on this patch. Ho

Re: [HACKERS] logical decoding of two-phase transactions

2017-10-27 Thread Sokolov Yura
On 2017-10-26 22:01, Sokolov Yura wrote: On 2017-09-27 14:46, Stas Kelvich wrote: On 7 Sep 2017, at 18:58, Nikhil Sontakke wrote: Hi, FYI all, wanted to mention that I am working on an updated version of the latest patch that I plan to submit to a later CF. Cool! So what kind of architec

Re: [HACKERS] logical decoding of two-phase transactions

2017-10-26 Thread Sokolov Yura
On 2017-09-27 14:46, Stas Kelvich wrote: On 7 Sep 2017, at 18:58, Nikhil Sontakke wrote: Hi, FYI all, wanted to mention that I am working on an updated version of the latest patch that I plan to submit to a later CF. Cool! So what kind of architecture do you have in mind? Same way as is i

Re: [HACKERS] logical decoding of two-phase transactions

2017-09-27 Thread Stas Kelvich
> On 7 Sep 2017, at 18:58, Nikhil Sontakke wrote: > > Hi, > > FYI all, wanted to mention that I am working on an updated version of > the latest patch that I plan to submit to a later CF. > Cool! So what kind of architecture do you have in mind? Same way as is it was implemented before? As

Re: [HACKERS] logical decoding of two-phase transactions

2017-09-07 Thread Nikhil Sontakke
Hi, FYI all, wanted to mention that I am working on an updated version of the latest patch that I plan to submit to a later CF. Regards, Nikhils On 14 May 2017 at 04:02, Dmitry Dolgov <9erthali...@gmail.com> wrote: > On 13 May 2017 at 22:22, Tom Lane wrote: >> >> Apparently you are not testing

Re: [HACKERS] Logical decoding on standby

2017-06-26 Thread Craig Ringer
On 21 June 2017 at 17:30, sanyam jain wrote: > Hi, > After changing > sendTimeLineIsHistoric = state->currTLI == ThisTimeLineID; > to > sendTimeLineIsHistoric = state->currTLI != ThisTimeLineID; > > I was facing another issue. > On promotion of a cascaded server ThisTimeLineID in the standby serve

Re: [HACKERS] Logical decoding on standby

2017-06-26 Thread Craig Ringer
On 21 June 2017 at 13:28, sanyam jain wrote: > Hi, > > In this patch in walsender.c sendTimeLineIsHistoric is set to true when > current and ThisTimeLineID are equal. > > sendTimeLineIsHistoric = state->currTLI == ThisTimeLineID; > > > Shouldn't sendTimeLineIsHistoric is true when state->currTLI i

Re: [HACKERS] Logical decoding on standby

2017-06-26 Thread sanyam jain
Hi, >After changing >sendTimeLineIsHistoric = state->currTLI == ThisTimeLineID; >to >sendTimeLineIsHistoric = state->currTLI != ThisTimeLineID; > >I was facing another issue. >On promotion of a cascaded server ThisTimeLineID in the standby server having >>logical slot becomes 0. >Then i added a fu

Re: [HACKERS] Logical decoding on standby

2017-06-21 Thread sanyam jain
Hi, After changing sendTimeLineIsHistoric = state->currTLI == ThisTimeLineID; to sendTimeLineIsHistoric = state->currTLI != ThisTimeLineID; I was facing another issue. On promotion of a cascaded server ThisTimeLineID in the standby server having logical slot becomes 0. Then i added a function cal

Re: [HACKERS] Logical decoding on standby

2017-06-20 Thread sanyam jain
Sent: Wednesday, April 5, 2017 3:25:50 PM To: Andres Freund Cc: Craig Ringer; Simon Riggs; Thom Brown; Michael Paquier; Petr Jelinek; PostgreSQL Hackers Subject: Re: [HACKERS] Logical decoding on standby On Wed, Apr 5, 2017 at 10:32 AM, Andres Freund wrote: > On 2017-04-05 17:18:24 +0800,

Re: [HACKERS] logical decoding of two-phase transactions

2017-05-13 Thread Dmitry Dolgov
On 13 May 2017 at 22:22, Tom Lane wrote: > > Apparently you are not testing against current HEAD. That's been there > since d10c626de (a whole two days now ;-)) Indeed, I was working on a more than two-day old antiquity. Unfortunately, it's even more complicated to apply this patch against the c

Re: [HACKERS] logical decoding of two-phase transactions

2017-05-13 Thread Tom Lane
Dmitry Dolgov <9erthali...@gmail.com> writes: > Just a note about this patch. Of course time flies by and it needs rebase, > but also there are few failing tests right now: > ERROR: function pg_wal_lsn_diff(pg_lsn, unknown) does not exist Apparently you are not testing against current HEAD. Tha

Re: [HACKERS] logical decoding of two-phase transactions

2017-05-13 Thread Dmitry Dolgov
Hi > On 4 April 2017 at 19:13, Masahiko Sawada wrote: > > Other than that issue current patch still could not pass 'make check' > test of contrib/test_decoding. Just a note about this patch. Of course time flies by and it needs rebase, but also there are few failing tests right now: * one that

Re: [HACKERS] Logical decoding truncate

2017-05-11 Thread Euler Taveira
2017-05-11 4:23 GMT-03:00 Friedrich, Steffen < steffen.friedr...@dieboldnixdorf.com>: > I am writing a logical decoding output plugin decoding WAL to SQL which is > finally applied to target database. > > > > Is it possible to decode a TRUNCATE statement and the tables involved? > > Yes, use event

[HACKERS] Logical decoding truncate

2017-05-11 Thread Friedrich, Steffen
Hi, I am writing a logical decoding output plugin decoding WAL to SQL which is finally applied to target database. Is it possible to decode a TRUNCATE statement and the tables involved? Assuming the SQL statement "TRUNCATE x, y;", I am interested in decoding the operation TRUNCATE and the corr

Re: [HACKERS] Logical decoding on standby

2017-04-06 Thread Craig Ringer
On 5 April 2017 at 23:25, Robert Haas wrote: > On Wed, Apr 5, 2017 at 10:32 AM, Andres Freund wrote: >> On 2017-04-05 17:18:24 +0800, Craig Ringer wrote: >>> On 5 April 2017 at 04:19, Andres Freund wrote: >>> > On 2017-04-04 22:32:40 +0800, Craig Ringer wrote: >>> >> I'm much happier with this.

Re: [HACKERS] Logical decoding on standby

2017-04-05 Thread Robert Haas
On Wed, Apr 5, 2017 at 10:32 AM, Andres Freund wrote: > On 2017-04-05 17:18:24 +0800, Craig Ringer wrote: >> On 5 April 2017 at 04:19, Andres Freund wrote: >> > On 2017-04-04 22:32:40 +0800, Craig Ringer wrote: >> >> I'm much happier with this. I'm still fixing some issues in the tests >> >> for

Re: [HACKERS] Logical decoding on standby

2017-04-05 Thread Andres Freund
On 2017-04-05 17:18:24 +0800, Craig Ringer wrote: > On 5 April 2017 at 04:19, Andres Freund wrote: > > On 2017-04-04 22:32:40 +0800, Craig Ringer wrote: > >> I'm much happier with this. I'm still fixing some issues in the tests > >> for 03 and tidying them up, but 03 should allow 01 and 02 to be >

Re: [HACKERS] Logical decoding on standby

2017-04-05 Thread Craig Ringer
On 5 April 2017 at 04:19, Andres Freund wrote: > On 2017-04-04 22:32:40 +0800, Craig Ringer wrote: >> I'm much happier with this. I'm still fixing some issues in the tests >> for 03 and tidying them up, but 03 should allow 01 and 02 to be >> reviewed in their proper context now. > > To me this ver

Re: [HACKERS] Logical decoding on standby

2017-04-04 Thread Andres Freund
On 2017-04-04 22:32:40 +0800, Craig Ringer wrote: > I'm much happier with this. I'm still fixing some issues in the tests > for 03 and tidying them up, but 03 should allow 01 and 02 to be > reviewed in their proper context now. To me this very clearly is too late for v10, and now should be moved t

Re: [HACKERS] logical decoding of two-phase transactions

2017-04-04 Thread Andres Freund
On 2017-04-04 13:06:13 +0300, Stas Kelvich wrote: > That is just argument against Andres concern that prepared transaction > is able to deadlock with decoding process — at least no such cases in > regression tests. There's few longer / adverse xacts, that doesn't say much. > And that concern is

Re: [HACKERS] logical decoding of two-phase transactions

2017-04-04 Thread Masahiko Sawada
On Tue, Apr 4, 2017 at 7:06 PM, Stas Kelvich wrote: > >> On 4 Apr 2017, at 04:23, Masahiko Sawada wrote: >> >> >> I reviewed this patch but when I tried to build contrib/test_decoding >> I got the following error. >> > > Thanks! > > Yes, seems that 18ce3a4a changed ProcessUtility_hook signature.

Re: [HACKERS] Logical decoding on standby

2017-04-04 Thread Craig Ringer
On 4 April 2017 at 22:32, Craig Ringer wrote: > Hi all > > Here's the final set of three patches on top of what's already committed. > > The first is catalog_xmin logging, which is unchanged from the prior post. > > The 2nd is support for conflict with recovery, with changes that > should address

Re: [HACKERS] Logical decoding on standby

2017-04-04 Thread Craig Ringer
Hi all Here's the final set of three patches on top of what's already committed. The first is catalog_xmin logging, which is unchanged from the prior post. The 2nd is support for conflict with recovery, with changes that should address Andres's concerns there. The 3rd actually enables decoding

Re: [HACKERS] logical decoding of two-phase transactions

2017-04-04 Thread Stas Kelvich
> On 4 Apr 2017, at 04:23, Masahiko Sawada wrote: > > > I reviewed this patch but when I tried to build contrib/test_decoding > I got the following error. > Thanks! Yes, seems that 18ce3a4a changed ProcessUtility_hook signature. Updated. > There are still some unnecessary code in v5 patch.

Re: [HACKERS] logical decoding of two-phase transactions

2017-04-03 Thread Masahiko Sawada
On Thu, Mar 30, 2017 at 12:55 AM, Stas Kelvich wrote: > >> On 28 Mar 2017, at 18:08, Andres Freund wrote: >> >> On 2017-03-28 15:55:15 +0100, Simon Riggs wrote: >>> >>> >>> That assertion is obviously false... the plugin can resolve this in >>> various ways, if we allow it. >> >> Handling it by b

Re: [HACKERS] Logical decoding on standby

2017-04-03 Thread Craig Ringer
On 3 April 2017 at 15:27, Craig Ringer wrote: > On 3 April 2017 at 13:46, Craig Ringer wrote: > >> OK, updated catalog_xmin logging patch attached. > > Ahem, that should be v5. ... and here's v6, which returns to the separate xl_xact_catalog_xmin_advance approach. pgintented. This is what I fa

Re: [HACKERS] Logical decoding on standby

2017-04-03 Thread Craig Ringer
On 3 April 2017 at 13:46, Craig Ringer wrote: > OK, updated catalog_xmin logging patch attached. Ahem, that should be v5. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services From 7f742f582e1f6f8f23c4e9d78cd0298180e5387c Mon

Re: [HACKERS] Logical decoding on standby

2017-04-02 Thread Craig Ringer
On 31 March 2017 at 12:49, Craig Ringer wrote: > On 31 March 2017 at 01:16, Andres Freund wrote: >>> @@ -9633,6 +9643,12 @@ xlog_redo(XLogReaderState *record) >>> SetTransactionIdLimit(checkPoint.oldestXid, >>> checkPoint.oldestXidDB); >>> >>> /* >>> + *

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Craig Ringer
On 31 March 2017 at 12:49, Craig Ringer wrote: > On 31 March 2017 at 01:16, Andres Freund wrote: >> The comment and code don't quite square to me - it's far from obvious >> that LogStandbySnapshot does something like that. I'd even say it's a >> bad idea to have it do that. > > So you prefer the

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Craig Ringer
On 31 March 2017 at 01:16, Andres Freund wrote: >> @@ -9633,6 +9643,12 @@ xlog_redo(XLogReaderState *record) >> SetTransactionIdLimit(checkPoint.oldestXid, >> checkPoint.oldestXidDB); >> >> /* >> + * There can be no concurrent writers to oldestCatalogXmin

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Craig Ringer
On 31 March 2017 at 01:16, Andres Freund wrote: > On 2017-03-29 08:01:34 +0800, Craig Ringer wrote: >> On 28 March 2017 at 23:22, Andres Freund wrote: >> >> >> --- a/doc/src/sgml/protocol.sgml >> >> +++ b/doc/src/sgml/protocol.sgml >> >> @@ -2034,6 +2034,8 @@ The commands accepted in walsender mo

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Andres Freund
On 2017-03-30 19:40:08 +0100, Simon Riggs wrote: > On 30 March 2017 at 18:16, Andres Freund wrote: > > >> /* > >> * Each page of XLOG file has a header like this: > >> */ > >> -#define XLOG_PAGE_MAGIC 0xD097 /* can be used as WAL version > >> indicator */ > >> +#define XLOG_PAGE_MAGIC

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Simon Riggs
On 30 March 2017 at 18:16, Andres Freund wrote: >> /* >> * Each page of XLOG file has a header like this: >> */ >> -#define XLOG_PAGE_MAGIC 0xD097 /* can be used as WAL version >> indicator */ >> +#define XLOG_PAGE_MAGIC 0xD100 /* can be used as WAL version >> indicator */ > >

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Andres Freund
On 2017-03-29 08:01:34 +0800, Craig Ringer wrote: > On 28 March 2017 at 23:22, Andres Freund wrote: > > >> --- a/doc/src/sgml/protocol.sgml > >> +++ b/doc/src/sgml/protocol.sgml > >> @@ -2034,6 +2034,8 @@ The commands accepted in walsender mode are: > >> > >>Drops a replication slo

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Andres Freund
> @@ -9633,6 +9643,12 @@ xlog_redo(XLogReaderState *record) > SetTransactionIdLimit(checkPoint.oldestXid, > checkPoint.oldestXidDB); > > /* > + * There can be no concurrent writers to oldestCatalogXmin > during > + * recovery, so no need to

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Simon Riggs
On 30 March 2017 at 15:27, Andres Freund wrote: > On 2017-03-30 15:26:02 +0100, Simon Riggs wrote: >> On 30 March 2017 at 09:07, Craig Ringer wrote: >> >> > Attached. >> >> * Cleaned up in 3 places >> * Added code for faked up RunningTransactions in xlog.c >> * Ensure catalog_xmin doesn't go back

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Andres Freund
On 2017-03-30 15:26:02 +0100, Simon Riggs wrote: > On 30 March 2017 at 09:07, Craig Ringer wrote: > > > Attached. > > * Cleaned up in 3 places > * Added code for faked up RunningTransactions in xlog.c > * Ensure catalog_xmin doesn't go backwards > > All else looks good. Comments before commit?

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Simon Riggs
On 30 March 2017 at 09:07, Craig Ringer wrote: > Attached. * Cleaned up in 3 places * Added code for faked up RunningTransactions in xlog.c * Ensure catalog_xmin doesn't go backwards All else looks good. Comments before commit? -- Simon Riggshttp://www.2ndQuadrant.com/ Postgre

Re: [HACKERS] Logical decoding on standby

2017-03-30 Thread Craig Ringer
On 30 March 2017 at 11:34, Craig Ringer wrote: > On 29 March 2017 at 23:13, Simon Riggs wrote: >> On 29 March 2017 at 10:17, Craig Ringer wrote: >>> On 29 March 2017 at 16:44, Craig Ringer wrote: >>> * Split oldestCatalogXmin tracking into separate patch >>> >>> Regarding this, Simon raise

Re: [HACKERS] Logical decoding on standby

2017-03-29 Thread Craig Ringer
On 29 March 2017 at 23:13, Simon Riggs wrote: > On 29 March 2017 at 10:17, Craig Ringer wrote: >> On 29 March 2017 at 16:44, Craig Ringer wrote: >> >>> * Split oldestCatalogXmin tracking into separate patch >> >> Regarding this, Simon raised concerns about xlog volume here. >> >> It's pretty neg

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-29 Thread Stas Kelvich
> On 28 Mar 2017, at 18:08, Andres Freund wrote: > > On 2017-03-28 15:55:15 +0100, Simon Riggs wrote: >> >> >> That assertion is obviously false... the plugin can resolve this in >> various ways, if we allow it. > > Handling it by breaking replication isn't handling it (e.g. timeouts in > dec

Re: [HACKERS] Logical decoding on standby

2017-03-29 Thread Simon Riggs
On 29 March 2017 at 10:17, Craig Ringer wrote: > On 29 March 2017 at 16:44, Craig Ringer wrote: > >> * Split oldestCatalogXmin tracking into separate patch > > Regarding this, Simon raised concerns about xlog volume here. > > It's pretty negligible. > > We only write a new record when a vacuum ru

Re: [HACKERS] Logical decoding on standby

2017-03-29 Thread Craig Ringer
On 29 March 2017 at 16:44, Craig Ringer wrote: > * Split oldestCatalogXmin tracking into separate patch Regarding this, Simon raised concerns about xlog volume here. It's pretty negligible. We only write a new record when a vacuum runs after catalog_xmin advances on the slot with the currently

Re: [HACKERS] Logical decoding on standby

2017-03-29 Thread Craig Ringer
On 29 March 2017 at 08:11, Craig Ringer wrote: > On 29 March 2017 at 08:01, Craig Ringer wrote: > >> I just notice that I failed to remove the docs changes regarding >> dropping slots becoming db-specific, so I'll post a follow-up for that >> in a sec. > > Attached. ... and here's the next in th

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-28 Thread Craig Ringer
On 28 Mar. 2017 23:08, "Andres Freund" wrote: > > > >> I don't think its for us to say what the plugin is allowed to do. We > > >> decided on a plugin architecture, so we have to trust that the plugin > > >> author resolves the issues. We can document them so those choices are > > >> clear. > > >

Re: [HACKERS] Logical decoding on standby

2017-03-28 Thread Craig Ringer
On 29 March 2017 at 08:01, Craig Ringer wrote: > I just notice that I failed to remove the docs changes regarding > dropping slots becoming db-specific, so I'll post a follow-up for that > in a sec. Attached. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Developmen

Re: [HACKERS] Logical decoding on standby

2017-03-28 Thread Craig Ringer
On 28 March 2017 at 23:22, Andres Freund wrote: >> --- a/doc/src/sgml/protocol.sgml >> +++ b/doc/src/sgml/protocol.sgml >> @@ -2034,6 +2034,8 @@ The commands accepted in walsender mode are: >> >>Drops a replication slot, freeing any reserved server-side resources. >> If >>

Re: [HACKERS] Logical decoding on standby

2017-03-28 Thread Andres Freund
Hi, On 2017-03-27 16:03:48 +0800, Craig Ringer wrote: > On 27 March 2017 at 14:08, Craig Ringer wrote: > > > So this patch makes ReplicationSlotAcquire check that the slot > > database matches the current database and refuse to acquire the slot > > if it does not. > > New patch attached that dr

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-28 Thread Andres Freund
On 2017-03-28 15:55:15 +0100, Simon Riggs wrote: > On 28 March 2017 at 15:38, Andres Freund wrote: > > On 2017-03-28 15:32:49 +0100, Simon Riggs wrote: > >> On 28 March 2017 at 03:53, Craig Ringer wrote: > >> > On 28 March 2017 at 09:25, Andres Freund wrote: > >> > > >> >> If you actually need s

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-28 Thread Simon Riggs
On 28 March 2017 at 15:38, Andres Freund wrote: > On 2017-03-28 15:32:49 +0100, Simon Riggs wrote: >> On 28 March 2017 at 03:53, Craig Ringer wrote: >> > On 28 March 2017 at 09:25, Andres Freund wrote: >> > >> >> If you actually need separate decoding of 2PC, then you want to wait for >> >> the

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-28 Thread Andres Freund
On 2017-03-28 15:32:49 +0100, Simon Riggs wrote: > On 28 March 2017 at 03:53, Craig Ringer wrote: > > On 28 March 2017 at 09:25, Andres Freund wrote: > > > >> If you actually need separate decoding of 2PC, then you want to wait for > >> the PREPARE to be replicated. If that replication has to wa

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-28 Thread Simon Riggs
On 28 March 2017 at 03:53, Craig Ringer wrote: > On 28 March 2017 at 09:25, Andres Freund wrote: > >> If you actually need separate decoding of 2PC, then you want to wait for >> the PREPARE to be replicated. If that replication has to wait for the >> to-be-replicated prepared transaction to comm

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Craig Ringer
On 28 March 2017 at 10:53, Craig Ringer wrote: > However, even once I add an option to force decoding of 2pc xacts with > catalog changes to test_decoding, I cannot reproduce the expected > locking issues so far. See tests in attached updated version, in > contrib/test_decoding/sql/prepare.sql .

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Craig Ringer
On 28 March 2017 at 09:25, Andres Freund wrote: > If you actually need separate decoding of 2PC, then you want to wait for > the PREPARE to be replicated. If that replication has to wait for the > to-be-replicated prepared transaction to commit prepared, and commit > prepare will only happen onc

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Andres Freund
On 2017-03-28 03:30:28 +0100, Simon Riggs wrote: > On 28 March 2017 at 02:25, Andres Freund wrote: > > On 2017-03-28 04:12:41 +0300, Stas Kelvich wrote: > >> > >> > On 28 Mar 2017, at 00:25, Andres Freund wrote: > >> > > >> > Hi, > >> > > >> > On 2017-03-28 00:19:29 +0300, Stas Kelvich wrote: > >

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Simon Riggs
On 28 March 2017 at 02:25, Andres Freund wrote: > On 2017-03-28 04:12:41 +0300, Stas Kelvich wrote: >> >> > On 28 Mar 2017, at 00:25, Andres Freund wrote: >> > >> > Hi, >> > >> > On 2017-03-28 00:19:29 +0300, Stas Kelvich wrote: >> >> Ok, here it is. >> > >> > On a very quick skim, this doesn't s

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Andres Freund
On 2017-03-28 04:12:41 +0300, Stas Kelvich wrote: > > > On 28 Mar 2017, at 00:25, Andres Freund wrote: > > > > Hi, > > > > On 2017-03-28 00:19:29 +0300, Stas Kelvich wrote: > >> Ok, here it is. > > > > On a very quick skim, this doesn't seem to solve the issues around > > deadlocks of prepared

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Stas Kelvich
> On 28 Mar 2017, at 00:25, Andres Freund wrote: > > Hi, > > On 2017-03-28 00:19:29 +0300, Stas Kelvich wrote: >> Ok, here it is. > > On a very quick skim, this doesn't seem to solve the issues around > deadlocks of prepared transactions vs. catalog tables. What if the > prepared transaction

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Craig Ringer
On 28 March 2017 at 08:50, Stas Kelvich wrote: > >> On 28 Mar 2017, at 00:19, Stas Kelvich wrote: >> >> * It is actually doesn’t pass one of mine regression tests. I’ve added >> expected output >> as it should be. I’ll try to send follow up message with fix, but right now >> sending it >> as is

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Craig Ringer
On 28 March 2017 at 05:25, Andres Freund wrote: > On a very quick skim, this doesn't seem to solve the issues around > deadlocks of prepared transactions vs. catalog tables. What if the > prepared transaction contains something like LOCK pg_class; (there's a > lot more realistic examples)? Then

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Stas Kelvich
> On 28 Mar 2017, at 00:19, Stas Kelvich wrote: > > * It is actually doesn’t pass one of mine regression tests. I’ve added > expected output > as it should be. I’ll try to send follow up message with fix, but right now > sending it > as is, as you asked. > > Fixed. I forgot to postpone Reor

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Andres Freund
Hi, On 2017-03-28 00:19:29 +0300, Stas Kelvich wrote: > Ok, here it is. On a very quick skim, this doesn't seem to solve the issues around deadlocks of prepared transactions vs. catalog tables. What if the prepared transaction contains something like LOCK pg_class; (there's a lot more realistic

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Stas Kelvich
> On 27 Mar 2017, at 16:29, Craig Ringer wrote: > > On 27 March 2017 at 17:53, Stas Kelvich wrote: > >> I’m heavily underestimated amount of changes there, but almost finished >> and will send updated patch in several hours. > > Oh, brilliant! Please post whatever you have before you knock of

Re: [HACKERS] Logical decoding on standby

2017-03-27 Thread Craig Ringer
On 27 March 2017 at 16:20, Simon Riggs wrote: > On 27 March 2017 at 09:03, Craig Ringer wrote: > >> I think this one's ready to go. > > Looks like something I could commit. Full review by me while offline > today, aiming to commit tomorrow barring issues raised. Great. Meanwhile I'm going to be

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Craig Ringer
On 27 March 2017 at 17:53, Stas Kelvich wrote: > I’m heavily underestimated amount of changes there, but almost finished > and will send updated patch in several hours. Oh, brilliant! Please post whatever you have before you knock off for the day anyway, even if it's just a WIP, so I can pick it

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Stas Kelvich
> On 27 Mar 2017, at 12:26, Craig Ringer wrote: > > On 27 March 2017 at 09:31, Craig Ringer wrote: > >> We're in the last week of the CF. If you have a patch that's nearly >> ready or getting there, now would be a good time to post it for help >> and input from others. >> >> I would really li

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-27 Thread Craig Ringer
On 27 March 2017 at 09:31, Craig Ringer wrote: > We're in the last week of the CF. If you have a patch that's nearly > ready or getting there, now would be a good time to post it for help > and input from others. > > I would really like to get this in, but we're running out of time. > > Even if y

Re: [HACKERS] Logical decoding on standby

2017-03-27 Thread Simon Riggs
On 27 March 2017 at 09:03, Craig Ringer wrote: > I think this one's ready to go. Looks like something I could commit. Full review by me while offline today, aiming to commit tomorrow barring issues raised. -- Simon Riggshttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 S

Re: [HACKERS] Logical decoding on standby

2017-03-27 Thread Craig Ringer
On 27 March 2017 at 14:08, Craig Ringer wrote: > So this patch makes ReplicationSlotAcquire check that the slot > database matches the current database and refuse to acquire the slot > if it does not. New patch attached that drops above requirement, so slots can still be dropped from any DB. Th

Re: [HACKERS] Logical decoding on standby

2017-03-26 Thread Craig Ringer
Hi Here's the next patch in the split-up series, drop db-specific (logical) replication slots on DROP DATABASE. Current behaviour is to ERROR if logical slots exist on the DB, whether in-use or not. With this patch we can DROP a database if it has logical slots so long as they are not active. I

Re: [HACKERS] Logical decoding on standby

2017-03-26 Thread Craig Ringer
On 20 March 2017 at 17:33, Andres Freund wrote: > Have you checked how high the overhead of XLogReadDetermineTimeline is? > A non-local function call, especially into a different translation-unit > (no partial inlining), for every single page might end up being > noticeable. That's fine in the c

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-26 Thread Craig Ringer
On 20 March 2017 at 21:47, Stas Kelvich wrote: > >> On 20 Mar 2017, at 16:39, Craig Ringer wrote: >> >> On 20 March 2017 at 20:57, Stas Kelvich wrote: >>> On 20 Mar 2017, at 15:17, Craig Ringer wrote: > I thought about having special field (or reusing one of the existing > fi

Re: [HACKERS] Logical decoding on standby

2017-03-25 Thread Simon Riggs
On 24 March 2017 at 06:23, Craig Ringer wrote: > On 23 March 2017 at 17:44, Craig Ringer wrote: > > Minor update to catalog_xmin walsender patch to fix failure to > parenthesize definition of PROCARRAY_PROC_FLAGS_MASK . > > This one's ready to go. Working on drop slots on DB drop now. Committed.

Re: [HACKERS] Logical decoding on standby

2017-03-23 Thread Craig Ringer
On 23 March 2017 at 17:44, Craig Ringer wrote: Minor update to catalog_xmin walsender patch to fix failure to parenthesize definition of PROCARRAY_PROC_FLAGS_MASK . This one's ready to go. Working on drop slots on DB drop now. -- Craig Ringer http://www.2ndQuadrant.com/ Pos

Re: [HACKERS] Logical decoding on standby

2017-03-23 Thread Craig Ringer
On 23 March 2017 at 16:07, Craig Ringer wrote: > If preferred I can instead add > > proc.h: > > #define PROC_RESERVED 0x20 > > procarray.h: > > #define PROCARRAY_REPLICATION_SLOTS 0x20 > > and then test for (flags & PROCARRAY_REPLICATION_SLOTS) Attached done that way. -- Craig Ringer

Re: [HACKERS] Logical decoding on standby

2017-03-23 Thread Craig Ringer
On 23 March 2017 at 00:13, Simon Riggs wrote: > On 22 March 2017 at 08:53, Craig Ringer wrote: > >> I'm splitting up the rest of the decoding on standby patch set with >> the goal of getting minimal functionality for creating and managing >> slots on standbys in, so we can maintain slots on stand

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Craig Ringer
On 23 March 2017 at 12:41, Andres Freund wrote: > On 2017-03-23 12:14:02 +0800, Craig Ringer wrote: >> On 23 March 2017 at 09:39, Andres Freund wrote: >> > I still think decoding-on-standby is simply not the right approach as >> > the basic/first HA approach for logical rep. It's a nice later-on

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Andres Freund
On 2017-03-23 12:14:02 +0800, Craig Ringer wrote: > On 23 March 2017 at 09:39, Andres Freund wrote: > > I still think decoding-on-standby is simply not the right approach as > > the basic/first HA approach for logical rep. It's a nice later-on > > feature. But that's an irrelevant aside. > > I

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Craig Ringer
On 23 March 2017 at 00:13, Simon Riggs wrote: > On 22 March 2017 at 08:53, Craig Ringer wrote: > >> I'm splitting up the rest of the decoding on standby patch set with >> the goal of getting minimal functionality for creating and managing >> slots on standbys in, so we can maintain slots on stand

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Craig Ringer
On 23 March 2017 at 09:39, Andres Freund wrote: > We can't just assume that snapbuild is going to work correctly when it's > prerequisites - pinned xmin horizon - isn't working. Makes sense. >> What do _you_ see as the minimum acceptable way to achieve the ability >> for a logical decoding clie

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Andres Freund
On 2017-03-23 09:14:07 +0800, Craig Ringer wrote: > On 23 March 2017 at 07:31, Andres Freund wrote: > > On 2017-03-23 06:55:53 +0800, Craig Ringer wrote: > > >> I was thinking that by disallowing snapshot use and output plugin > >> invocation we'd avoid the need to support cancellation on recover

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Craig Ringer
On 23 March 2017 at 07:31, Andres Freund wrote: > On 2017-03-23 06:55:53 +0800, Craig Ringer wrote: >> I was thinking that by disallowing snapshot use and output plugin >> invocation we'd avoid the need to support cancellation on recovery >> conflicts, etc, simplifying things considerably. > > Th

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Craig Ringer
On 23 March 2017 at 00:17, Andres Freund wrote: > On 2017-03-22 15:59:42 +, Simon Riggs wrote: >> On 22 March 2017 at 13:06, Andres Freund wrote: >> >> >> The parts I think are important for Pg10 are: >> > >> >> * Ability to create logical slots on replicas >> > >> > Doesn't this also imply r

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Andres Freund
On 2017-03-23 06:55:53 +0800, Craig Ringer wrote: > On 22 March 2017 at 21:06, Andres Freund wrote: > > Hi, > > > > On 2017-03-21 09:05:26 +0800, Craig Ringer wrote: > >> > 0002 should be doable as a whole this release, I have severe doubts that > >> > 0003 as a whole has a chance for 10 - the cod

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Craig Ringer
On 22 March 2017 at 21:06, Andres Freund wrote: > Hi, > > On 2017-03-21 09:05:26 +0800, Craig Ringer wrote: >> > 0002 should be doable as a whole this release, I have severe doubts that >> > 0003 as a whole has a chance for 10 - the code is in quite a raw shape, >> > there's a significant number o

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Andres Freund
On 2017-03-22 15:59:42 +, Simon Riggs wrote: > On 22 March 2017 at 13:06, Andres Freund wrote: > > >> The parts I think are important for Pg10 are: > > > >> * Ability to create logical slots on replicas > > > > Doesn't this also imply recovery conflicts on DROP DATABASE? > > Not needed until

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Simon Riggs
On 22 March 2017 at 08:53, Craig Ringer wrote: > I'm splitting up the rest of the decoding on standby patch set with > the goal of getting minimal functionality for creating and managing > slots on standbys in, so we can maintain slots on standbys and use > them when the standby is promoted to ma

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Simon Riggs
On 22 March 2017 at 13:06, Andres Freund wrote: >> The parts I think are important for Pg10 are: > >> * Ability to create logical slots on replicas > > Doesn't this also imply recovery conflicts on DROP DATABASE? Not needed until the slot is in use, which is a later patch. > Besides, > allowing

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Andres Freund
On 2017-03-22 14:58:29 +, Simon Riggs wrote: > On 22 March 2017 at 13:06, Andres Freund wrote: > > > But I have to admit, I've *severe* doubts about getting the whole > > infrastructure for slot creation on replica into 10. The work is far > > from ready, and we're mere days away from freeze

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Simon Riggs
On 22 March 2017 at 13:06, Andres Freund wrote: > But I have to admit, I've *severe* doubts about getting the whole > infrastructure for slot creation on replica into 10. The work is far > from ready, and we're mere days away from freeze. If Craig has to guess what would be acceptable, then its

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Andres Freund
Hi, On 2017-03-21 09:05:26 +0800, Craig Ringer wrote: > > 0002 should be doable as a whole this release, I have severe doubts that > > 0003 as a whole has a chance for 10 - the code is in quite a raw shape, > > there's a significant number of open ends. I'd suggest breaking of bits > > that are i

Re: [HACKERS] Logical decoding on standby

2017-03-22 Thread Craig Ringer
On 22 March 2017 at 10:51, Craig Ringer wrote: > Hi all > > Updated timeline following patch attached. > > There's a change in read_local_xlog_page to ensure we maintain > ThisTimeLineID properly, otherwise it's just comment changes. OK, so we're looking OK with the TL following. I'm splitting u

Re: [HACKERS] Logical decoding on standby

2017-03-21 Thread Craig Ringer
Hi all Updated timeline following patch attached. There's a change in read_local_xlog_page to ensure we maintain ThisTimeLineID properly, otherwise it's just comment changes. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Service

Re: [HACKERS] Logical decoding on standby

2017-03-21 Thread Craig Ringer
On 21 March 2017 at 09:05, Craig Ringer wrote: > Thanks, that's a helpful point. The commit in question is 978b2f65. I > didn't notice that it introduced XLogReader use in twophase.c, though > I should've realised given the discussion about fetching recent 2pc > info from xlog. I don't see any po

Re: [HACKERS] Logical decoding on standby

2017-03-21 Thread Simon Riggs
On 21 March 2017 at 02:21, Craig Ringer wrote: > On 20 March 2017 at 17:33, Andres Freund wrote: > >>> Subject: [PATCH 2/3] Follow timeline switches in logical decoding >> >> FWIW, the title doesn't really seem accurate to me. > > Yeah, it's not really at the logical decoding layer at all. > > "T

Re: [HACKERS] Logical decoding on standby

2017-03-20 Thread Craig Ringer
On 20 March 2017 at 17:33, Andres Freund wrote: >> Subject: [PATCH 2/3] Follow timeline switches in logical decoding > > FWIW, the title doesn't really seem accurate to me. Yeah, it's not really at the logical decoding layer at all. "Teach xlogreader to follow timeline switches" ? >> Logical s

Re: [HACKERS] Logical decoding on standby

2017-03-20 Thread Craig Ringer
.On 20 March 2017 at 17:33, Andres Freund wrote: > Hi, > > Have you checked how high the overhead of XLogReadDetermineTimeline is? > A non-local function call, especially into a different translation-unit > (no partial inlining), for every single page might end up being > noticeable. That's fine

Re: [HACKERS] Logical decoding on standby

2017-03-20 Thread Craig Ringer
On 19 March 2017 at 22:12, Petr Jelinek wrote: > I am slightly worried about impact of the readTimeLineHistory() call but > I think it should be called so little that it should not matter. Pretty much my thinking too. > That brings us to the big patch 0003. > > I still don't like the "New in 10

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-20 Thread Stas Kelvich
> On 20 Mar 2017, at 16:39, Craig Ringer wrote: > > On 20 March 2017 at 20:57, Stas Kelvich wrote: >> >>> On 20 Mar 2017, at 15:17, Craig Ringer wrote: >>> I thought about having special field (or reusing one of the existing fields) in snapshot struct to force filtering xmax

  1   2   3   4   >