Re: [HACKERS] Logical replication and multimaster

On 12/13/15 7:37 AM, David Fetter wrote: As I understand it, pushing these into a library has been proposed but not rejected. That it hasn't happened yet is mostly about the lack of tuits (the round ones) to rewrite the functionality as libraries and refactor pg_dump/pg_restore to use only

Re: [HACKERS] Logical replication and multimaster

On Tue, 15 Dec 2015 21:48:52 -0600 Jim Nasby wrote: > On 12/13/15 7:37 AM, David Fetter wrote: > > As I understand it, pushing these into a library has been proposed but > > not rejected. That it hasn't happened yet is mostly about the lack of > > tuits (the round

Re: [HACKERS] Logical replication and multimaster

I have updated DTM page at PoastgreSQL WiKi adding information about multimaster. Also we have created repository at github with our version of PostgreSQL and DTM extensions: multimaster, pg_dtm,

Re: [HACKERS] Logical replication and multimaster

On 13 December 2015 at 11:02, Andres Freund wrote: > On December 13, 2015 10:19:07 AM CET, Simon Riggs > wrote: > >I didn't see the patch for this anywhere. Where is the code? > > Where I'd the code for all of pg logical? > Thanks for asking, perhaps

Re: [HACKERS] Logical replication and multimaster

On 2015-12-13 11:39:32 +, Simon Riggs wrote: > On 13 December 2015 at 11:02, Andres Freund wrote: > > On December 13, 2015 10:19:07 AM CET, Simon Riggs > > wrote: > > >I didn't see the patch for this anywhere. Where is the code? > > > > Where is the

Re: [HACKERS] Logical replication and multimaster

On December 13, 2015 10:19:07 AM CET, Simon Riggs wrote: >I didn't see the patch for this anywhere. Where is the code? Where I'd the code for all of pg logical? --- Please excuse brevity and formatting - I am writing this on my mobile phone. -- Sent via pgsql-hackers

Re: [HACKERS] Logical replication and multimaster

On 6 December 2015 at 17:39, Konstantin Knizhnik wrote: > I have integrated pglogical_output in multimaster, using bdr_apply from > BDR as template for implementation of receiver part. > I didn't see the patch for this anywhere. Where is the code? -- Simon Riggs

Re: [HACKERS] Logical replication and multimaster

On 12/13/2015 12:19 PM, Simon Riggs wrote: On 6 December 2015 at 17:39, Konstantin Knizhnik > wrote: I have integrated pglogical_output in multimaster, using bdr_apply from BDR as template for implementation of receiver part.

Re: [HACKERS] Logical replication and multimaster

On Sat, Dec 12, 2015 at 06:48:58PM +0800, Craig Ringer wrote: > Being able to access pg_dump and pg_restore's dependency resolution logic, > object dumping routines, etc from regular SQL and from the SPI would be > wonderful. As I understand it, pushing these into a library has been proposed but

Re: [HACKERS] Logical replication and multimaster

On 13 December 2015 at 11:53, Andres Freund wrote: > > Thanks for asking, perhaps our plans weren't public enough. pglogical has > > already been announced as open source, under the postgres licence and > that > > it will be a submission to core PostgreSQL, just as BDR was.

Re: [HACKERS] Logical replication and multimaster

On 12 December 2015 at 02:24, Robert Haas wrote: > On Fri, Dec 11, 2015 at 5:16 AM, Andres Freund wrote: > > On 2015-12-11 18:12:55 +0800, Craig Ringer wrote: > >> On 10 December 2015 at 03:19, Robert Haas > wrote: > >> > On

Re: [HACKERS] Logical replication and multimaster

On 10 December 2015 at 03:19, Robert Haas wrote: > On Sun, Dec 6, 2015 at 10:24 PM, Craig Ringer > wrote: > > > * A way to securely make a libpq connection from a bgworker without > messing > > with passwords etc. Generate one-time cookies,

Re: [HACKERS] Logical replication and multimaster

On 2015-12-11 18:12:55 +0800, Craig Ringer wrote: > On 10 December 2015 at 03:19, Robert Haas wrote: > > On Sun, Dec 6, 2015 at 10:24 PM, Craig Ringer > > wrote: > > > * A way to securely make a libpq connection from a bgworker without > > messing >

Re: [HACKERS] Logical replication and multimaster

On 2015-12-11 19:24, Robert Haas wrote: On Fri, Dec 11, 2015 at 5:16 AM, Andres Freund wrote: On 2015-12-11 18:12:55 +0800, Craig Ringer wrote: On 10 December 2015 at 03:19, Robert Haas wrote: On Sun, Dec 6, 2015 at 10:24 PM, Craig Ringer

Re: [HACKERS] Logical replication and multimaster

On Fri, Dec 11, 2015 at 5:16 AM, Andres Freund wrote: > On 2015-12-11 18:12:55 +0800, Craig Ringer wrote: >> On 10 December 2015 at 03:19, Robert Haas wrote: >> > On Sun, Dec 6, 2015 at 10:24 PM, Craig Ringer >> > wrote: >> > > *

Re: [HACKERS] Logical replication and multimaster

On Sun, Dec 6, 2015 at 10:24 PM, Craig Ringer wrote: > * An API to enumerate currently registered bgworkers > * A way to securely make a libpq connection from a bgworker without messing > with passwords etc. Generate one-time cookies, sometihng like that. > * (unimportant

Re: [HACKERS] Logical replication and multimaster

On 2015-12-03 09:54:23 +0300, konstantin knizhnik wrote: > But right now performance of Multimaster is not limited by logical > replication protocol - if I remove DTM and use asynchronous replication > (lightweight version of BDR:) > then I get 38k TPS instead of 12k. My guess is that that's to

Re: [HACKERS] Logical replication and multimaster

Hi, I have integrated pglogical_output in multimaster, using bdr_apply from BDR as template for implementation of receiver part. The time of insert is reduced almost 10 times comparing with logical replication based on decoder_raw/receiver_raw plugins which performs logical replication using

Re: [HACKERS] Logical replication and multimaster

On 7 December 2015 at 01:39, Konstantin Knizhnik wrote: > > I have integrated pglogical_output in multimaster > Excellent. I just pushed a change to pglogical_output that exposes the row contents (and the rest of the reorder change buffer contents) to hooks that want

Re: [HACKERS] Logical replication and multimaster

> There are definitely two clear places where additional help would be > useful and welcome right now. > Here's a shortlist of replication-related open items I put together earlier. These are all independent things it'd be helpful to have when working with logical replication in PostgreSQL. There

Re: [HACKERS] Logical replication and multimaster

On 3 December 2015 at 20:39, Simon Riggs wrote: > On 30 November 2015 at 17:20, Konstantin Knizhnik < > k.knizh...@postgrespro.ru> wrote: > > >> But looks like there is not so much sense in having multiple network >> connection between one pair of nodes. >> It seems to be

Re: [HACKERS] Logical replication and multimaster

On 30 November 2015 at 17:20, Konstantin Knizhnik wrote: > But looks like there is not so much sense in having multiple network > connection between one pair of nodes. > It seems to be better to have one connection between nodes, but provide > parallel execution of

Re: [HACKERS] Logical replication and multimaster

On Dec 3, 2015, at 10:34 AM, Craig Ringer wrote: > On 3 December 2015 at 14:54, konstantin knizhnik > wrote: > >> I'd really like to collaborate using pglogical_output if at all possible. >> Petr's working really hard to get the pglogical downstrem out too, with me

Re: [HACKERS] Logical replication and multimaster

On 3 December 2015 at 12:06, konstantin knizhnik wrote: > > On Dec 3, 2015, at 10:34 AM, Craig Ringer wrote: > > On 3 December 2015 at 14:54, konstantin knizhnik < > k.knizh...@postgrespro.ru> wrote: > >> >> I'd really like to collaborate using pglogical_output if at

Re: [HACKERS] Logical replication and multimaster

On Thu, Dec 3, 2015 at 8:34 AM, Craig Ringer wrote: > On 3 December 2015 at 14:54, konstantin knizhnik < > k.knizh...@postgrespro.ru> wrote: > >> >> Are there some principle problems with it? In BDR it was handled in >> alternative way, using executor callback. It will be

Re: [HACKERS] Logical replication and multimaster

On 3 December 2015 at 19:06, konstantin knizhnik wrote: > > On Dec 3, 2015, at 10:34 AM, Craig Ringer wrote: > > On 3 December 2015 at 14:54, konstantin knizhnik < > k.knizh...@postgrespro.ru> wrote: > >> >> I'd really like to collaborate using pglogical_output if at

Re: [HACKERS] Logical replication and multimaster

On 3 December 2015 at 20:39, Simon Riggs wrote: > On 30 November 2015 at 17:20, Konstantin Knizhnik < > k.knizh...@postgrespro.ru> wrote: > > >> But looks like there is not so much sense in having multiple network >> connection between one pair of nodes. >> It seems to be

Re: [HACKERS] Logical replication and multimaster

On 3 December 2015 at 15:27, konstantin knizhnik wrote: > > On Dec 3, 2015, at 4:18 AM, Craig Ringer wrote: > > > Excellent. > > > > It should be possible to make that a separate extension. You can use C > functions from other extensions by exposing a single pg_proc

Re: [HACKERS] Logical replication and multimaster

On 2015-12-03 14:32, Craig Ringer wrote: On 3 December 2015 at 15:27, konstantin knizhnik > wrote: 3. What is the right way of creation of background worker requiring access to shared memory, i.e. having control structure in

Re: [HACKERS] Logical replication and multimaster

On Mon, Nov 30, 2015 at 11:20 AM, Konstantin Knizhnik wrote: > We have implemented ACID multimaster based on logical replication and our > DTM (distributed transaction manager) plugin. > Good news is that it works and no inconsistency is detected. > But unfortunately it

Re: [HACKERS] Logical replication and multimaster

Thank you for reply. On 12/02/2015 08:30 PM, Robert Haas wrote: Logical decoding only begins decoding a transaction once the transaction is complete. So I would guess that the sequence of operations here is something like this - correct me if I'm wrong: 1. Do the transaction. 2. PREPARE. 3.

Re: [HACKERS] Logical replication and multimaster

On 1 December 2015 at 00:20, Konstantin Knizhnik wrote: > We have implemented ACID multimaster based on logical replication and our DTM (distributed transaction manager) plugin. What are you using for an output plugin and for replay? I'd really like to collaborate

Re: [HACKERS] Logical replication and multimaster

On 3 December 2015 at 04:18, Konstantin Knizhnik wrote: > The problem is that transactions are delivered to replica through single > channel: logical replication slot. > And while such transaction is waiting acknowledgement from arbiter, it is > blocking replication

Re: [HACKERS] Logical replication and multimaster

On Dec 3, 2015, at 4:09 AM, Craig Ringer wrote: > On 1 December 2015 at 00:20, Konstantin Knizhnik > wrote: > > > We have implemented ACID multimaster based on logical replication and our > > DTM (distributed transaction manager) plugin. > > What are you using for

Re: [HACKERS] Logical replication and multimaster

On 3 December 2015 at 14:54, konstantin knizhnik wrote: > > I'd really like to collaborate using pglogical_output if at all possible. > Petr's working really hard to get the pglogical downstrem out too, with me > helping where I can. > > And where I can get

Re: [HACKERS] Logical replication and multimaster

On Dec 3, 2015, at 4:18 AM, Craig Ringer wrote: > Excellent. > > It should be possible to make that a separate extension. You can use C > functions from other extensions by exposing a single pg_proc function with > 'internal' return type that populates a struct of function pointers for the >

Re: [HACKERS] Logical replication and multimaster

On 3 December 2015 at 01:30, Robert Haas wrote: > 1. Do the transaction. > 2. PREPARE. > 3. Replay the transaction. > As Konstantin noted they aren't using Pg's 2PC. They actually couldn't if they wanted to because logical decoding does not support decoding an xact at

[HACKERS] Logical replication and multimaster

Hello all, We have implemented ACID multimaster based on logical replication and our DTM (distributed transaction manager) plugin. Good news is that it works and no inconsistency is detected. But unfortunately it is very very slow... At standalone PostgreSQL I am able to achieve about 3