Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-18 Thread Peter Geoghegan
On 16 October 2012 15:26, Jan Wieck janwi...@yahoo.com wrote: This means that the transition time from the existing, trigger based approach to the new WAL based mechanism will see both technologies in parallel, which is no small thing to support. So, you're talking about a shim between the two

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-18 Thread Christopher Browne
On Thu, Oct 18, 2012 at 9:49 AM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 16 October 2012 15:26, Jan Wieck janwi...@yahoo.com wrote: This means that the transition time from the existing, trigger based approach to the new WAL based mechanism will see both technologies in parallel, which

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-18 Thread Peter Geoghegan
On 18 October 2012 16:18, Christopher Browne cbbro...@gmail.com wrote: A shim adds complexity, but retains the upgrade across versions use case, and reduces the need to keep supporting elder versions of Slony. Right. Upgrading across major versions is likely to continue to remain a very

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-16 Thread Jan Wieck
On 10/15/2012 4:43 PM, Simon Riggs wrote: Jan spoke at length at PgCon, for all to hear, that what we are building is a much better way than the trigger logging approach Slony uses. I don't take that as carte blanche for approval of everything being done, but its going in the right direction

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-16 Thread Jan Wieck
On 10/15/2012 3:25 PM, Andres Freund wrote: On Monday, October 15, 2012 09:18:57 PM Peter Geoghegan wrote: On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I think Robert is right that if Slony can't use the API, it is unlikely any other replication system could use it. I

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Bruce Momjian
On Mon, Oct 15, 2012 at 09:57:19AM +0200, Andres Freund wrote: On Monday, October 15, 2012 04:54:20 AM Robert Haas wrote: On Thu, Oct 11, 2012 at 3:15 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: IMHO that's a good thing, and I'd hope this new logical replication to live

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 08:19:54 PM Bruce Momjian wrote: On Mon, Oct 15, 2012 at 09:57:19AM +0200, Andres Freund wrote: On Monday, October 15, 2012 04:54:20 AM Robert Haas wrote: On Thu, Oct 11, 2012 at 3:15 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: IMHO that's a

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Hannu Krosing
On 10/11/2012 01:42 PM, Andres Freund wrote: On Thursday, October 11, 2012 09:15:47 AM Heikki Linnakangas wrote: ... If the only meaningful advantage is reducing the amount of WAL written, I can't help thinking that we should just try to address that in the existing solutions, even if it seems

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 08:38:07 PM Hannu Krosing wrote: On 10/11/2012 01:42 PM, Andres Freund wrote: On Thursday, October 11, 2012 09:15:47 AM Heikki Linnakangas wrote: ... If the only meaningful advantage is reducing the amount of WAL written, I can't help thinking that we should

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Hannu Krosing
On 10/15/2012 04:54 AM, Robert Haas wrote: On Thu, Oct 11, 2012 at 3:15 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: IMHO that's a good thing, and I'd hope this new logical replication to live outside core as well, as much as possible. But whether or not something is in core is just a

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Hannu Krosing
On 10/15/2012 08:44 PM, Andres Freund wrote: On Monday, October 15, 2012 08:38:07 PM Hannu Krosing wrote: On 10/11/2012 01:42 PM, Andres Freund wrote: On Thursday, October 11, 2012 09:15:47 AM Heikki Linnakangas wrote: ... If the only meaningful advantage is reducing the amount of WAL written,

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Hannu Krosing
On 10/15/2012 04:54 AM, Robert Haas wrote: PS. I'd love to see a basic Slony plugin for this, for example, to see how much extra code on top of the posted patches you need to write in a plugin like that to make it functional. I'm worried that it's a lot.. I agree. I would go so far as to say

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Peter Geoghegan
On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I think Robert is right that if Slony can't use the API, it is unlikely any other replication system could use it. I don't accept that. Clearly there is a circular dependency, and someone has to go first - why should the Slony guys

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Robert Haas
On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I think Robert is right that if Slony can't use the API, it is unlikely any other replication system could use it. I don't accept that. Clearly there

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 09:18:57 PM Peter Geoghegan wrote: On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I think Robert is right that if Slony can't use the API, it is unlikely any other replication system could use it. I don't accept that. Clearly there is a circular

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I think Robert is right that if Slony can't use the API, it is unlikely any other replication system could

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Christopher Browne
On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I think Robert is right that if Slony can't use the API, it is unlikely any other replication system could use it. I don't accept that. Clearly there

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 10:03:40 PM Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I think Robert is right that if Slony can't use

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Simon Riggs
On 15 October 2012 21:03, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I think Robert is right that if Slony can't use

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Monday, October 15, 2012 10:08:28 PM Christopher Browne wrote: On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I think Robert is right that if Slony can't use the API, it is unlikely any other

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Christopher Browne
On Mon, Oct 15, 2012 at 4:51 PM, Andres Freund and...@2ndquadrant.com wrote: On Monday, October 15, 2012 10:08:28 PM Christopher Browne wrote: On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 15 October 2012 19:19, Bruce Momjian br...@momjian.us wrote: I

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Andres Freund
On Tuesday, October 16, 2012 12:13:14 AM Christopher Browne wrote: On Mon, Oct 15, 2012 at 4:51 PM, Andres Freund and...@2ndquadrant.com wrote: On Monday, October 15, 2012 10:08:28 PM Christopher Browne wrote: On Mon, Oct 15, 2012 at 3:18 PM, Peter Geoghegan pe...@2ndquadrant.com wrote:

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-15 Thread Steve Singer
On 12-10-15 04:51 PM, Andres Freund wrote: Well, as a crosscheck, could you list your requirements? Do you need anything more than outputting data in a format compatible to whats stored in sl_log_*? You wouldn't have sl_actionseq, everything else should be there (Well, you would need to do

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-14 Thread Robert Haas
On Thu, Oct 11, 2012 at 3:15 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: IMHO that's a good thing, and I'd hope this new logical replication to live outside core as well, as much as possible. But whether or not something is in core is just a political decision, not a reason to

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-11 Thread Heikki Linnakangas
On 22.09.2012 20:00, Andres Freund wrote: [[basic-schema]] .Architecture Schema [ditaa] -- Traditional Stuff +-+-+-+-++ | Backend | Backend | Backend | Autovac | ...|

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-10-11 Thread Andres Freund
On Thursday, October 11, 2012 09:15:47 AM Heikki Linnakangas wrote: On 22.09.2012 20:00, Andres Freund wrote: [[basic-schema]] .Architecture Schema [ditaa] - - Traditional Stuff

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents (really attached)

2012-09-22 Thread m...@rpzdesign.com
Andres, nice job on the writeup. I think one aspect you are missing is that there must be some way for the multi-masters to re-stabilize their data sets and quantify any data loss. You cannot do this without some replication intelligence in each row of each table so that no matter how