Re: [HACKERS] logical changeset generation v3 - git repository

2012-12-13 Thread Andres Freund
Hi Peter! Thanks for the review, you raise many noteworthy points. This is going to be a long mail... On 2012-12-13 00:05:41 +, Peter Geoghegan wrote: I'm very glad that you followed my earlier recommendation of splitting your demo logical changeset consumer into a contrib module, in the

Re: [HACKERS] logical changeset generation v3 - git repository

2012-12-12 Thread Peter Geoghegan
On 9 December 2012 19:14, Andres Freund and...@2ndquadrant.com wrote: I pushed a new version which - is rebased ontop of master - is based ontop of the new xlogreader (biggest part) - is base ontop of the new binaryheap.h - some fixes - some more comments I decided to take another look at

Re: [HACKERS] logical changeset generation v3 - git repository

2012-12-09 Thread Andres Freund
On 2012-11-15 02:26:53 +0100, Andres Freund wrote: On 2012-11-15 01:27:46 +0100, Andres Freund wrote: In response to this you will soon find the 14 patches that currently implement $subject. As its not very wieldly to send around that many/big patches all the time, until the next major

Re: [HACKERS] logical changeset generation v3

2012-11-21 Thread Michael Paquier
On Wed, Nov 21, 2012 at 4:34 PM, Andres Freund and...@2ndquadrant.comwrote: On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: Ah, I see. Could you try the following diff? diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index

Re: [HACKERS] logical changeset generation v3

2012-11-21 Thread Andres Freund
On 2012-11-21 18:35:34 +0900, Michael Paquier wrote: On Wed, Nov 21, 2012 at 4:34 PM, Andres Freund and...@2ndquadrant.comwrote: On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: Ah, I see. Could you try the following diff? diff --git a/src/backend/replication/logical/snapbuild.c

Re: [HACKERS] logical changeset generation v3

2012-11-21 Thread Michael Paquier
On Thu, Nov 22, 2012 at 8:25 AM, Andres Freund and...@2ndquadrant.comwrote: I really don't understand whats going on here then. Youve said you made sure that there is a catalog snapshot. Which means you would need something like: WARNING: connecting to postgres WARNING: Initiating logical

Re: [HACKERS] logical changeset generation v3

2012-11-21 Thread Andres Freund
On 2012-11-22 09:13:30 +0900, Michael Paquier wrote: On Thu, Nov 22, 2012 at 8:25 AM, Andres Freund and...@2ndquadrant.comwrote: I really don't understand whats going on here then. Youve said you made sure that there is a catalog snapshot. Which means you would need something like:

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-20 09:30:40 +0900, Michael Paquier wrote: On Mon, Nov 19, 2012 at 5:50 PM, Andres Freund and...@2ndquadrant.comwrote: On 2012-11-19 16:28:55 +0900, Michael Paquier wrote: I am just looking at this patch and will provide some comments. By the way, you forgot the installation

Re: [HACKERS] logical changeset generation v3 - Source for Slony

2012-11-20 Thread Andres Freund
Hi, On 2012-11-19 19:50:32 -0500, Steve Singer wrote: On 12-11-18 11:07 AM, Andres Freund wrote: I think we should provide some glue code to do this, otherwise people will start replicating all the bugs I hacked into this... More seriously: I think we should have support code here, no user

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Michael Paquier
On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund and...@2ndquadrant.comwrote: Those aren't unexpected. Perhaps I should not make it a warning then... A short explanation: We can only decode tuples we see in the WAL when we already have a timetravel catalog snapshot before that transaction

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Michael Paquier
On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund and...@2ndquadrant.comwrote: On 2012-11-20 09:30:40 +0900, Michael Paquier wrote: On Mon, Nov 19, 2012 at 5:50 PM, Andres Freund and...@2ndquadrant.com wrote: On 2012-11-19 16:28:55 +0900, Michael Paquier wrote: I am just looking at this

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-21 15:28:30 +0900, Michael Paquier wrote: On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund and...@2ndquadrant.comwrote: On 2012-11-20 09:30:40 +0900, Michael Paquier wrote: Btw, here are some extra comments based on my progress, hope it will be useful for other people playing

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund and...@2ndquadrant.comwrote: Those aren't unexpected. Perhaps I should not make it a warning then... A short explanation: We can only decode tuples we see in the WAL when we already

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Michael Paquier
On Wed, Nov 21, 2012 at 4:31 PM, Andres Freund and...@2ndquadrant.comwrote: On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund and...@2ndquadrant.com wrote: It implies that snapstate-nrrunning has lost touch with reality... Yes, I can

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund and...@2ndquadrant.comwrote: Those aren't unexpected. Perhaps I should not make it a warning then... A short explanation: We can only decode tuples we see in the WAL when we already

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Michael Paquier
On Wed, Nov 21, 2012 at 4:30 PM, Andres Freund and...@2ndquadrant.comwrote: On 2012-11-21 15:28:30 +0900, Michael Paquier wrote: On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund and...@2ndquadrant.com wrote: On 2012-11-20 09:30:40 +0900, Michael Paquier wrote: Btw, here are some extra

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-21 16:47:11 +0900, Michael Paquier wrote: On Wed, Nov 21, 2012 at 4:30 PM, Andres Freund and...@2ndquadrant.comwrote: On 2012-11-21 15:28:30 +0900, Michael Paquier wrote: On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund and...@2ndquadrant.com wrote: On 2012-11-20 09:30:40

Re: [HACKERS] logical changeset generation v3

2012-11-19 Thread Andres Freund
Hi Michael, On 2012-11-19 16:28:55 +0900, Michael Paquier wrote: I have been able to fetch your code (thanks Andrea!) and some it. For the time being I am spending some time reading the code and understanding the whole set of features you are trying to implement inside core, even if I got

Re: [HACKERS] logical changeset generation v3 - Source for Slony

2012-11-19 Thread Steve Singer
First, you can add me to the list of people saying 'wow', I'm impressed. The approach I am taking to reviewing this to try and answer the following question 1) How might a future version of slony be able to use logical replication as described by your patch and design documents and what

Re: [HACKERS] logical changeset generation v3 - Source for Slony

2012-11-19 Thread Andres Freund
Hi Steve! On 2012-11-17 22:50:35 -0500, Steve Singer wrote: First, you can add me to the list of people saying 'wow', I'm impressed. Thanks! The approach I am taking to reviewing this to try and answer the following question 1) How might a future version of slony be able to use logical

Re: [HACKERS] logical changeset generation v3

2012-11-19 Thread Michael Paquier
On Mon, Nov 19, 2012 at 5:50 PM, Andres Freund and...@2ndquadrant.comwrote: Hi Michael, On 2012-11-19 16:28:55 +0900, Michael Paquier wrote: I have been able to fetch your code (thanks Andrea!) and some it. For the time being I am spending some time reading the code and understanding the

Re: [HACKERS] logical changeset generation v3 - Source for Slony

2012-11-19 Thread Steve Singer
On 12-11-18 11:07 AM, Andres Freund wrote: Hi Steve! I think we should provide some glue code to do this, otherwise people will start replicating all the bugs I hacked into this... More seriously: I think we should have support code here, no user will want to learn the intracacies of feedback

Re: [HACKERS] logical changeset generation v3

2012-11-18 Thread Michael Paquier
On Fri, Nov 16, 2012 at 5:16 PM, Andrea Suisani sick...@opinioni.netwrote: Il 16/11/2012 05:34, Michael Paquier ha scritto: Do you have a git repository or something where all the 14 patches are applied? I would like to test the feature globally. Sorry I recall that you put a link somewhere

Re: [HACKERS] logical changeset generation v3

2012-11-18 Thread Michael Paquier
Hi Andres, I have been able to fetch your code (thanks Andrea!) and some it. For the time being I am spending some time reading the code and understanding the whole set of features you are trying to implement inside core, even if I got some background from what you presented at PGCon and from the

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Markus Wanner
On 11/16/2012 03:05 PM, Andres Freund wrote: I'd like to provide a comparison of the proposed change set format to the one used in Postgres-R. Uh, sorry to interrupt you right here, but thats not the proposed format ;) Understood. Sorry, I didn't mean to imply that. It's pretty obvious to

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Markus Wanner
On 11/16/2012 03:14 PM, Andres Freund wrote: Whats the data type of the COID in -R? It's short for CommitOrderId, a 32bit global transaction identifier, being wrapped-around, very much like TransactionIds are. (In that sense, it's global, but unique only for a certain amount of time). In the

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Hannu Krosing
On 11/16/2012 02:46 PM, Markus Wanner wrote: Andres, On 11/15/2012 01:27 AM, Andres Freund wrote: In response to this you will soon find the 14 patches that currently implement $subject. Congratulations on that piece of work. I'd like to provide a comparison of the proposed change set

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Markus Wanner
On 11/17/2012 02:30 PM, Hannu Krosing wrote: Is it possible to replicate UPDATEs and DELETEs without a primary key in PostgreSQL-R No. There must be some way to logically identify the tuple. Note, though, that theoretically any (unconditional) unique key would suffice. In practice, that usually

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Hannu Krosing
On 11/17/2012 03:00 PM, Markus Wanner wrote: On 11/17/2012 02:30 PM, Hannu Krosing wrote: Is it possible to replicate UPDATEs and DELETEs without a primary key in PostgreSQL-R No. There must be some way to logically identify the tuple. It can be done as selecting on _all_ attributes and

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Hannu Krosing
On 11/17/2012 03:00 PM, Markus Wanner wrote: On 11/17/2012 02:30 PM, Hannu Krosing wrote: Is it possible to replicate UPDATEs and DELETEs without a primary key in PostgreSQL-R No. There must be some way to logically identify the tuple. Note, though, that theoretically any (unconditional)

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Markus Wanner
Hannu, On 11/17/2012 03:40 PM, Hannu Krosing wrote: On 11/17/2012 03:00 PM, Markus Wanner wrote: On 11/17/2012 02:30 PM, Hannu Krosing wrote: Is it possible to replicate UPDATEs and DELETEs without a primary key in PostgreSQL-R No. There must be some way to logically identify the tuple. It

Re: [HACKERS] logical changeset generation v3

2012-11-16 Thread Andrea Suisani
Il 16/11/2012 05:34, Michael Paquier ha scritto: Do you have a git repository or something where all the 14 patches are applied? I would like to test the feature globally. Sorry I recall that you put a link somewhere but I cannot remember its email...

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-16 Thread Markus Wanner
Andres, On 11/15/2012 01:27 AM, Andres Freund wrote: In response to this you will soon find the 14 patches that currently implement $subject. Congratulations on that piece of work. I'd like to provide a comparison of the proposed change set format to the one used in Postgres-R. I hope for

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-16 Thread Andres Freund
Hi Markus, On 2012-11-16 14:46:39 +0100, Markus Wanner wrote: On 11/15/2012 01:27 AM, Andres Freund wrote: In response to this you will soon find the 14 patches that currently implement $subject. Congratulations on that piece of work. Thanks. I'd like to provide a comparison of the

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-16 Thread Andres Freund
Hi, On 2012-11-16 14:46:39 +0100, Markus Wanner wrote: You may have noticed that there's an additional COID field. This is an identifier for the transaction that last changed this tuple. Together with the primary key, it effectively identifies the exact version of a tuple (during its

Re: [HACKERS] logical changeset generation v3

2012-11-15 Thread Andres Freund
Hi, On Thursday, November 15, 2012 05:08:26 AM Michael Paquier wrote: Looks like cool stuff @-@ I might be interested in looking at that a bit as I think I will hopefully be hopefully be able to grab some time in the next couple of weeks. Are some of those patches already submitted to a CF?

Re: [HACKERS] logical changeset generation v3

2012-11-15 Thread Michael Paquier
Do you have a git repository or something where all the 14 patches are applied? I would like to test the feature globally. Sorry I recall that you put a link somewhere but I cannot remember its email... On Thu, Nov 15, 2012 at 6:34 PM, Andres Freund and...@anarazel.de wrote: Hi, On Thursday,

[HACKERS] logical changeset generation v3

2012-11-14 Thread Andres Freund
Hi, In response to this you will soon find the 14 patches that currently implement $subject. I'll go over each one after showing off for a bit: Start postgres: Start postgres instance (with pg_hba.conf allowing replication cons): $ postgres -D ~/tmp/pgdev-lcr \ -c wal_level=logical \

Re: [HACKERS] logical changeset generation v3 - git repository

2012-11-14 Thread Andres Freund
On 2012-11-15 01:27:46 +0100, Andres Freund wrote: In response to this you will soon find the 14 patches that currently implement $subject. As its not very wieldly to send around that many/big patches all the time, until the next major version I will just update the git tree at: Web:

Re: [HACKERS] logical changeset generation v3

2012-11-14 Thread Josh Berkus
On 11/14/12 4:27 PM, Andres Freund wrote: Hi, In response to this you will soon find the 14 patches that currently implement $subject. I'll go over each one after showing off for a bit: Lemme be the first to say, wow. Impressive work. Now the debugging starts ... -- Josh Berkus

Re: [HACKERS] logical changeset generation v3

2012-11-14 Thread Michael Paquier
Looks like cool stuff @-@ I might be interested in looking at that a bit as I think I will hopefully be hopefully be able to grab some time in the next couple of weeks. Are some of those patches already submitted to a CF? -- Michael Paquier http://michael.otacoo.com