Re: [HACKERS] logical changeset generation v3 - git repository

2012-12-13 Thread Andres Freund
Hi Peter! Thanks for the review, you raise many noteworthy points. This is going to be a long mail... On 2012-12-13 00:05:41 +, Peter Geoghegan wrote: > I'm very glad that you followed my earlier recommendation of splitting > your demo logical changeset consumer into a contrib module, in the

Re: [HACKERS] logical changeset generation v3 - git repository

2012-12-12 Thread Peter Geoghegan
On 9 December 2012 19:14, Andres Freund wrote: > I pushed a new version which > > - is rebased ontop of master > - is based ontop of the new xlogreader (biggest part) > - is base ontop of the new binaryheap.h > - some fixes > - some more comments I decided to take another look at this, following

Re: [HACKERS] logical changeset generation v3

2012-12-10 Thread Andres Freund
Hi, On 2012-11-19 09:50:30 +0100, Andres Freund wrote: > On 2012-11-19 16:28:55 +0900, Michael Paquier wrote: > > After launching some SQLs, the logical receiver is stuck just after sending > > INIT_LOGICAL_REPLICATION, please see bt of process waiting: > > Its waiting till it sees initial an init

Re: [HACKERS] logical changeset generation v3 - git repository

2012-12-09 Thread Andres Freund
On 2012-11-15 02:26:53 +0100, Andres Freund wrote: > On 2012-11-15 01:27:46 +0100, Andres Freund wrote: > > In response to this you will soon find the 14 patches that currently > > implement $subject. > > As its not very wieldly to send around that many/big patches all the > time, until the next "m

Re: [HACKERS] logical changeset generation v3

2012-11-21 Thread Andres Freund
On 2012-11-22 09:13:30 +0900, Michael Paquier wrote: > On Thu, Nov 22, 2012 at 8:25 AM, Andres Freund wrote: > > > I really don't understand whats going on here then. Youve said you made > > sure that there is a catalog snapshot. Which means you would need > > something like: > > WARNING: connecti

Re: [HACKERS] logical changeset generation v3

2012-11-21 Thread Michael Paquier
On Thu, Nov 22, 2012 at 8:25 AM, Andres Freund wrote: > I really don't understand whats going on here then. Youve said you made > sure that there is a catalog snapshot. Which means you would need > something like: > WARNING: connecting to postgres > WARNING: Initiating logical rep > LOG: comput

Re: [HACKERS] logical changeset generation v3

2012-11-21 Thread Andres Freund
On 2012-11-21 18:35:34 +0900, Michael Paquier wrote: > On Wed, Nov 21, 2012 at 4:34 PM, Andres Freund wrote: > > > On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: > > > > Ah, I see. Could you try the following diff? > > > > diff --git a/src/backend/replication/logical/snapbuild.c > > b/src/ba

Re: [HACKERS] logical changeset generation v3

2012-11-21 Thread Michael Paquier
On Wed, Nov 21, 2012 at 4:34 PM, Andres Freund wrote: > On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: > > Ah, I see. Could you try the following diff? > > diff --git a/src/backend/replication/logical/snapbuild.c > b/src/backend/replication/logical/snapbuild.c > index df24b33..797a126 100644

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-21 16:47:11 +0900, Michael Paquier wrote: > On Wed, Nov 21, 2012 at 4:30 PM, Andres Freund wrote: > > > On 2012-11-21 15:28:30 +0900, Michael Paquier wrote: > > > On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund > >wrote: > > > > > > > On 2012-11-20 09:30:40 +0900, Michael Paquier wrote:

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Michael Paquier
On Wed, Nov 21, 2012 at 4:30 PM, Andres Freund wrote: > On 2012-11-21 15:28:30 +0900, Michael Paquier wrote: > > On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund >wrote: > > > > > On 2012-11-20 09:30:40 +0900, Michael Paquier wrote: > > > > Btw, here are some extra comments based on my progress, ho

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: > On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund wrote: > > > Those aren't unexpected. Perhaps I should not make it a warning then... > > > > A short explanation: > > > > We can only decode tuples we see in the WAL when we already have a > > tim

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Michael Paquier
On Wed, Nov 21, 2012 at 4:31 PM, Andres Freund wrote: > On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: > > On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund >wrote: > > > It implies that snapstate->nrrunning has lost touch with reality... > > > > > Yes, I can reproduce in 10-20 seconds in one

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-21 14:57:08 +0900, Michael Paquier wrote: > On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund wrote: > > > Those aren't unexpected. Perhaps I should not make it a warning then... > > > > A short explanation: > > > > We can only decode tuples we see in the WAL when we already have a > > tim

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-21 15:28:30 +0900, Michael Paquier wrote: > On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund wrote: > > > On 2012-11-20 09:30:40 +0900, Michael Paquier wrote: > > > Btw, here are some extra comments based on my progress, hope it will be > > > useful for other people playing around with you

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Michael Paquier
On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund wrote: > On 2012-11-20 09:30:40 +0900, Michael Paquier wrote: > > On Mon, Nov 19, 2012 at 5:50 PM, Andres Freund >wrote: > > > On 2012-11-19 16:28:55 +0900, Michael Paquier wrote: > > > > I am just looking at this patch and will provide some comments

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Michael Paquier
On Tue, Nov 20, 2012 at 8:22 PM, Andres Freund wrote: > Those aren't unexpected. Perhaps I should not make it a warning then... > > A short explanation: > > We can only decode tuples we see in the WAL when we already have a > timetravel catalog snapshot before that transaction started. To build >

Re: [HACKERS] logical changeset generation v3 - Source for Slony

2012-11-20 Thread Andres Freund
Hi, On 2012-11-19 19:50:32 -0500, Steve Singer wrote: > On 12-11-18 11:07 AM, Andres Freund wrote: > >I think we should provide some glue code to do this, otherwise people > >will start replicating all the bugs I hacked into this... More > >seriously: I think we should have support code here, no u

Re: [HACKERS] logical changeset generation v3

2012-11-20 Thread Andres Freund
On 2012-11-20 09:30:40 +0900, Michael Paquier wrote: > On Mon, Nov 19, 2012 at 5:50 PM, Andres Freund wrote: > > On 2012-11-19 16:28:55 +0900, Michael Paquier wrote: > > > I am just looking at this patch and will provide some comments. > > > By the way, you forgot the installation part of pg_receiv

Re: [HACKERS] logical changeset generation v3 - Source for Slony

2012-11-19 Thread Steve Singer
On 12-11-18 11:07 AM, Andres Freund wrote: Hi Steve! I think we should provide some glue code to do this, otherwise people will start replicating all the bugs I hacked into this... More seriously: I think we should have support code here, no user will want to learn the intracacies of feedback m

Re: [HACKERS] logical changeset generation v3

2012-11-19 Thread Michael Paquier
On Mon, Nov 19, 2012 at 5:50 PM, Andres Freund wrote: > Hi Michael, > > > On 2012-11-19 16:28:55 +0900, Michael Paquier wrote: > > I have been able to fetch your code (thanks Andrea!) and some it. For the > > time being I am spending some time reading the code and understanding the > > whole set o

Re: [HACKERS] logical changeset generation v3 - Source for Slony

2012-11-19 Thread Andres Freund
Hi Steve! On 2012-11-17 22:50:35 -0500, Steve Singer wrote: > First, you can add me to the list of people saying 'wow', I'm impressed. Thanks! > The approach I am taking to reviewing this to try and answer the following > question > > 1) How might a future version of slony be able to use logical

Re: [HACKERS] logical changeset generation v3 - Source for Slony

2012-11-19 Thread Steve Singer
First, you can add me to the list of people saying 'wow', I'm impressed. The approach I am taking to reviewing this to try and answer the following question 1) How might a future version of slony be able to use logical replication as described by your patch and design documents and what woul

Re: [HACKERS] logical changeset generation v3

2012-11-19 Thread Andres Freund
Hi Michael, On 2012-11-19 16:28:55 +0900, Michael Paquier wrote: > I have been able to fetch your code (thanks Andrea!) and some it. For the > time being I am spending some time reading the code and understanding the > whole set of features you are trying to implement inside core, even if I > got

Re: [HACKERS] logical changeset generation v3

2012-11-18 Thread Michael Paquier
Hi Andres, I have been able to fetch your code (thanks Andrea!) and some it. For the time being I am spending some time reading the code and understanding the whole set of features you are trying to implement inside core, even if I got some background from what you presented at PGCon and from the

Re: [HACKERS] logical changeset generation v3

2012-11-18 Thread Michael Paquier
On Fri, Nov 16, 2012 at 5:16 PM, Andrea Suisani wrote: > Il 16/11/2012 05:34, Michael Paquier ha scritto: > > Do you have a git repository or something where all the 14 patches are >> applied? I would like to test the feature globally. >> Sorry I recall that you put a link somewhere but I cannot

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Markus Wanner
Hannu, On 11/17/2012 03:40 PM, Hannu Krosing wrote: > On 11/17/2012 03:00 PM, Markus Wanner wrote: >> On 11/17/2012 02:30 PM, Hannu Krosing wrote: >>> Is it possible to replicate UPDATEs and DELETEs without a primary key in >>> PostgreSQL-R >> No. There must be some way to logically identify the t

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Hannu Krosing
On 11/17/2012 03:00 PM, Markus Wanner wrote: On 11/17/2012 02:30 PM, Hannu Krosing wrote: Is it possible to replicate UPDATEs and DELETEs without a primary key in PostgreSQL-R No. There must be some way to logically identify the tuple. Note, though, that theoretically any (unconditional) unique

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Hannu Krosing
On 11/17/2012 03:00 PM, Markus Wanner wrote: On 11/17/2012 02:30 PM, Hannu Krosing wrote: Is it possible to replicate UPDATEs and DELETEs without a primary key in PostgreSQL-R No. There must be some way to logically identify the tuple. It can be done as selecting on _all_ attributes and updatin

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Markus Wanner
On 11/17/2012 02:30 PM, Hannu Krosing wrote: > Is it possible to replicate UPDATEs and DELETEs without a primary key in > PostgreSQL-R No. There must be some way to logically identify the tuple. Note, though, that theoretically any (unconditional) unique key would suffice. In practice, that usuall

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Hannu Krosing
On 11/16/2012 02:46 PM, Markus Wanner wrote: Andres, On 11/15/2012 01:27 AM, Andres Freund wrote: In response to this you will soon find the 14 patches that currently implement $subject. Congratulations on that piece of work. I'd like to provide a comparison of the proposed change set format

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Markus Wanner
On 11/16/2012 03:14 PM, Andres Freund wrote: > Whats the data type of the "COID" in -R? It's short for CommitOrderId, a 32bit global transaction identifier, being wrapped-around, very much like TransactionIds are. (In that sense, it's global, but unique only for a certain amount of time). > In th

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-17 Thread Markus Wanner
On 11/16/2012 03:05 PM, Andres Freund wrote: >> I'd like to provide a comparison of the proposed change set format to >> the one used in Postgres-R. > > Uh, sorry to interrupt you right here, but thats not the "proposed > format" ;) Understood. Sorry, I didn't mean to imply that. It's pretty obvi

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-16 Thread Andres Freund
Hi, On 2012-11-16 14:46:39 +0100, Markus Wanner wrote: > You may have noticed that there's an additional COID field. This is an > identifier for the transaction that last changed this tuple. Together > with the primary key, it effectively identifies the exact version of a > tuple (during its lifet

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-16 Thread Andres Freund
Hi Markus, On 2012-11-16 14:46:39 +0100, Markus Wanner wrote: > On 11/15/2012 01:27 AM, Andres Freund wrote: > > In response to this you will soon find the 14 patches that currently > > implement $subject. > > Congratulations on that piece of work. Thanks. > I'd like to provide a comparison of t

Re: [HACKERS] logical changeset generation v3 - comparison to Postgres-R change set format

2012-11-16 Thread Markus Wanner
Andres, On 11/15/2012 01:27 AM, Andres Freund wrote: > In response to this you will soon find the 14 patches that currently > implement $subject. Congratulations on that piece of work. I'd like to provide a comparison of the proposed change set format to the one used in Postgres-R. I hope for t

Re: [HACKERS] logical changeset generation v3

2012-11-16 Thread Andrea Suisani
Il 16/11/2012 05:34, Michael Paquier ha scritto: Do you have a git repository or something where all the 14 patches are applied? I would like to test the feature globally. Sorry I recall that you put a link somewhere but I cannot remember its email... http://archives.postgresql.org/pgsql-hacke

Re: [HACKERS] logical changeset generation v3

2012-11-15 Thread Michael Paquier
Do you have a git repository or something where all the 14 patches are applied? I would like to test the feature globally. Sorry I recall that you put a link somewhere but I cannot remember its email... On Thu, Nov 15, 2012 at 6:34 PM, Andres Freund wrote: > Hi, > > On Thursday, November 15, 201

Re: [HACKERS] logical changeset generation v3

2012-11-15 Thread Andres Freund
Hi, On Thursday, November 15, 2012 05:08:26 AM Michael Paquier wrote: > Looks like cool stuff @-@ > I might be interested in looking at that a bit as I think I will hopefully > be hopefully be able to grab some time in the next couple of weeks. > Are some of those patches already submitted to a CF

Re: [HACKERS] logical changeset generation v3

2012-11-14 Thread Michael Paquier
Looks like cool stuff @-@ I might be interested in looking at that a bit as I think I will hopefully be hopefully be able to grab some time in the next couple of weeks. Are some of those patches already submitted to a CF? -- Michael Paquier http://michael.otacoo.com

Re: [HACKERS] logical changeset generation v3

2012-11-14 Thread Josh Berkus
On 11/14/12 4:27 PM, Andres Freund wrote: > Hi, > > In response to this you will soon find the 14 patches that currently > implement $subject. I'll go over each one after showing off for a bit: Lemme be the first to say, "wow". Impressive work. Now the debugging starts ... -- Josh Berkus Post

Re: [HACKERS] logical changeset generation v3 - git repository

2012-11-14 Thread Andres Freund
On 2012-11-15 01:27:46 +0100, Andres Freund wrote: > In response to this you will soon find the 14 patches that currently > implement $subject. As its not very wieldly to send around that many/big patches all the time, until the next "major" version I will just update the git tree at: Web: http:/