Re: [HACKERS] Changeset Extraction Interfaces

2014-01-08 Thread Andres Freund
On 2014-01-07 17:54:21 +0100, Andres Freund wrote: > On 2013-12-12 16:49:33 +0100, Andres Freund wrote: > > On 2013-12-12 10:01:21 -0500, Robert Haas wrote: > > > On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund > > > wrote: > > > > As far as I have been thinking of, this would be another catalog t

Re: [HACKERS] Changeset Extraction Interfaces

2014-01-07 Thread Andres Freund
On 2013-12-12 16:49:33 +0100, Andres Freund wrote: > On 2013-12-12 10:01:21 -0500, Robert Haas wrote: > > On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund > > wrote: > > > As far as I have been thinking of, this would be another catalog table > > > like > > > pg_decoding_plugin(oid, dpname name, d

Re: [HACKERS] Changeset Extraction Interfaces

2014-01-04 Thread Robert Haas
On Fri, Jan 3, 2014 at 10:12 AM, Andres Freund wrote: > On 2013-12-12 10:01:21 -0500, Robert Haas wrote: >> On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund >> wrote: >> > I think there'll always be a bit of a difference between slots for >> > physical and logical data, even if 90% of the implemen

Re: [HACKERS] Changeset Extraction Interfaces

2014-01-03 Thread Andres Freund
On 2013-12-12 10:01:21 -0500, Robert Haas wrote: > On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund wrote: > > I think there'll always be a bit of a difference between slots for > > physical and logical data, even if 90% of the implementation is the > > same. We can signal that difference by specify

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-17 Thread Robert Haas
On Tue, Dec 17, 2013 at 4:31 AM, Andres Freund wrote: > On 2013-12-16 23:01:16 -0500, Robert Haas wrote: >> On Sat, Dec 14, 2013 at 12:37 PM, Andres Freund >> wrote: >> > On 2013-12-14 11:50:00 -0500, Robert Haas wrote: >> >> Well, it still seems to me that the right way to think about this is >

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-17 Thread Andres Freund
On 2013-12-16 23:01:16 -0500, Robert Haas wrote: > On Sat, Dec 14, 2013 at 12:37 PM, Andres Freund > wrote: > > On 2013-12-14 11:50:00 -0500, Robert Haas wrote: > >> Well, it still seems to me that the right way to think about this is > >> that the change stream begins at a certain point, and the

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-16 Thread Robert Haas
On Sat, Dec 14, 2013 at 12:37 PM, Andres Freund wrote: > On 2013-12-14 11:50:00 -0500, Robert Haas wrote: >> On Fri, Dec 13, 2013 at 9:14 AM, Andres Freund >> wrote: >> > But can you imagine those users needing an exported snapshot? I can >> > think of several short-lived usages, but all of thos

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-14 Thread Jim Nasby
On 12/12/13 11:13 AM, Robert Haas wrote: I think it sucks (that's the technical term) to have to wait for all currently-running transactions to terminate before being able to begin streaming changes, because that could take a long time. And you might well know that the long-running transaction w

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-14 Thread Andres Freund
On 2013-12-14 11:50:00 -0500, Robert Haas wrote: > On Fri, Dec 13, 2013 at 9:14 AM, Andres Freund wrote: > > But can you imagine those users needing an exported snapshot? I can > > think of several short-lived usages, but all of those are unlikely to > > need a consistent view of the overall datab

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-14 Thread Robert Haas
On Fri, Dec 13, 2013 at 9:14 AM, Andres Freund wrote: >> If you imagine a scenario where somebody establishes a replication >> slot and then keeps it forever, not often. But if you're trying to do >> something more ad hoc, where replication slots might be used just for >> short periods of time an

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-13 Thread Andres Freund
On 2013-12-13 08:30:41 -0500, Robert Haas wrote: > On Thu, Dec 12, 2013 at 1:52 PM, Andres Freund wrote: > > Puh. I honestly have zero confidence in DBAs making an informed decision > > about something like this. Honestly, for a replication solution, how > > often do you think this will be an issu

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-13 Thread Robert Haas
On Thu, Dec 12, 2013 at 1:52 PM, Andres Freund wrote: > Puh. I honestly have zero confidence in DBAs making an informed decision > about something like this. Honestly, for a replication solution, how > often do you think this will be an issue? If you imagine a scenario where somebody establishes

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Andres Freund
On 2013-12-12 12:13:24 -0500, Robert Haas wrote: > On Thu, Dec 12, 2013 at 10:49 AM, Andres Freund > wrote: > > If we were to start out streaming changes before the last running > > transaction has finished, they would be visible in that exported > > snapshot and you couldn't use it to to roll fo

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Robert Haas
On Thu, Dec 12, 2013 at 10:49 AM, Andres Freund wrote: >> I hadn't realized that the options were going to be different for >> logical vs. physical. > > I don't see how we could avoid that, there just are some differences > between both. Right, I'm not complaining, just observing that it was a po

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Andres Freund
On 2013-12-12 10:01:21 -0500, Robert Haas wrote: > On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund wrote: > > I think there'll always be a bit of a difference between slots for > > physical and logical data, even if 90% of the implementation is the > > same. We can signal that difference by specify

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Robert Haas
On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund wrote: > I think there'll always be a bit of a difference between slots for > physical and logical data, even if 90% of the implementation is the > same. We can signal that difference by specifying logical/physical as an > option or having two differe

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Andres Freund
Hello Robert, On 2013-12-11 22:29:46 -0500, Robert Haas wrote: > On Wed, Dec 4, 2013 at 7:15 PM, Andres Freund wrote: > > There's basically three major 'verbs' that can be performed on a > > stream, currently named (walsender names): > > * INIT_LOGICAL_REPLICATION "name" "output_plugin" > > * STA

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-11 Thread Robert Haas
On Wed, Dec 4, 2013 at 7:15 PM, Andres Freund wrote: > There's basically three major 'verbs' that can be performed on a > stream, currently named (walsender names): > * INIT_LOGICAL_REPLICATION "name" "output_plugin" > * START_LOGICAL_REPLICATION "name" last_received ("option_name" value,...) > *

[HACKERS] Changeset Extraction Interfaces

2013-12-04 Thread Andres Freund
Hi, Short recap: >From the perspective of the user interface the changeset extraction feature consists out of two abstract interfaces that the "user" has to do with: 1) The "slot" or "changestream" management interface which manages individual streams of changes. The user can create and destroy