Re: [HACKERS] Changeset Extraction Interfaces

2014-01-08 Thread Andres Freund
On 2014-01-07 17:54:21 +0100, Andres Freund wrote: On 2013-12-12 16:49:33 +0100, Andres Freund wrote: On 2013-12-12 10:01:21 -0500, Robert Haas wrote: On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund and...@2ndquadrant.com wrote: As far as I have been thinking of, this would be another

Re: [HACKERS] Changeset Extraction Interfaces

2014-01-07 Thread Andres Freund
On 2013-12-12 16:49:33 +0100, Andres Freund wrote: On 2013-12-12 10:01:21 -0500, Robert Haas wrote: On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund and...@2ndquadrant.com wrote: As far as I have been thinking of, this would be another catalog table like pg_decoding_plugin(oid,

Re: [HACKERS] Changeset Extraction Interfaces

2014-01-04 Thread Robert Haas
On Fri, Jan 3, 2014 at 10:12 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-12 10:01:21 -0500, Robert Haas wrote: On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund and...@2ndquadrant.com wrote: I think there'll always be a bit of a difference between slots for physical and logical

Re: [HACKERS] Changeset Extraction Interfaces

2014-01-03 Thread Andres Freund
On 2013-12-12 10:01:21 -0500, Robert Haas wrote: On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund and...@2ndquadrant.com wrote: I think there'll always be a bit of a difference between slots for physical and logical data, even if 90% of the implementation is the same. We can signal that

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-17 Thread Andres Freund
On 2013-12-16 23:01:16 -0500, Robert Haas wrote: On Sat, Dec 14, 2013 at 12:37 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-14 11:50:00 -0500, Robert Haas wrote: Well, it still seems to me that the right way to think about this is that the change stream begins at a certain

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-17 Thread Robert Haas
On Tue, Dec 17, 2013 at 4:31 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-16 23:01:16 -0500, Robert Haas wrote: On Sat, Dec 14, 2013 at 12:37 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-14 11:50:00 -0500, Robert Haas wrote: Well, it still seems to me that the

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-16 Thread Robert Haas
On Sat, Dec 14, 2013 at 12:37 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-14 11:50:00 -0500, Robert Haas wrote: On Fri, Dec 13, 2013 at 9:14 AM, Andres Freund and...@2ndquadrant.com wrote: But can you imagine those users needing an exported snapshot? I can think of several

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-14 Thread Robert Haas
On Fri, Dec 13, 2013 at 9:14 AM, Andres Freund and...@2ndquadrant.com wrote: If you imagine a scenario where somebody establishes a replication slot and then keeps it forever, not often. But if you're trying to do something more ad hoc, where replication slots might be used just for short

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-14 Thread Andres Freund
On 2013-12-14 11:50:00 -0500, Robert Haas wrote: On Fri, Dec 13, 2013 at 9:14 AM, Andres Freund and...@2ndquadrant.com wrote: But can you imagine those users needing an exported snapshot? I can think of several short-lived usages, but all of those are unlikely to need a consistent view of

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-14 Thread Jim Nasby
On 12/12/13 11:13 AM, Robert Haas wrote: I think it sucks (that's the technical term) to have to wait for all currently-running transactions to terminate before being able to begin streaming changes, because that could take a long time. And you might well know that the long-running transaction

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-13 Thread Robert Haas
On Thu, Dec 12, 2013 at 1:52 PM, Andres Freund and...@2ndquadrant.com wrote: Puh. I honestly have zero confidence in DBAs making an informed decision about something like this. Honestly, for a replication solution, how often do you think this will be an issue? If you imagine a scenario where

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-13 Thread Andres Freund
On 2013-12-13 08:30:41 -0500, Robert Haas wrote: On Thu, Dec 12, 2013 at 1:52 PM, Andres Freund and...@2ndquadrant.com wrote: Puh. I honestly have zero confidence in DBAs making an informed decision about something like this. Honestly, for a replication solution, how often do you think this

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Andres Freund
Hello Robert, On 2013-12-11 22:29:46 -0500, Robert Haas wrote: On Wed, Dec 4, 2013 at 7:15 PM, Andres Freund and...@2ndquadrant.com wrote: There's basically three major 'verbs' that can be performed on a stream, currently named (walsender names): * INIT_LOGICAL_REPLICATION name

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Robert Haas
On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund and...@2ndquadrant.com wrote: I think there'll always be a bit of a difference between slots for physical and logical data, even if 90% of the implementation is the same. We can signal that difference by specifying logical/physical as an option or

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Andres Freund
On 2013-12-12 10:01:21 -0500, Robert Haas wrote: On Thu, Dec 12, 2013 at 7:04 AM, Andres Freund and...@2ndquadrant.com wrote: I think there'll always be a bit of a difference between slots for physical and logical data, even if 90% of the implementation is the same. We can signal that

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Robert Haas
On Thu, Dec 12, 2013 at 10:49 AM, Andres Freund and...@2ndquadrant.com wrote: I hadn't realized that the options were going to be different for logical vs. physical. I don't see how we could avoid that, there just are some differences between both. Right, I'm not complaining, just observing

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-12 Thread Andres Freund
On 2013-12-12 12:13:24 -0500, Robert Haas wrote: On Thu, Dec 12, 2013 at 10:49 AM, Andres Freund and...@2ndquadrant.com wrote: If we were to start out streaming changes before the last running transaction has finished, they would be visible in that exported snapshot and you couldn't use

Re: [HACKERS] Changeset Extraction Interfaces

2013-12-11 Thread Robert Haas
On Wed, Dec 4, 2013 at 7:15 PM, Andres Freund and...@2ndquadrant.com wrote: There's basically three major 'verbs' that can be performed on a stream, currently named (walsender names): * INIT_LOGICAL_REPLICATION name output_plugin * START_LOGICAL_REPLICATION name last_received (option_name

[HACKERS] Changeset Extraction Interfaces

2013-12-04 Thread Andres Freund
Hi, Short recap: From the perspective of the user interface the changeset extraction feature consists out of two abstract interfaces that the user has to do with: 1) The slot or changestream management interface which manages individual streams of changes. The user can create and destroy a