Re: [HACKERS] WIP: Failover Slots

On 14 August 2017 at 11:56, Craig Ringer wrote: > > I don't want to block failover slots on decoding on standby just because > decoding on standby would be nice to have. > However, during discussion with Tomas Munro a point has come up that does block failover slots as

Re: [HACKERS] WIP: Failover Slots

On 12 August 2017 at 08:03, Andres Freund wrote: > On 2017-08-02 16:35:17 -0400, Robert Haas wrote: > > I actually think failover slots are quite desirable, especially now > > that we've got logical replication in core. In a review of this > > thread I don't see anyone

Re: [HACKERS] WIP: Failover Slots

On 2017-08-02 16:35:17 -0400, Robert Haas wrote: > I actually think failover slots are quite desirable, especially now > that we've got logical replication in core. In a review of this > thread I don't see anyone saying otherwise. The debate has really > been about the right way of implementing

Re: [HACKERS] WIP: Failover Slots

On 11 August 2017 at 01:02, Robert Haas wrote: > Well, > anybody's welcome to write code without discussion and drop it to the > list, but if people don't like it, that's the risk you took by not > discussing it first. > Agreed, patches materializing doesn't mean they

Re: [HACKERS] WIP: Failover Slots

On Thu, Aug 10, 2017 at 2:38 AM, Craig Ringer wrote: > Yep, so again, you're pushing slots "up" the tree, by name, with a 1:1 > correspondence, and using globally unique slot names to manage state. Yes, that's what I'm imagining. (Whether I should instead be imagining

Re: [HACKERS] WIP: Failover Slots

On 9 August 2017 at 23:42, Robert Haas wrote: > On Tue, Aug 8, 2017 at 4:00 AM, Craig Ringer > wrote: > >> - When a standby connects to a master, it can optionally supply a list > >> of slot names that it cares about. > > > > Wouldn't that

Re: [HACKERS] WIP: Failover Slots

On Tue, Aug 8, 2017 at 4:00 AM, Craig Ringer wrote: >> - When a standby connects to a master, it can optionally supply a list >> of slot names that it cares about. > > Wouldn't that immediately exclude use for PITR and snapshot recovery? I have > people right now who want

Re: [HACKERS] WIP: Failover Slots

On 3 August 2017 at 04:35, Robert Haas wrote: > On Tue, Jul 25, 2017 at 8:44 PM, Craig Ringer > wrote: > > No. The whole approach seems to have been bounced from core. I don't > agree > > and continue to think this functionality is desirable but I

Re: [HACKERS] WIP: Failover Slots

On Tue, Jul 25, 2017 at 8:44 PM, Craig Ringer wrote: > No. The whole approach seems to have been bounced from core. I don't agree > and continue to think this functionality is desirable but I don't get to > make that call. I actually think failover slots are quite

Re: [HACKERS] WIP: Failover Slots

On 26 July 2017 at 00:16, Thom Brown wrote: > On 8 April 2016 at 07:13, Craig Ringer wrote: > > On 6 April 2016 at 22:17, Andres Freund wrote: > > > >> > >> Quickly skimming 0001 in [4] there appear to be a number of issues: > >> *

Re: [HACKERS] WIP: Failover Slots

On 8 April 2016 at 07:13, Craig Ringer wrote: > On 6 April 2016 at 22:17, Andres Freund wrote: > >> >> Quickly skimming 0001 in [4] there appear to be a number of issues: >> * LWLockHeldByMe() is only for debugging, not functional differences >> *

Re: [HACKERS] WIP: Failover Slots

On 6 April 2016 at 22:17, Andres Freund wrote: > Quickly skimming 0001 in [4] there appear to be a number of issues: > * LWLockHeldByMe() is only for debugging, not functional differences > * ReplicationSlotPersistentData is now in an xlog related header > * The code and

Re: [HACKERS] WIP: Failover Slots

On 6 April 2016 at 15:17, Andres Freund wrote: > On 2016-04-06 14:30:21 +0100, Simon Riggs wrote: > > On 6 April 2016 at 14:15, Craig Ringer wrote: > > ... > > > > Nice summary > > > > Failover slots are optional. And they work on master. > > > >

Re: [HACKERS] WIP: Failover Slots

On 2016-04-06 14:30:21 +0100, Simon Riggs wrote: > On 6 April 2016 at 14:15, Craig Ringer wrote: > ... > > Nice summary > > Failover slots are optional. And they work on master. > > While the other approach could also work, it will work later and still > require a slot

Re: [HACKERS] WIP: Failover Slots

On 6 April 2016 at 14:15, Craig Ringer wrote: ... Nice summary Failover slots are optional. And they work on master. While the other approach could also work, it will work later and still require a slot on the master. => I don't see why having Failover Slots in 9.6

Re: [HACKERS] WIP: Failover Slots

A few thoughts on failover slots vs the alternative of pushing catalog_xmin up to the master via a replica's slot and creating independent slots on replicas. Failover slots: --- + Failover slots are very easy for applications. They "just work" and are transparent for failover. This is great

Re: [HACKERS] WIP: Failover Slots

On 6 April 2016 at 17:43, Simon Riggs wrote: > On 25 January 2016 at 14:25, Craig Ringer wrote: > > >> I'd like to get failover slots in place for 9.6 since the're fairly >> self-contained and meet an immediate need: allowing replication using slots

Re: [HACKERS] WIP: Failover Slots

On 25 January 2016 at 14:25, Craig Ringer wrote: > I'd like to get failover slots in place for 9.6 since the're fairly > self-contained and meet an immediate need: allowing replication using slots > (physical or logical) to follow a failover event. > I'm a bit confused

Re: [HACKERS] WIP: Failover Slots

On 5 April 2016 at 04:19, Oleksii Kliukin wrote: > Thank you for the update. I’ve got some rejects when applying the > 0001-Allow-replication-slots-to-follow-failover.patch after the "Dirty > replication slots when confirm_lsn is changed” changes. I think it should > be

Re: [HACKERS] WIP: Failover Slots

Hi, > On 17 Mar 2016, at 09:34, Craig Ringer wrote: > > OK, here's the latest failover slots patch, rebased on top of today's master > plus, in order: > > - Dirty replication slots when confirm_lsn is changed > >

Re: [HACKERS] WIP: Failover Slots

On 15 March 2016 at 21:40, Craig Ringer wrote: > Here's a new failover slots rev, addressing the issues Oleksii Kliukin > raised and adding a bunch of TAP tests. > Ahem, just found an issue here. I'll need to send another revision. -- Craig Ringer

Re: [HACKERS] WIP: Failover Slots

Here's a new failover slots rev, addressing the issues Oleksii Kliukin raised and adding a bunch of TAP tests. In particular, for the checkpoint issue I landed up moving CheckPointReplicationSlots to occur at the start of a checkpoint, before writing WAL is prohibited. As the comments note it's

Re: [HACKERS] WIP: Failover Slots

On 24 February 2016 at 18:02, Craig Ringer wrote: > I really want to focus on the first patch, timeline following for logical > slots. That part is much less invasive and is useful stand-alone. I'll move > it to a separate CF entry and post it to a separate thread as I

Re: [HACKERS] WIP: Failover Slots

On 24 February 2016 at 03:53, Oleksii Kliukin wrote: > > I found the following issue when shutting down a master with a connected > replica that uses a physical failover slot: > > 2016-02-23 20:33:42.546 CET,,,54998,,56ccb3f3.d6d6,3,,2016-02-23 20:33:07 >

Re: [HACKERS] WIP: Failover Slots

> On 23 Feb 2016, at 11:30, Craig Ringer wrote: > > > Updated patch > > ... attached > > I've split it up a bit more too, so it's easier to tell what change is for > what and fixed the issues mentioned by Oleksii. I've also removed some > unrelated documentation

Re: [HACKERS] WIP: Failover Slots

On 22 February 2016 at 23:39, Oleksii Kliukin wrote: > What it’s doing is calling pg_basebackup first to initialize the replica, > and that actually failed with: > > _basebackup: unexpected termination of replication stream: ERROR: > requested WAL segment

Re: [HACKERS] WIP: Failover Slots

Hi, > On 16 Feb 2016, at 09:11, Craig Ringer wrote: > > > > Revision attached. There was a file missing from the patch too. > All attached patches apply normally. I only took a look at first 2, but also tried to run the Patroni with the modified version to check

Re: [HACKERS] WIP: Failover Slots

Hi, here is my code level review: 0001: This one looks ok except for broken indentation in the new notes in xlogreader.c and .h. It's maybe slightly overdocumented but given the complicated way the timeline reading works it's probably warranted. 0002: +/* + * No

Re: [HACKERS] WIP: Failover Slots

On Fri, Jan 22, 2016 at 11:51 AM, Andres Freund wrote: > I think it's technically quite possible to maintain the required > resources on multiple nodes. The question is how would you configure on > which nodes the resources need to be maintained? I can't come up with a >

Re: [HACKERS] WIP: Failover Slots

Hi all Here's v3 of failover slots. It doesn't add the UI yet, but it's now functionally complete except for timeline following for logical slots, and I have a plan for that. From 533a9327b54ba744b0a1fb0048e8cfe7d3d45ea1 Mon Sep 17 00:00:00 2001 From: Craig Ringer Date:

Re: [HACKERS] WIP: Failover Slots

Hi all Here's v2 of the failover slots patch. It replicates a logical slot to a physical streaming replica downstream, keeping the slots in sync. After the downstream is promoted a client can replay from the logical slot. UI to allow creation of non-failover slots is pending. There's more

Re: [HACKERS] WIP: Failover Slots

On 23 January 2016 at 00:40, Robert Haas wrote: > It occurred to me to wonder if it might be better to > propagate logical slots partially or entirely outside the WAL stream, > because with this design you will end up with the logical slots on > every replica, including

Re: [HACKERS] WIP: Failover Slots

On 23 January 2016 at 00:51, Andres Freund wrote: > Not propagating them through the WAL also has the rather large advantage > of not barring the way to using such slots on standbys. > Yeah. So you could have a read-replica that has a slot and it has child nodes you can

Re: [HACKERS] WIP: Failover Slots

On Fri, Jan 22, 2016 at 2:46 AM, Craig Ringer wrote: > It's also going to be necessary to handle what happens when a new failover > slot (physical or logical) is created on the master where it conflicts with > the name of a non-failover physical slot that was created on the

Re: [HACKERS] WIP: Failover Slots

On 2016-01-22 11:40:24 -0500, Robert Haas wrote: > It occurred to me to wonder if it might be better to > propagate logical slots partially or entirely outside the WAL stream, > because with this design you will end up with the logical slots on > every replica, including cascaded replicas, and I'm

Re: [HACKERS] WIP: Failover Slots

On Fri, Jan 1, 2016 at 7:50 PM, Simon Riggs wrote: > Failover Slots > In the current patch, any slot defined on a master will generate WAL, > leading to a pending-slot being present on all standby nodes. When a standby > is promoted, the slot becomes usable and will have

Re: [HACKERS] WIP: Failover Slots

On 21 January 2016 at 16:31, Robert Haas wrote: > On Fri, Jan 1, 2016 at 7:50 PM, Simon Riggs wrote: > > Failover Slots > > In the current patch, any slot defined on a master will generate WAL, > > leading to a pending-slot being present on all

Re: [HACKERS] WIP: Failover Slots

On 22 January 2016 at 00:31, Robert Haas wrote: > On Fri, Jan 1, 2016 at 7:50 PM, Simon Riggs wrote: > > Failover Slots > > In the current patch, any slot defined on a master will generate WAL, > > leading to a pending-slot being present on all

Re: [HACKERS] WIP: Failover Slots

On 20 January 2016 at 21:02, Craig Ringer wrote: > TL;DR: this doesn't work yet, working on it. > Nothing is logged on slot creation because ReplicationSlot->data.failover is never true. Once that's fixed by - for now - making all slots failover slots, there's a crash in

Re: [HACKERS] WIP: Failover Slots

On 2 January 2016 at 08:50, Simon Riggs wrote: > Patch is WIP, posted for comment, so you can see where I'm going. > I've applied this on a branch of master and posted it, with some comment editorialization, as

[HACKERS] WIP: Failover Slots

Failover Slots If Logical Decoding is taking place from a master, then if we failover and create a new master we would like to continue the Logical Decoding from the new master. To facilitate this, we introduce the concept of “Failover Slots”, which are slots that generate WAL, so that any