Re: [HACKERS] Replication identifiers, take 3

2014-10-29 Thread Simon Riggs
On 2 October 2014 09:49, Heikki Linnakangas hlinnakan...@vmware.com wrote: What I've previously suggested (and which works well in BDR) is to add the internal id to the XLogRecord struct. There's 2 free bytes of padding that can be used for that purpose. Adding a field to XLogRecord for

Re: [HACKERS] Replication identifiers, take 3

2014-10-04 Thread Jim Nasby
On 10/2/14, 7:28 AM, Robert Haas wrote: On Thu, Oct 2, 2014 at 4:49 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: An origin column in the table itself helps tremendously to debug issues with the replication system. In many if not most scenarios, I think you'd want to have that extra

Re: [HACKERS] Replication identifiers, take 3

2014-10-02 Thread Heikki Linnakangas
On 09/23/2014 09:24 PM, Andres Freund wrote: I've previously started two threads about replication identifiers. Check http://archives.postgresql.org/message-id/20131114172632.GE7522%40alap2.anarazel.de and http://archives.postgresql.org/message-id/20131211153833.GB25227%40awork2.anarazel.de .

Re: [HACKERS] Replication identifiers, take 3

2014-10-02 Thread Andres Freund
On 2014-10-02 11:49:31 +0300, Heikki Linnakangas wrote: On 09/23/2014 09:24 PM, Andres Freund wrote: I've previously started two threads about replication identifiers. Check http://archives.postgresql.org/message-id/20131114172632.GE7522%40alap2.anarazel.de and

Re: [HACKERS] Replication identifiers, take 3

2014-10-02 Thread Robert Haas
On Thu, Oct 2, 2014 at 4:49 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: An origin column in the table itself helps tremendously to debug issues with the replication system. In many if not most scenarios, I think you'd want to have that extra column, even if it's not strictly required.

Re: [HACKERS] Replication identifiers, take 3

2014-09-29 Thread Andres Freund
On 2014-09-27 12:11:16 -0400, Steve Singer wrote: On 09/26/2014 06:05 PM, Andres Freund wrote: On 2014-09-26 14:57:12 -0400, Robert Haas wrote: Sure, it'll possibly not be trivial to move them elsewhere. On the other hand, the padding bytes have been unused for 8+ years without somebody

Re: [HACKERS] Replication identifiers, take 3

2014-09-29 Thread Robert Haas
On Sat, Sep 27, 2014 at 12:11 PM, Steve Singer st...@ssinger.info wrote: If we were now increasing the WAL record size anyway for some unrelated reason, would we be willing to increase it by a further 2 bytes for the node identifier? Obviously not. Otherwise Andres would be proposing to put

Re: [HACKERS] Replication identifiers, take 3

2014-09-27 Thread Steve Singer
On 09/26/2014 06:05 PM, Andres Freund wrote: On 2014-09-26 14:57:12 -0400, Robert Haas wrote: Sure, it'll possibly not be trivial to move them elsewhere. On the other hand, the padding bytes have been unused for 8+ years without somebody laying claim on them but me. I don't think it's a good

Re: [HACKERS] Replication identifiers, take 3

2014-09-27 Thread Steve Singer
On 09/26/2014 10:21 AM, Andres Freund wrote: On 2014-09-26 09:53:09 -0400, Robert Haas wrote: On Fri, Sep 26, 2014 at 5:05 AM, Andres Freund and...@2ndquadrant.com wrote: Let me try to summarize the information requirements for each of these things. For #1, you need to know, after crash

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Petr Jelinek
On 26/09/14 04:44, Robert Haas wrote: On Tue, Sep 23, 2014 at 2:24 PM, Andres Freund and...@2ndquadrant.com wrote: Note that it depends on the replication solution whether these external identifiers need to be coordinated across systems or not. I think it's *good* if we don't propose a

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Andres Freund
On 2014-09-25 22:44:49 -0400, Robert Haas wrote: Thanks for this write-up. On Tue, Sep 23, 2014 at 2:24 PM, Andres Freund and...@2ndquadrant.com wrote: 1) The ability Monitor how for replication has progressed in a crashsafe manner to allow it to restart at the right point after

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Robert Haas
On Fri, Sep 26, 2014 at 5:05 AM, Andres Freund and...@2ndquadrant.com wrote: Let me try to summarize the information requirements for each of these things. For #1, you need to know, after crash recovery, for each standby, the last commit LSN which the client has confirmed via a feedback

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Andres Freund
On 2014-09-26 09:53:09 -0400, Robert Haas wrote: On Fri, Sep 26, 2014 at 5:05 AM, Andres Freund and...@2ndquadrant.com wrote: Let me try to summarize the information requirements for each of these things. For #1, you need to know, after crash recovery, for each standby, the last commit LSN

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Robert Haas
On Fri, Sep 26, 2014 at 10:21 AM, Andres Freund and...@2ndquadrant.com wrote: As explained above this isn't happening on the level of individual AMs. Well, that's even worse. You want to grab 100% of the available generic bitspace applicable to all record types for purposes specific to logical

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Robert Haas
On Fri, Sep 26, 2014 at 10:55 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-26 10:40:37 -0400, Robert Haas wrote: On Fri, Sep 26, 2014 at 10:21 AM, Andres Freund and...@2ndquadrant.com wrote: As explained above this isn't happening on the level of individual AMs. Well, that's

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Andres Freund
On 2014-09-26 11:02:16 -0400, Robert Haas wrote: On Fri, Sep 26, 2014 at 10:55 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-09-26 10:40:37 -0400, Robert Haas wrote: On Fri, Sep 26, 2014 at 10:21 AM, Andres Freund and...@2ndquadrant.com wrote: As explained above this isn't

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Robert Haas
On Fri, Sep 26, 2014 at 12:32 PM, Andres Freund and...@2ndquadrant.com wrote: Huh? That's just to say that the unused bit space is, in fact, unused. But so what? We've always been very careful about using up things like infomask bits, because there are only so many bits available, and when

Re: [HACKERS] Replication identifiers, take 3

2014-09-26 Thread Andres Freund
On 2014-09-26 14:57:12 -0400, Robert Haas wrote: On Fri, Sep 26, 2014 at 12:32 PM, Andres Freund and...@2ndquadrant.com wrote: Huh? That's just to say that the unused bit space is, in fact, unused. But so what? We've always been very careful about using up things like infomask bits,

Re: [HACKERS] Replication identifiers, take 3

2014-09-25 Thread Robert Haas
Thanks for this write-up. On Tue, Sep 23, 2014 at 2:24 PM, Andres Freund and...@2ndquadrant.com wrote: 1) The ability Monitor how for replication has progressed in a crashsafe manner to allow it to restart at the right point after errors/crashes. 2) Efficiently identify the origin of

[HACKERS] Replication identifiers, take 3

2014-09-23 Thread Andres Freund
Hi, I've previously started two threads about replication identifiers. Check http://archives.postgresql.org/message-id/20131114172632.GE7522%40alap2.anarazel.de and http://archives.postgresql.org/message-id/20131211153833.GB25227%40awork2.anarazel.de . The've also been discussed in the course of