Re: [HACKERS] Standalone synchronous master

2014-01-27 Thread Josh Berkus
On 01/26/2014 07:56 PM, Rajeev rastogi wrote: > I shall rework to improve this patch. Below are the summarization of all > discussions, which will be used as input for improving the patch: > > 1. Method of degrading the synchronous mode: > a. Expose the configuration variable to a new SQL-ca

Re: [HACKERS] Standalone synchronous master

2014-01-27 Thread Robert Haas
On Sun, Jan 26, 2014 at 10:56 PM, Rajeev rastogi wrote: > On 01/25/2014, Josh Berkus wrote: >> > ISTM the consensus is that we need better monitoring/administration >> > interfaces so that people can script the behavior they want in >> > external tools. Also, a new synchronous apply replication mo

Re: [HACKERS] Standalone synchronous master

2014-01-26 Thread Rajeev rastogi
On 01/25/2014, Josh Berkus wrote: > > ISTM the consensus is that we need better monitoring/administration > > interfaces so that people can script the behavior they want in > > external tools. Also, a new synchronous apply replication mode would > > be handy, but that'd be a whole different patch.

Re: [HACKERS] Standalone synchronous master

2014-01-26 Thread Hannu Krosing
On 01/24/2014 10:29 PM, Josh Berkus wrote: > On 01/24/2014 12:47 PM, Heikki Linnakangas wrote: >> ISTM the consensus is that we need better monitoring/administration >> interfaces so that people can script the behavior they want in external >> tools. Also, a new synchronous apply replication mode w

Re: [HACKERS] Standalone synchronous master

2014-01-24 Thread Florian Pflug
On Jan24, 2014, at 22:29 , Josh Berkus wrote: > On 01/24/2014 12:47 PM, Heikki Linnakangas wrote: >> ISTM the consensus is that we need better monitoring/administration >> interfaces so that people can script the behavior they want in external >> tools. Also, a new synchronous apply replication mo

Re: [HACKERS] Standalone synchronous master

2014-01-24 Thread Josh Berkus
On 01/24/2014 12:47 PM, Heikki Linnakangas wrote: > ISTM the consensus is that we need better monitoring/administration > interfaces so that people can script the behavior they want in external > tools. Also, a new synchronous apply replication mode would be handy, > but that'd be a whole different

Re: [HACKERS] Standalone synchronous master

2014-01-24 Thread Heikki Linnakangas
ISTM the consensus is that we need better monitoring/administration interfaces so that people can script the behavior they want in external tools. Also, a new synchronous apply replication mode would be handy, but that'd be a whole different patch. We don't have a patch on the table that we cou

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Florian Pflug
On Jan13, 2014, at 22:30 , "Joshua D. Drake" wrote: > On 01/13/2014 01:14 PM, Jim Nasby wrote: >> >> On 1/13/14, 12:21 PM, Joshua D. Drake wrote: >>> >>> On 01/13/2014 10:12 AM, Hannu Krosing wrote: >> In other words, if we're going to have auto-degrade, the most >> intelligent place for

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Joshua D. Drake
On 01/13/2014 01:14 PM, Jim Nasby wrote: On 1/13/14, 12:21 PM, Joshua D. Drake wrote: On 01/13/2014 10:12 AM, Hannu Krosing wrote: In other words, if we're going to have auto-degrade, the most intelligent place for it is in RepMgr/HandyRep/OmniPITR/pgPoolII/whatever. It's also the *easiest*

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Andres Freund
On 2014-01-13 15:14:21 -0600, Jim Nasby wrote: > On 1/13/14, 12:21 PM, Joshua D. Drake wrote: > > > >On 01/13/2014 10:12 AM, Hannu Krosing wrote: > In other words, if we're going to have auto-degrade, the most > intelligent place for it is in > RepMgr/HandyRep/OmniPITR/pgPoolII/whatever

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Jim Nasby
On 1/13/14, 12:21 PM, Joshua D. Drake wrote: On 01/13/2014 10:12 AM, Hannu Krosing wrote: In other words, if we're going to have auto-degrade, the most intelligent place for it is in RepMgr/HandyRep/OmniPITR/pgPoolII/whatever. It's also the *easiest* place. Anything we do *inside* Postgres is

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Joshua D. Drake
On 01/13/2014 10:12 AM, Hannu Krosing wrote: In other words, if we're going to have auto-degrade, the most intelligent place for it is in RepMgr/HandyRep/OmniPITR/pgPoolII/whatever. It's also the *easiest* place. Anything we do *inside* Postgres is going to have a really, really hard time dete

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Hannu Krosing
On 01/13/2014 04:12 PM, Florian Pflug wrote: > On Jan12, 2014, at 04:18 , Josh Berkus wrote: >> Thing is, when we talk about auto-degrade, we need to determine things >> like "Is the replica down or is this just a network blip"? and take >> action according to the user's desired configuration. Th

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Florian Pflug
On Jan12, 2014, at 04:18 , Josh Berkus wrote: > Thing is, when we talk about auto-degrade, we need to determine things > like "Is the replica down or is this just a network blip"? and take > action according to the user's desired configuration. This is not > something, realistically, that we can

Re: [HACKERS] Standalone synchronous master

2014-01-13 Thread Rajeev rastogi
> On Sun, Jan 12, Amit Kapila wrote: > >> How would that work? Would it be a tool in contrib? There already > >> is a timeout, so if a tool checked more frequently than the timeout, > >> it should work. The durable notification of the admin would happen > >> in the tool, right? > > > > Well, y

Re: [HACKERS] Standalone synchronous master

2014-01-12 Thread Rajeev rastogi
On 13th January 2013, Josh Berkus Wrote: > I'm leading this off with a review of the features offered by the > actual patch submitted. My general discussion of the issues of Sync > Degrade, which justifies my specific suggestions below, follows that. > Rajeev, please be aware that other hackers

Re: [HACKERS] Standalone synchronous master

2014-01-12 Thread Amit Kapila
> On 01/11/2014 08:52 PM, Amit Kapila wrote:> It is better than async mode > in a way such that in async mode it never >> waits for commits to be written to standby, but in this new mode it will >> do so unless it is not possible (all sync standby's goes down). >> Can't we use existing wal_sender_t

Re: [HACKERS] Standalone synchronous master

2014-01-12 Thread Stephen Frost
* Josh Berkus (j...@agliodbs.com) wrote: > Well, then that becomes a reason to want better/more configurability. I agree with this- the challenge is figuring out what those options should be and how we should document them. > In the couple of sync rep sites I admin, I *would* want to wait. That'

Re: [HACKERS] Standalone synchronous master

2014-01-12 Thread Josh Berkus
On 01/12/2014 12:35 PM, Stephen Frost wrote: > * Josh Berkus (j...@agliodbs.com) wrote: >> You don't want to handle all of those issues the same way as far as sync >> rep is concerned. For example, if the standby is restaring, you >> probably want to wait instead of degrading. > > *What*?! Certa

Re: [HACKERS] Standalone synchronous master

2014-01-12 Thread Kevin Grittner
Josh Berkus wrote: >> Add a new parameter : > >> synchronous_standalone_master = on | off > > I think this is a TERRIBLE name for any such parameter.  What does > "synchronous standalone" even mean?  A better name for the parameter > would be "auto_degrade_sync_replication" or > "synchronou

Re: [HACKERS] Standalone synchronous master

2014-01-12 Thread Stephen Frost
* Josh Berkus (j...@agliodbs.com) wrote: > On 01/11/2014 08:52 PM, Amit Kapila wrote:> It is better than async mode > in a way such that in async mode it never > > waits for commits to be written to standby, but in this new mode it will > > do so unless it is not possible (all sync standby's goes d

Re: [HACKERS] Standalone synchronous master

2014-01-12 Thread Josh Berkus
All, I'm leading this off with a review of the features offered by the actual patch submitted. My general discussion of the issues of Sync Degrade, which justifies my specific suggestions below, follows that. Rajeev, please be aware that other hackers may have different opinions than me on what

Re: [HACKERS] Standalone synchronous master

2014-01-12 Thread Florian Pflug
On Jan11, 2014, at 18:53 , Andres Freund wrote: > On 2014-01-11 18:28:31 +0100, Florian Pflug wrote: >> Hm, I was about to suggest that you can set statement_timeout before >> doing COMMIT to limit the amount of time you want to wait for the >> standby to respond. Interestingly, however, that does

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Amit Kapila
On Sat, Jan 11, 2014 at 9:41 PM, Bruce Momjian wrote: > On Sat, Jan 11, 2014 at 01:29:23PM +0530, Amit Kapila wrote: >> Okay, this is one way of providing this new mode, others could be: >> >> a. >> Have just one GUC sync_standalone_mode = true|false and make >> this as PGC_POSTMASTER parameter, s

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Bruce Momjian
On Sat, Jan 11, 2014 at 07:18:02PM -0800, Josh Berkus wrote: > In other words, if we're going to have auto-degrade, the most > intelligent place for it is in > RepMgr/HandyRep/OmniPITR/pgPoolII/whatever. It's also the *easiest* > place. Anything we do *inside* Postgres is going to have a really,

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Amit Kapila
On Sun, Jan 12, 2014 at 8:48 AM, Josh Berkus wrote: > On 01/10/2014 06:27 PM, Bruce Momjian wrote: >> How would that work? Would it be a tool in contrib? There already is a >> timeout, so if a tool checked more frequently than the timeout, it >> should work. The durable notification of the admi

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Josh Berkus
On 01/10/2014 06:27 PM, Bruce Momjian wrote: > How would that work? Would it be a tool in contrib? There already is a > timeout, so if a tool checked more frequently than the timeout, it > should work. The durable notification of the admin would happen in the > tool, right? Well, you know what

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Tom Lane
Mark Kirkwood writes [slightly rearranged] > My 2c is: > The current behavior in CAP theorem speak is 'Cap' - i.e focused on > consistency at the expense of availability. A reasonable thing to want. > The other behavior being asked for is 'cAp' - i.e focused on > availability. Also a reasonabl

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Mark Kirkwood
On 11/01/14 13:25, Stephen Frost wrote: Adrian, * Adrian Klaver (adrian.kla...@gmail.com) wrote: A) Change the existing sync mode to allow the master and standby fall out of sync should a standby fall over. I'm not sure that anyone is argueing for this.. B) Create a new mode that does this

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Andres Freund
On 2014-01-11 18:28:31 +0100, Florian Pflug wrote: > Hm, I was about to suggest that you can set statement_timeout before > doing COMMIT to limit the amount of time you want to wait for the > standby to respond. Interestingly, however, that doesn't seem to work, > which is weird, since AFAICS state

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Tom Lane
Florian Pflug writes: > Hm, I was about to suggest that you can set statement_timeout before > doing COMMIT to limit the amount of time you want to wait for the > standby to respond. Interestingly, however, that doesn't seem to work, > which is weird, since AFAICS statement_timeout simply generate

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Florian Pflug
On Jan11, 2014, at 01:48 , Joshua D. Drake wrote: > On 01/10/2014 04:38 PM, Stephen Frost wrote: >> Adrian, >> >> * Adrian Klaver (adrian.kla...@gmail.com) wrote: >>> On 01/10/2014 04:25 PM, Stephen Frost wrote: * Adrian Klaver (adrian.kla...@gmail.com) wrote: > A) Change the existing sy

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Bruce Momjian
On Sat, Jan 11, 2014 at 01:29:23PM +0530, Amit Kapila wrote: > Okay, this is one way of providing this new mode, others could be: > > a. > Have just one GUC sync_standalone_mode = true|false and make > this as PGC_POSTMASTER parameter, so that user is only > allowed to set this mode at startup. Ev

Re: [HACKERS] Standalone synchronous master

2014-01-11 Thread Amit Kapila
On Fri, Jan 10, 2014 at 9:17 PM, Bruce Momjian wrote: > On Fri, Jan 10, 2014 at 10:21:42AM +0530, Amit Kapila wrote: >> Here I think if user is aware from beginning that this is the behaviour, >> then may be the importance of message is not very high. >> What I want to say is that if we provide a

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Bruce Momjian
On Fri, Jan 10, 2014 at 03:27:10PM -0800, Josh Berkus wrote: > On 01/10/2014 01:49 PM, Andres Freund wrote: > > On 2014-01-10 10:59:23 -0800, Joshua D. Drake wrote: > >> > >> On 01/10/2014 07:47 AM, Bruce Momjian wrote: > >> > >>> I know there was a desire to remove this TODO item, but I think we h

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Bruce Momjian
On Fri, Jan 10, 2014 at 03:17:34PM -0800, Josh Berkus wrote: > The purpose of sync rep is to know determinatively whether or not you > have lost data when disaster strikes. If knowing for certain isn't > important to you, then use async. > > BTW, people are using RAID1 as an analogy to 2-node syn

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Peter Eisentraut
On Wed, 2014-01-08 at 17:56 -0500, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > That's why you should configure a second standby as another (candidate) > > synchronous replica, also listed in synchronous_standby_names. > > Perhaps we should stress in the docs that thi

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Adrian Klaver
On 01/10/2014 04:48 PM, Joshua D. Drake wrote: On 01/10/2014 04:38 PM, Stephen Frost wrote: Adrian, * Adrian Klaver (adrian.kla...@gmail.com) wrote: On 01/10/2014 04:25 PM, Stephen Frost wrote: * Adrian Klaver (adrian.kla...@gmail.com) wrote: A) Change the existing sync mode to allow the ma

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Jim Nasby
On 1/10/14, 6:19 PM, Adrian Klaver wrote: 1) Async. Runs at the speed of the master as it does not have to wait on the standby to signal a successful commit. There is some degree of offset between master and standby(s) due to latency. 2) Sync. Runs at the speed of the standby + latency between

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Joshua D. Drake
On 01/10/2014 04:38 PM, Stephen Frost wrote: Adrian, * Adrian Klaver (adrian.kla...@gmail.com) wrote: On 01/10/2014 04:25 PM, Stephen Frost wrote: * Adrian Klaver (adrian.kla...@gmail.com) wrote: A) Change the existing sync mode to allow the master and standby fall out of sync should a stand

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Stephen Frost
Adrian, * Adrian Klaver (adrian.kla...@gmail.com) wrote: > On 01/10/2014 04:25 PM, Stephen Frost wrote: > >* Adrian Klaver (adrian.kla...@gmail.com) wrote: > >>A) Change the existing sync mode to allow the master and standby > >>fall out of sync should a standby fall over. > > > >I'm not sure that

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Adrian Klaver
On 01/10/2014 04:25 PM, Stephen Frost wrote: Adrian, * Adrian Klaver (adrian.kla...@gmail.com) wrote: A) Change the existing sync mode to allow the master and standby fall out of sync should a standby fall over. I'm not sure that anyone is argueing for this.. Looks like here, unless I am r

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Stephen Frost
Adrian, * Adrian Klaver (adrian.kla...@gmail.com) wrote: > A) Change the existing sync mode to allow the master and standby > fall out of sync should a standby fall over. I'm not sure that anyone is argueing for this.. > B) Create a new mode that does this without changing the existing sync mod

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Adrian Klaver
On 01/10/2014 03:38 PM, Joshua D. Drake wrote: On 01/10/2014 03:17 PM, Josh Berkus wrote: Any continuous replication should not be a SPOF. The current behavior guarantees that a two node sync cluster is a SPOF. The proposed behavior removes that. Again, if that's your goal, then use async re

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Joshua D. Drake
On 01/10/2014 03:17 PM, Josh Berkus wrote: Any continuous replication should not be a SPOF. The current behavior guarantees that a two node sync cluster is a SPOF. The proposed behavior removes that. Again, if that's your goal, then use async replication. I think I have gone about this the

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Josh Berkus
On 01/10/2014 01:49 PM, Andres Freund wrote: > On 2014-01-10 10:59:23 -0800, Joshua D. Drake wrote: >> >> On 01/10/2014 07:47 AM, Bruce Momjian wrote: >> >>> I know there was a desire to remove this TODO item, but I think we have >>> brought up enough new issues that we can keep it to see if we can

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Josh Berkus
On 01/10/2014 02:59 PM, Joshua D. Drake wrote: > > On 01/10/2014 02:47 PM, Andres Freund wrote: > >> Really, the commits themselves are sent to the server at exactly the >> same speed independent of sync/async. The only thing that's delayed is >> the *notificiation* of the client that sent the co

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Hannu Krosing
On 01/10/2014 11:59 PM, Joshua D. Drake wrote: > > On 01/10/2014 02:57 PM, Stephen Frost wrote: > >> Yes, if you have a BBU that memory is authoritative in most >> cases. But >> in that case the argument of having two disks is pretty much >> pointless, >> the SPOF suddenly became the ba

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Joshua D. Drake
On 01/10/2014 02:57 PM, Stephen Frost wrote: Yes, if you have a BBU that memory is authoritative in most cases. But in that case the argument of having two disks is pretty much pointless, the SPOF suddenly became the battery + ram. If that is a concern then use multiple controller

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Joshua D. Drake
On 01/10/2014 02:47 PM, Andres Freund wrote: Really, the commits themselves are sent to the server at exactly the same speed independent of sync/async. The only thing that's delayed is the *notificiation* of the client that sent the commit. Not the commit itself. Which is irrelevant to the po

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Stephen Frost
Greetings, On Friday, January 10, 2014, Andres Freund wrote: > Hi, > > On 2014-01-10 17:28:55 -0500, Stephen Frost wrote: > > > Why do you know that you didn't loose any transactions? Trivial network > > > hiccups, a restart of a standby, IO overload on the standby all can > > > cause a very shor

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Andres Freund
On 2014-01-10 14:44:28 -0800, Joshua D. Drake wrote: > > On 01/10/2014 02:33 PM, Andres Freund wrote: > > > >On 2014-01-10 14:29:58 -0800, Joshua D. Drake wrote: > >>db02 goes down. It doesn't matter why. It is down. db01 continues to accept > >>orders, allow people to log into the website and we

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Jeff Janes
On Fri, Jan 10, 2014 at 2:33 PM, Andres Freund wrote: > On 2014-01-10 14:29:58 -0800, Joshua D. Drake wrote: > > db02 goes down. It doesn't matter why. It is down. db01 continues to > accept > > orders, allow people to log into the website and we can still service > > accounts. The continuity of s

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Andres Freund
Hi, On 2014-01-10 17:28:55 -0500, Stephen Frost wrote: > > Why do you know that you didn't loose any transactions? Trivial network > > hiccups, a restart of a standby, IO overload on the standby all can > > cause a very short interruptions in the walsender connection - leading > > to degradation.

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Joshua D. Drake
On 01/10/2014 02:33 PM, Andres Freund wrote: On 2014-01-10 14:29:58 -0800, Joshua D. Drake wrote: db02 goes down. It doesn't matter why. It is down. db01 continues to accept orders, allow people to log into the website and we can still service accounts. The continuity of service continues. W

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Adrian Klaver
On 01/10/2014 02:33 PM, Andres Freund wrote: On 2014-01-10 14:29:58 -0800, Joshua D. Drake wrote: db02 goes down. It doesn't matter why. It is down. db01 continues to accept orders, allow people to log into the website and we can still service accounts. The continuity of service continues. Why

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Andres Freund
On 2014-01-10 14:29:58 -0800, Joshua D. Drake wrote: > db02 goes down. It doesn't matter why. It is down. db01 continues to accept > orders, allow people to log into the website and we can still service > accounts. The continuity of service continues. Why is that configuration advantageous over a

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Joshua D. Drake
On 01/10/2014 01:49 PM, Andres Freund wrote: I know I am the one that instigated all of this so I want to be very clear on what I and what I am confident that my customers would expect. If a synchronous slave goes down, the master continues to operate. That is all. I don't care if it is config

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Stephen Frost
Andres, On Friday, January 10, 2014, Andres Freund wrote: > On 2014-01-10 17:02:08 -0500, Stephen Frost wrote: > > * Andres Freund (and...@2ndquadrant.com ) wrote: > > > On 2014-01-10 10:59:23 -0800, Joshua D. Drake wrote: > > > > If a synchronous slave goes down, the master continues to operate.

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Andres Freund
On 2014-01-10 17:02:08 -0500, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > On 2014-01-10 10:59:23 -0800, Joshua D. Drake wrote: > > > If a synchronous slave goes down, the master continues to operate. That is > > > all. I don't care if it is configurable (I would be fi

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > On 2014-01-10 10:59:23 -0800, Joshua D. Drake wrote: > > If a synchronous slave goes down, the master continues to operate. That is > > all. I don't care if it is configurable (I would be fine with that). I don't > > care if it is not automatic (e.g

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Andres Freund
On 2014-01-10 10:59:23 -0800, Joshua D. Drake wrote: > > On 01/10/2014 07:47 AM, Bruce Momjian wrote: > > >I know there was a desire to remove this TODO item, but I think we have > >brought up enough new issues that we can keep it to see if we can come > >up with a solution. I have added a link

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Jim Nasby
On 1/10/14, 12:59 PM, Joshua D. Drake wrote: I know I am the one that instigated all of this so I want to be very clear on what I and what I am confident that my customers would expect. If a synchronous slave goes down, the master continues to operate. That is all. I don't care if it is config

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Joshua D. Drake
On 01/10/2014 07:47 AM, Bruce Momjian wrote: I know there was a desire to remove this TODO item, but I think we have brought up enough new issues that we can keep it to see if we can come up with a solution. I have added a link to this discussion on the TODO item. I think we will need at leas

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Hannu Krosing
On 01/10/2014 05:09 PM, Simon Riggs wrote: > On 10 January 2014 15:47, Bruce Momjian wrote: > >> I know there was a desire to remove this TODO item, but I think we have >> brought up enough new issues that we can keep it to see if we can come >> up with a solution. > Can you summarise what you thi

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Simon Riggs
On 10 January 2014 15:47, Bruce Momjian wrote: > I know there was a desire to remove this TODO item, but I think we have > brought up enough new issues that we can keep it to see if we can come > up with a solution. Can you summarise what you think the new issues are? All I see is some further r

Re: [HACKERS] Standalone synchronous master

2014-01-10 Thread Bruce Momjian
On Fri, Jan 10, 2014 at 10:21:42AM +0530, Amit Kapila wrote: > On Thu, Jan 9, 2014 at 10:45 PM, Bruce Momjian wrote: > > > > I think RAID-1 is a very good comparison because it is successful > > technology and has similar issues. > > > > RAID-1 is like Postgres synchronous_standby_names mode in th

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Michael Paquier
On Fri, Jan 10, 2014 at 3:23 AM, Simon Riggs wrote: > On 8 January 2014 21:40, Tom Lane wrote: >> Kevin Grittner writes: >>> I'm torn on whether we should cave to popular demand on this; but >>> if we do, we sure need to be very clear in the documentation about >>> what a successful return from

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Amit Kapila
On Thu, Jan 9, 2014 at 10:45 PM, Bruce Momjian wrote: > > I think RAID-1 is a very good comparison because it is successful > technology and has similar issues. > > RAID-1 is like Postgres synchronous_standby_names mode in the sense that > the RAID-1 controller will not return success until writes

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Jim Nasby
On 1/9/14, 9:01 AM, Hannu Krosing wrote: Yeah, and I think that the logging command that was suggested allows >for that*if configured correctly*. *But* for relying on this, we would also need to make logging *synchronous*, which would probably not go down well with many people, as it makes thin

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Simon Riggs
On 8 January 2014 21:40, Tom Lane wrote: > Kevin Grittner writes: >> I'm torn on whether we should cave to popular demand on this; but >> if we do, we sure need to be very clear in the documentation about >> what a successful return from a commit request means. Sooner or >> later, Murphy's Law b

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Josh Berkus
Robert, > I think the problem here is that we tend to have a limited view of > "the right way to use synch rep". If I have 5 nodes, and I set 1 > synchronous and the other 3 asynchronous, I've set up a "known > successor" in the event that the leader fails. In this scenario > though, if the "succe

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Bruce Momjian
On Thu, Jan 9, 2014 at 09:36:47AM -0800, Jeff Janes wrote: > Oh, right. Because the main reason for a sync replica degrading is that > it's down. In which case it isn't going to record anything. This would > still be useful for sync rep candidates, though, and I'll document why >

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Jeff Janes
On Wed, Jan 8, 2014 at 3:00 PM, Josh Berkus wrote: > On 01/08/2014 01:49 PM, Tom Lane wrote: > > Josh Berkus writes: > >> If we really want auto-degrading sync rep, then we'd (at a minimum) need > >> a way to determine *from the replica* whether or not it was in degraded > >> mode when the maste

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Bruce Momjian
On Thu, Jan 9, 2014 at 04:55:22PM +0100, Hannu Krosing wrote: > On 01/09/2014 04:15 PM, MauMau wrote: > > From: "Hannu Krosing" > >> On 01/09/2014 01:57 PM, MauMau wrote: > >>> Let me ask a (probably) stupid question. How is the sync rep > >>> different from RAID-1? > >>> > >>> When I first saw

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
On 01/09/2014 04:15 PM, MauMau wrote: > From: "Hannu Krosing" >> On 01/09/2014 01:57 PM, MauMau wrote: >>> Let me ask a (probably) stupid question. How is the sync rep >>> different from RAID-1? >>> >>> When I first saw sync rep, I expected that it would provide the same >>> guarantees as RAID-1

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread MauMau
From: "Hannu Krosing" On 01/09/2014 01:57 PM, MauMau wrote: Let me ask a (probably) stupid question. How is the sync rep different from RAID-1? When I first saw sync rep, I expected that it would provide the same guarantees as RAID-1 in terms of durability (data is always mirrored on two serv

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
On 01/09/2014 02:01 AM, Jim Nasby wrote: > On 1/8/14, 6:05 PM, Tom Lane wrote: >> Josh Berkus writes: >>> >On 01/08/2014 03:27 PM, Tom Lane wrote: >>What we lack, and should work on, is a way for sync mode to have M larger >>than one. AFAICS, right now we'll report commit as soon a

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
On 01/09/2014 01:57 PM, MauMau wrote: > From: "Andres Freund" >> On 2014-01-08 14:42:37 -0800, Joshua D. Drake wrote: >>> If we have the following: >>> >>> db0->db1:down >>> >>> Using the model (as I understand it) that is being discussed we have >>> increased our failure rate because the moment d

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
On 01/08/2014 11:49 PM, Tom Lane wrote: > "Joshua D. Drake" writes: >> On 01/08/2014 01:55 PM, Tom Lane wrote: >>> Sync mode is about providing a guarantee that the data exists on more than >>> one server *before* we tell the client it's committed. If you don't need >>> that guarantee, you should

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
On 01/09/2014 12:05 AM, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: >> On 2014-01-08 17:56:37 -0500, Stephen Frost wrote: >>> * Andres Freund (and...@2ndquadrant.com) wrote: That's why you should configure a second standby as another (candidate) synchronous repl

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread Hannu Krosing
On 01/09/2014 05:09 AM, Robert Treat wrote: > On Wed, Jan 8, 2014 at 6:15 PM, Josh Berkus wrote: >> Stephen, >> >> >>> I'm aware, my point was simply that we should state, up-front in >>> 25.2.7.3 *and* where we document synchronous_standby_names, that it >>> requires at least three servers to be

Re: [HACKERS] Standalone synchronous master

2014-01-09 Thread MauMau
From: "Andres Freund" On 2014-01-08 14:42:37 -0800, Joshua D. Drake wrote: If we have the following: db0->db1:down Using the model (as I understand it) that is being discussed we have increased our failure rate because the moment db1:down we also lose db0. The node db0 may be up but if it i

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Robert Treat
On Wed, Jan 8, 2014 at 6:15 PM, Josh Berkus wrote: > Stephen, > > >> I'm aware, my point was simply that we should state, up-front in >> 25.2.7.3 *and* where we document synchronous_standby_names, that it >> requires at least three servers to be involved to be a workable >> solution. > > It's a wo

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Jim Nasby
On 1/8/14, 6:05 PM, Tom Lane wrote: Josh Berkus writes: >On 01/08/2014 03:27 PM, Tom Lane wrote: >>What we lack, and should work on, is a way for sync mode to have M larger >>than one. AFAICS, right now we'll report commit as soon as there's one >>up-to-date replica, and some high-reliability

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Tom Lane
Josh Berkus writes: > On 01/08/2014 03:27 PM, Tom Lane wrote: >> What we lack, and should work on, is a way for sync mode to have M larger >> than one. AFAICS, right now we'll report commit as soon as there's one >> up-to-date replica, and some high-reliability cases are going to want >> more. >

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Jeff Janes
On Wed, Jan 8, 2014 at 2:56 PM, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > That's why you should configure a second standby as another (candidate) > > synchronous replica, also listed in synchronous_standby_names. > > Perhaps we should stress in the docs that this

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Josh Berkus
On 01/08/2014 03:27 PM, Tom Lane wrote: > Good point, but C can't solve this for you just by logging. If C was the > first to go down, it has no way to know whether A and B committed more > transactions before dying; and it's unlikely to have logged its own crash, > either. Sure. But if we *knew

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Jeff Janes
On Wed, Jan 8, 2014 at 2:23 PM, Joshua D. Drake wrote: > > On 01/08/2014 01:55 PM, Tom Lane wrote: > > Sync mode is about providing a guarantee that the data exists on more than >> one server *before* we tell the client it's committed. If you don't need >> that guarantee, you shouldn't be using

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Tom Lane
Josh Berkus writes: > HOWEVER, we've already kind of set up an indeterminate situation with > allowing sync rep groups and candidate sync rep servers. Consider this: > 1. Master server A is configured with sync replica B and candidate sync > replica C > 2. A rolling power/network failure event

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Josh Berkus
On 01/08/2014 03:18 PM, Stephen Frost wrote: > Do you really feel that a WARNING and increasing the docs to point > out that three systems are necessary, particularly under the 'high > availability' documentation and options, is a bad idea? I fail to see > how that does anything but clarify the us

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Stephen Frost
Josh, * Josh Berkus (j...@agliodbs.com) wrote: > > I'm aware, my point was simply that we should state, up-front in > > 25.2.7.3 *and* where we document synchronous_standby_names, that it > > requires at least three servers to be involved to be a workable > > solution. > > It's a workable solutio

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Tom Lane
Stephen Frost writes: > I'm aware, my point was simply that we should state, up-front in > 25.2.7.3 *and* where we document synchronous_standby_names, that it > requires at least three servers to be involved to be a workable > solution. It only requires that if your requirements include both redu

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Josh Berkus
Stephen, > I'm aware, my point was simply that we should state, up-front in > 25.2.7.3 *and* where we document synchronous_standby_names, that it > requires at least three servers to be involved to be a workable > solution. It's a workable solution with 2 servers. That's a "low-availability, hi

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > On 2014-01-08 17:56:37 -0500, Stephen Frost wrote: > > * Andres Freund (and...@2ndquadrant.com) wrote: > > > That's why you should configure a second standby as another (candidate) > > > synchronous replica, also listed in synchronous_standby_names.

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Andres Freund
On 2014-01-08 14:52:07 -0800, Joshua D. Drake wrote: > On 01/08/2014 02:46 PM, Andres Freund wrote: > >>The idea is that we know that data on db0 is not written until we know for a > >>fact that db1 also has that data. That is great and a guarantee of data > >>integrity between the two nodes. > > >

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Josh Berkus
On 01/08/2014 01:49 PM, Tom Lane wrote: > Josh Berkus writes: >> If we really want auto-degrading sync rep, then we'd (at a minimum) need >> a way to determine *from the replica* whether or not it was in degraded >> mode when the master died. What good do messages to the master log do >> you if t

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Andres Freund
On 2014-01-08 17:56:37 -0500, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > That's why you should configure a second standby as another (candidate) > > synchronous replica, also listed in synchronous_standby_names. > > Perhaps we should stress in the docs that this is,

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Joshua D. Drake
On 01/08/2014 02:49 PM, Tom Lane wrote: Then you don't understand the point of sync mode, and you shouldn't be using it. The point is *exactly* to refuse to commit transactions unless we can guarantee the data's been replicated. I understand exactly that and I don't disagree, except in the c

Re: [HACKERS] Standalone synchronous master

2014-01-08 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > That's why you should configure a second standby as another (candidate) > synchronous replica, also listed in synchronous_standby_names. Perhaps we should stress in the docs that this is, in fact, the *only* reasonable mode in which to run with syn

  1   2   >