Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-25 Thread Fujii Masao
On Sat, Mar 19, 2011 at 4:29 AM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2011-03-18 at 20:19 +0100, Markus Wanner wrote: Simon, On 03/18/2011 05:19 PM, Simon Riggs wrote: Simon Riggssi...@2ndquadrant.com  wrote: In PostgreSQL other users cannot observe the commit until an

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-23 Thread Markus Wanner
On 03/22/2011 09:33 PM, Robert Haas wrote: We might have a version of synchronous replication that works this way some day, but it's not the version were shipping with 9.1. The slave acknowledges the WAL records when they hit the disk (i.e. fsync) not when they are applied; WAL apply can lag

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-23 Thread Robert Haas
On Wed, Mar 23, 2011 at 3:27 AM, Markus Wanner mar...@bluegap.ch wrote: On 03/22/2011 09:33 PM, Robert Haas wrote: We might have a version of synchronous replication that works this way some day, but it's not the version were shipping with 9.1.  The slave acknowledges the WAL records when they

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-23 Thread Markus Wanner
On 03/23/2011 12:52 PM, Robert Haas wrote: Yes. What this won't do is let you build a big load-balancing network (at least not without great caution about what you assume). This sounds too strong to me. Session-aware load balancing is pretty common these days. It's the default mode of

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-23 Thread Robert Haas
On Wed, Mar 23, 2011 at 8:16 AM, Markus Wanner mar...@bluegap.ch wrote: On 03/23/2011 12:52 PM, Robert Haas wrote: Yes.  What this won't do is let you build a big load-balancing network (at least not without great caution about what you assume). This sounds too strong to me.  Session-aware

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-22 Thread Aidan Van Dyk
On Fri, Mar 18, 2011 at 3:41 PM, Markus Wanner mar...@bluegap.ch wrote: On 03/18/2011 08:29 PM, Simon Riggs wrote: We could do that easily enough, actually, if we wished. Do we wish? I personally don't see any problem letting a standby show a snapshot before the master.  I'd consider it

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-22 Thread Robert Haas
On Fri, Mar 18, 2011 at 5:08 PM, Aidan Van Dyk ai...@highrise.ca wrote: On Fri, Mar 18, 2011 at 3:41 PM, Markus Wanner mar...@bluegap.ch wrote: On 03/18/2011 08:29 PM, Simon Riggs wrote: We could do that easily enough, actually, if we wished. Do we wish? I personally don't see any problem

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-19 Thread Markus Wanner
On 03/18/2011 10:48 PM, Kevin Grittner wrote: the least of the evils. I guess we should document it, though, so nobody has a false expectation that seeing something on the replica means that a connection looking at the master will see something that current. Agreed. Note, however, that even

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
Hi, sorry for being late to join that bike-shedding discussion. On 03/07/2011 05:09 PM, Alvaro Herrera wrote: I think these terms are used inconsistenly enough across the industry that what would make the most sense would be to use the common term and document accurately what we mean by it,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 9:16 AM, MARK CALLAGHAN mdcal...@gmail.com wrote: On Fri, Mar 18, 2011 at 9:27 AM, Markus Wanner mar...@bluegap.ch wrote: Google invented the term semi-syncronous for something that's essentially the same that we have, now, I think.  However, I full heartedly hate that

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
MARK CALLAGHAN mdcal...@gmail.com wrote: Markus Wanner mar...@bluegap.ch wrote: Google invented the term semi-syncronous for something that's essentially the same that we have, now, I think. However, I full heartedly hate that term (based on the reasoning that there's no semi-pregnant,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 13:16 +, MARK CALLAGHAN wrote: On Fri, Mar 18, 2011 at 9:27 AM, Markus Wanner mar...@bluegap.ch wrote: Google invented the term semi-syncronous for something that's essentially the same that we have, now, I think. However, I full heartedly hate that term (based on

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
Mark, On 03/18/2011 02:16 PM, MARK CALLAGHAN wrote: We didn't invent the term, we just implemented something that Heikki Tuuri briefly described, for example: http://bugs.mysql.com/bug.php?id=7440 Oh, okay, good to know who to blame ;-) However, I didn't mean to offend anybody. I do not

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
Hi, On 03/18/2011 02:40 PM, Kevin Grittner wrote: Then the only thing you would consider sync replication, as far as I can see, is two phase commit I think waiting for the ACK before actually making the changes from the transaction visible (COMMIT) would suffice for disallowing such an

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: In PostgreSQL other users cannot observe the commit until an acknowledgement has been received. Really? I hadn't picked up on that. That makes for a lot of complication on crash-and-recovery of a master, but if we can pull it off, that's really

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 11:07 -0400, Robert Haas wrote: On Fri, Mar 18, 2011 at 10:55 AM, Greg Stark gsst...@mit.edu wrote: On Thu, Mar 17, 2011 at 5:46 PM, Robert Haas robertmh...@gmail.com wrote: What makes more sense to me after having thought about this more carefully is to simply make a

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
On 03/18/2011 03:52 PM, Kevin Grittner wrote: Really? I hadn't picked up on that. That makes for a lot of complication on crash-and-recovery of a master What complication do you have in mind here? I think of it the opposite way (at least for Postgres, that is): committing a transaction

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Heikki Linnakangas
On 18.03.2011 16:52, Kevin Grittner wrote: Simon Riggssi...@2ndquadrant.com wrote: In PostgreSQL other users cannot observe the commit until an acknowledgement has been received. Really? I hadn't picked up on that. That makes for a lot of complication on crash-and-recovery of a master,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread MARK CALLAGHAN
On Fri, Mar 18, 2011 at 2:19 PM, Markus Wanner mar...@bluegap.ch wrote: Their documentation [1] isn't entirely clear on that first: the master blocks after the commit is done and waits until at least one semisynchronous slave acknowledges that it has received all events for the transaction

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread MARK CALLAGHAN
On Fri, Mar 18, 2011 at 2:37 PM, Markus Wanner mar...@bluegap.ch wrote: Hi, On 03/18/2011 02:40 PM, Kevin Grittner wrote: Then the only thing you would consider sync replication, as far as I can see, is two phase commit I think waiting for the ACK before actually making the changes from the

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 17:47 +0200, Heikki Linnakangas wrote: On 18.03.2011 16:52, Kevin Grittner wrote: Simon Riggssi...@2ndquadrant.com wrote: In PostgreSQL other users cannot observe the commit until an acknowledgement has been received. Really? I hadn't picked up on that. That

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
On 18.03.2011 16:52, Kevin Grittner wrote: Simon Riggssi...@2ndquadrant.com wrote: In PostgreSQL other users cannot observe the commit until an acknowledgement has been received. Really? I hadn't picked up on that. That makes for a lot of complication on crash-and-recovery of a

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 12:19 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2011-03-18 at 17:47 +0200, Heikki Linnakangas wrote: On 18.03.2011 16:52, Kevin Grittner wrote: Simon Riggssi...@2ndquadrant.com  wrote: In PostgreSQL other users cannot observe the commit until an

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Simon Riggs si...@2ndquadrant.com wrote: No, only in the case where you choose not to failover to the standby when you crash, which would be a fairly strange choice after the effort to set up the standby. In a correctly configured and operated

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Greg Stark
On Fri, Mar 18, 2011 at 4:33 PM, Robert Haas robertmh...@gmail.com wrote: The fundamental problem here is that once you update CLOG and flush the corresponding WAL record, there is no going backward.  You can hold the system in some intermediate state where the transaction still holds locks

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
On 03/18/2011 06:35 PM, Greg Stark wrote: I think promising that the COMMIT doesn't return until the transaction and all previous transactions are replicated is enough. We don't have to promise that nobody else will see it either. Those same transactions eventually have to commit as well No,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
Simon, On 03/18/2011 05:19 PM, Simon Riggs wrote: Simon Riggssi...@2ndquadrant.com wrote: In PostgreSQL other users cannot observe the commit until an acknowledgement has been received. On other nodes as well? To me that means the standby needs to hold back COMMIT of an ACKed transaction,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
On 03/18/2011 05:27 PM, Kevin Grittner wrote: Basically, what Heikki addresses. It has to be committed after crash and recovery, and deal with replicas which may or may not have been notified and may or may not have applied the transaction. Huh? I'm not quite following here. Committing

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 20:19 +0100, Markus Wanner wrote: Simon, On 03/18/2011 05:19 PM, Simon Riggs wrote: Simon Riggssi...@2ndquadrant.com wrote: In PostgreSQL other users cannot observe the commit until an acknowledgement has been received. On other nodes as well? To me that means

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2011-03-18 at 20:19 +0100, Markus Wanner wrote: Simon Riggssi...@2ndquadrant.com wrote: In PostgreSQL other users cannot observe the commit until an acknowledgement has been received. On other nodes as well? To me that means the

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Markus Wanner
On 03/18/2011 08:29 PM, Simon Riggs wrote: We could do that easily enough, actually, if we wished. Do we wish? I personally don't see any problem letting a standby show a snapshot before the master. I'd consider it unneeded network traffic. But then again, I'm completely biased. Regards

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 3:29 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2011-03-18 at 20:19 +0100, Markus Wanner wrote: Simon, On 03/18/2011 05:19 PM, Simon Riggs wrote: Simon Riggssi...@2ndquadrant.com  wrote: In PostgreSQL other users cannot observe the commit until an

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Since the current solution is intended to support data-loss-free failover, but NOT to guarantee a consistent view of the world from a SQL level, I doubt it's worth paying any price for this. Well, that brings us back to the question of why we would

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 17:08 -0400, Aidan Van Dyk wrote: On Fri, Mar 18, 2011 at 3:41 PM, Markus Wanner mar...@bluegap.ch wrote: On 03/18/2011 08:29 PM, Simon Riggs wrote: We could do that easily enough, actually, if we wished. Do we wish? I personally don't see any problem letting a

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Simon Riggs
On Fri, 2011-03-18 at 16:24 -0500, Kevin Grittner wrote: Robert Haas robertmh...@gmail.com wrote: Since the current solution is intended to support data-loss-free failover, but NOT to guarantee a consistent view of the world from a SQL level, I doubt it's worth paying any price for this.

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 5:24 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: Since the current solution is intended to support data-loss-free failover, but NOT to guarantee a consistent view of the world from a SQL level, I doubt it's worth

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote: Well, the idea is that we don't want to let people depend on the value until it's guaranteed to be durably committed. OK, so if you see it on the replica, you know it is in at least two places. I guess that makes sense. It kinda feels wrong to see a

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-18 Thread Robert Haas
On Fri, Mar 18, 2011 at 5:48 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Robert Haas robertmh...@gmail.com wrote: Well, the idea is that we don't want to let people depend on the value until it's guaranteed to be durably committed. OK, so if you see it on the replica, you know it is

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Simon Riggs
On Mon, 2011-03-07 at 17:44 +0900, Fujii Masao wrote: The above check should be required also after pg_ctl reload since synchronous_standby_names can be changed by SIGHUP? Or how about just removing that? If the patch I submitted is committed,empty synchronous_standby_names and

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Andrew Dunstan
On 03/07/2011 02:29 AM, Heikki Linnakangas wrote: On 07.03.2011 01:28, Simon Riggs wrote: On Sun, 2011-03-06 at 18:09 -0500, Andrew Dunstan wrote: On 03/06/2011 05:51 PM, Simon Riggs wrote: Efficient transaction-controlled synchronous replication. I'm glad this is in, but I thought we

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Heikki Linnakangas
On 07.03.2011 15:30, Andrew Dunstan wrote: Previously, Simon said: Truly synchronous requires two-phase commit, which this never was. So I too am confused about how it's now become truly synchronous. Are we saying this give the same or better guarantees than a 2PC setup? The guarantee we

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Andrew Dunstan
On 03/07/2011 09:02 AM, Heikki Linnakangas wrote: On 07.03.2011 15:30, Andrew Dunstan wrote: Previously, Simon said: Truly synchronous requires two-phase commit, which this never was. So I too am confused about how it's now become truly synchronous. Are we saying this give the same or

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Aidan Van Dyk
On Mon, Mar 7, 2011 at 2:21 PM, Andrew Dunstan and...@dunslane.net wrote: For me, that's enough to call it synchronous replication. It provides a useful guarantee to the client. But you could argue for an even stricter definition, requiring atomicity so that if a transaction is not

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Kevin Grittner
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: if you pull the power plug, the transaction that was just being committed might be committed on the master, but not yet on the standby. For me, that's enough to call it synchronous replication. It provides useful guarantee to

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Andrew Dunstan
On 03/07/2011 09:29 AM, Aidan Van Dyk wrote: On Mon, Mar 7, 2011 at 2:21 PM, Andrew Dunstanand...@dunslane.net wrote: For me, that's enough to call it synchronous replication. It provides a useful guarantee to the client. But you could argue for an even stricter definition, requiring

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Aidan Van Dyk
On Mon, Mar 7, 2011 at 2:29 PM, Aidan Van Dyk ai...@highrise.ca wrote: They they are either already hosed or already using 2PC. Sorry, to expand on my all too brief comment, even *without* replication, they are hosed. Once you issue commit, you have know knowledge if the commit is durable, (or

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Kevin Grittner
Andrew Dunstan and...@dunslane.net wrote: Synchronous replication - guarantees zero data loss by the means of atomic write operation, i.e. write either completes on both sides or not at all. So far, so good. Write is not considered complete until acknowledgement by both local

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Heikki Linnakangas
On 07.03.2011 17:03, Andrew Dunstan wrote: This is about expectations. The thing that worries me is that the use of this term might cause some people NOT to use 2PC because they think they are getting an equivalent guarantee, when in fact they are not. And that's hardly unreasonable. Here for

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Andrew Dunstan
On 03/07/2011 10:46 AM, Heikki Linnakangas wrote: On 07.03.2011 17:03, Andrew Dunstan wrote: This is about expectations. The thing that worries me is that the use of this term might cause some people NOT to use 2PC because they think they are getting an equivalent guarantee, when in fact they

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-07 Thread Alvaro Herrera
Excerpts from Andrew Dunstan's message of lun mar 07 12:51:49 -0300 2011: On 03/07/2011 10:46 AM, Heikki Linnakangas wrote: Hmm, I've read that wikipedia definition before, but the atomic part never caught my eye. You do get zero data loss with what we have; if a meteor strikes the

[HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-06 Thread Simon Riggs
On Sun, 2011-03-06 at 18:09 -0500, Andrew Dunstan wrote: On 03/06/2011 05:51 PM, Simon Riggs wrote: Efficient transaction-controlled synchronous replication. I'm glad this is in, but I thought we agreed NOT to call it synchronous replication. The discussion on the thread was that its

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-06 Thread Heikki Linnakangas
On 07.03.2011 01:28, Simon Riggs wrote: On Sun, 2011-03-06 at 18:09 -0500, Andrew Dunstan wrote: On 03/06/2011 05:51 PM, Simon Riggs wrote: Efficient transaction-controlled synchronous replication. I'm glad this is in, but I thought we agreed NOT to call it synchronous replication. The

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-06 Thread Simon Riggs
On Mon, 2011-03-07 at 09:29 +0200, Heikki Linnakangas wrote: I presume you didn't make allow_synchronous_standby=off the default behavior. You presume incorrectly. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services

Re: [HACKERS] Re: [COMMITTERS] pgsql: Efficient transaction-controlled synchronous replication.

2011-03-06 Thread Heikki Linnakangas
On 07.03.2011 09:48, Simon Riggs wrote: On Mon, 2011-03-07 at 09:29 +0200, Heikki Linnakangas wrote: I presume you didn't make allow_synchronous_standby=off the default behavior. Sorry, s/allow_synchronous_standby/allow_standalone_master You presume incorrectly. Ok, ok then. Thank you!